I am trying to perform in-memory updates on a very large PySpark DataFrame instead of making disk-based updates in a PostgreSQL database. I chose PySpark for its speed and scalability over direct database updates.