I’m working on a Python script to process large datasets, but I’m running into performance issues. The script involves reading data from CSV files, performing some calculations, and then writing the results back to a file. Here’s a simplified version of my code:

import pandas as pd

def process_data(file_path):
    data = pd.read_csv(file_path)
    # Some complex calculations
    result = data.apply(some_complex_function, axis=1)
    result.to_csv('output.csv', index=False)

def some_complex_function(row):
    # Placeholder for complex calculations
    return row

process_data('input.csv')

The processing time increases significantly with larger datasets. I’ve already tried using chunksize in pd.read_csv but didn’t see much improvement. What are some best practices or techniques in Python for optimizing performance when dealing with large datasets?

I would appreciate any insights or suggestions on how to make this script more efficient. Thank you!