I am trying to add a task using the SqlToS3Operator and running into trouble running it. I get:
ModuleNotFoundError: No module name 'pandas'
... trace...
Exception: pandas library not installed. run: pip install 'apache-airflow-providers-common-sql[pandas]'
The task in question is defined as:
get_events = SqlToS3Operator(
task_id='get_events',
...
query='select columns from event where greatest(created_at, updated_at) > {{ ts }}',
dag=dag,
)
and our requirements.txt file
apache-airflow>=2.9.0
apache-airflow-providers-amazon>=8.23.0
apache-airflow-providers-common-sql>=1.11.1
pandas>=2.1.4
Since adding the requirements.txt to the project (it wasn’t needed prior to this task) I updated the MWAA config to use the file and restarted. After restarting I am still getting the error (but at least the rest of my dags are still running).
What am I missing?