In my PostgresOperator, I would like to pass SQL file path as variable to sql parameter. I am reading this file path from a configuration file:
sql_execution = PostgresOperator(
task_id='sql_query',
postgres_conn_id=task_config['postgres_conn'],
sql=task_config['sql_file']
)
As far as I understood, the operator treat the variable content as SQL statement rather than file path to SQL file. The error is as follow:
psycopg2.errors.SyntaxError: syntax error at or near "scripts"
LINE 1: scripts/samples/hello_world.sql
I have tried the following variations:
Define full path for sql file:
sql_file = os.path.join(os.path.dirname(__file__), task_config['sql_file'])
sql_execution = PostgresOperator(
task_id=task_id,
postgres_conn_id=task_config['postgres_conn_id'],
sql=sql_file,
)
sql_file = os.path.join(os.path.dirname(__file__), task_config['sql_file'])
sql_execution = PostgresOperator(
task_id=task_id,
postgres_conn_id=task_config['postgres_conn_id'],
sql=f'{sql_file}',
)
The operator still treat it as SQL query rather than a file path to SQL file.