`I am working airflow xcom pull inside for task with DatabricksRunNowOperator as below.
-
values from xcom pull are giving s3 files path.
ex: “s3_filepath_1,s3_file_path_2” -
when I replace ‘,’ with ‘”,”‘ its not working .
-
so i tried this -> replace(‘,’,’@,@’) | replace(‘@’, ‘”‘). this is bringing string with ” like this-> “dev/dir/file”,”dev/dir2/file2″
-
how to escape replace(‘@’, ‘”‘) double quote inside this
CODE:
s3_to_databricks: DatabricksRunNowOperator = DatabricksRunNowOperator(
dag=dag,
task_id="s3_to_databricks",
job_id=api.get("databricks_jobs").get(ENVIRONMENT),
python_params=f"{{{{ ti.xcom_pull(task_ids=[params.extract_task_id], key='s3_object_keys') | replace('[','') | replace(']','') | replace("'",'') | replace(' ','') | replace(',','@,@') | replace('@', '"') }}}}",
)