I’m trying to run a Glue Job from Apache Airflow with the following code:

with TaskGroup(group_id= DAG_NAME + '_group', dag=dag) as tg:
    for job in jobs_params:

        glue = GlueJobOperator(
            task_id=f'run_{job["name"]}',
            job_name=job["name"],
            script_location=job["script_location"],
            script_args=job["script_args"],
            aws_conn_id='glue_job_conn',
            region_name='us-east-1',
            iam_role_name="AWSGlueServiceRoleReadS3",
            create_job_kwargs={"Command": {"PythonVersion": "3"},
                                "GlueVersion": "4.0", "WorkerType": "G.1X", "NumberOfWorkers": 2},
            deferrable=False,
            verbose=True,
            
            dag=dag,
        )  

I’ve tried everything but I keep getting the following error:

LAUNCH ERROR | Python 2 is not supported with this glue versionPlease refer logs for details.

What am I missing? I don’t understand why it’s trying to create a job with Python2 in the first place. But even using create_job_kwargs the problem is not solved.