I have the use-case below and not sure how to make this:
I have the following multi partitions:
multiple_partition = MultiPartitionsDefinition({
"date": DailyPartitionsDefinition(start_date="2024-06-06", end_offset=1),
"cycle": StaticPartitionsDefinition(['u0', 'u1', 'u2', 'u3', 'u4']),
})
And I have the below 6 assets (just a simple extract and transform):
extract_AUL -> transform_AUL
extract_FV -> transform_FV
extract_TN -> transform_TN
then, I try to define 5 schedules and 5 jobs to run the assets:
(schedule_1, job_1) -> (6 assets above, only the cycle 'u0')
(schedule_2, job_2) -> (6 assets above, only the cycle 'u1')
(schedule_3, job_3) -> (6 assets above, only the cycle 'u2')
(schedule_4, job_4) -> (6 assets above, only the cycle 'u3')
(schedule_5, job_5) -> (6 assets above, only the cycle 'u4')
I tried to setup this, but no clue how to make the (schedule, job) to run the specific partition per the 6 assets.
the (schedule_x, job_x) always tries to run all the assets and all partitions.
I use the define_asset_job
to define the job, and use the build_schedule_from_partitioned_job
to define the schedule. I checked the config
parameter in the define_asset_job
might help to do this kind of specific partition run, but I tried a lot of different config way and it just doesn’t work.
Hing Liu is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.