I’m doing functional testing and using Luigi to run a monthly rollup task. There is separate container for performing tasks and one container for database. This task only runs its dependent tasks, which are 30 daily rollup tasks. These daily rollup tasks run 24 hourly tasks.

The daily rollup task fails sometimes, both in the pipeline and locally. Whenever it fails, it throws this error:

taskrunner: Traceback (most recent call last):
taskrunner:   File "/usr/lib/python3.11/site-packages/luigi/worker.py", line 188, in run
taskrunner:     raise RuntimeError('Unfulfilled %s at run time: %s' % (deps, ', '.join(missing)))
taskrunner: RuntimeError: Unfulfilled dependencies at run time:.........

I can’t share the complete error. 🙁

The error indicates that the task failed due to an unfulfilled dependency. However, all hourly rollup tasks are completed with the status as DONE. This issue does not occur every time; sometimes it passes.