I’m quite new to Celery and distributed task queue, I’m looking for some suggestion in regards best approach to utilize.
Concept:

• as input I have lots of text file to process, divided in “slot” or folder
• each folder will be uploaded in a Minio (or equivalent object store)
• I will require to distribute task, potentially using Celery and python executor (I will require to perform some NER detection using Spacy)

This is how I think should be:

How can I add “sensor” to let Celery worker being triggered when new file/folder is uploaded?
Is there any best alternative?

Many thx in advance for any suggestion