I’m currently utilizing Celery for task management within my application, and I’m facing a challenge regarding task distribution to specific workers based on their unique hostnames. In my use case, I’m deploying multiple worker containers for a particular application. These containers are required to connect to various VPNs for specific requests or processing tasks.
The challenge arises from the dynamic nature of container deployment, where each container’s hostname is unique and dynamically assigned and it is stored on database with metadata information to be used by the application to select which worker should run a particular task.
Given this scenario, I’m seeking a solution to send tasks directly to a specific worker based on it’s hostname. This would ensure efficient task distribution, even in dynamic deployment environments where workers may connect to different VPNs dynamically and have different hostname each time.
The logic to select the worker is already in place.
The logic to create container/worker is already in place.
The logic to requeue a task that is sorted to an invalid queue is already in place.
Scenario
Workers
- Worker01 (hostname=a333) connected to VPN 1
- Worker02 (hostname=b999) connected to VPN 2
- Worker03 (hostname=c777) connected to VPN 3
- Worker04 (hostname=c444) connected to VPN 1
Tasks
- Task01 needs to run on Worker01|04 (identified by it’s hostname)
- Task02 needs to run on Worker02 (identified by it’s hostname)
- Task03 needs to run on Worker01|04 (identified by it’s hostname)
- Task04 needs to run on Worker03 (identified by it’s hostname)
I’ve been working on configuring different exchanges and routing keys for my Celery setup, but I seem to be encountering some issues.
Despite my efforts, it appears that something is missing in the configuration.
Thank you in advance for any help!