I have been coding a python based data extractor from an API for a web application. This web application uses RBMQ as a broker to pass batch details to the extractor, which fetches data to pass back to RBMQ (to decide the destination). Although I have successfully developed the extractor, my current task is to integrate it with RBMQ. However, I am now faced with new problems to solve:
- Integrating the extractor with asynchronous behavior.
- Ensuring the extractor can handle multiple batches (assuming that this can be solved with multiprocessing).
Someone, as amateur as myself, suggested using Celery to implement asynchronous and multiprocessing behavior. I have tried to read some documentation on Celery, but I’m having trouble making sense of it. For me, Celery seems a bit steep on the learning curve. I would appreciate some advice on how to do it. Are there better resources to learn it? or altogether Are there any better solutions available?