I have a php process producer-a which places a job on RabbitMQ a – deferring processing to a third-party service. The 3rd party service completes processing a and places a response onto queue b. I have a listener consumer-b which picks up jobs on queue b to continue processing.
I need to ensure that producer-a has completed it’s process entirely before the listener consumer-b picks up the job to continue processing.
The problem which I am facing is that by using RMQ – I am asynchronously performing tasks – however I need to serialise this process whilst still using RMQ.
In case the above doesn’t make sense:
Example timeline if serialised:
Process A ------> Process B ------> Process C
Example timeline with message queue
Internal Process Process A----------------------------------|
Deferred Process Process B -----|
Internal Process |Process C---|
Process A needs to complete fully before Process C. The problem is that Process B is deferred to a 3rd party service via a message queue, and I need to ensure that Process A which sent the job to Process B completed before Process C picks up the job from Process A. As can be seen from the timeline above, Process C has completed before Process A has completed.
Update:
I didn’t think to state the obvious but seem to be getting obvious suggestions / solutions. producer-a simply needs to send a TCP message to Loggly (sometimes takes over 2s) and also needs to update the database to advise that the process was deferred. Sometimes, the message comes back and is picked up by Process C before Process A has completed.
2
What you have is a scatter-gather problem.
Process A has to scatter his (possibly intermediate) results between Process B and the French Foreign Legion. Process B must gather the work products from Process A and the French Foreign Legion, and cannot proceed until he has both of them.
Presumably, RabbitMQ has a capability to wait on multiple queues simultaneously. Establish a third queue, for completion notifications from Process A to Process B. Process B then waits on both queues, and, using magic cookies embedded in the messages, matches the French Foreign Legion dispatches to the Process A completions. When it has both pieces of the golden ticket, Process B continues. If it only has one piece of the cookie, it continues waiting on both queues.
Let Producer-A put some state into DB or some other persistent store and then Consumer-B can look at that state and determine whether to pick up the job from Producer-A or not.
Another option is, since you are using a message queue, let Producer-A send another message upon its completion into queue C. Consumber-B should not pick up any message from queue B until it recieves message from queue C.
If you need producer a to have finished before consumer b starts processing then either:
- producer a should add the message to queue a as the last thing it does
- producer a should add a message to another queue as the last thing it does and consumer b should wait either for that flag message before it picks up the main message to process or else pick up the main message to process and then wait for the confirmation flag message on the other queue.