I am developing a multi-user document storage system with the following constraints:
- The order of saving documents must be sequential per user.
- I cannot save documents for the same user in parallel.
- Ideally, different users should be able to save in parallel.
- My architecture is based on NodeJS and Google Cloud.
Currently, I am using Google Cloud Tasks. I have created a number of queues and distributed users among these queues. This approach ensures sequential processing for each user, but if one user enqueues many tasks, it penalizes other users in the same queue.
I would like to find a solution that allows Cloud Tasks to dequeue considering not only the order of entry but also the load of tasks per user, something like a round-robin approach. Is there a way to achieve this with Google Cloud Tasks?
Alternatively, I have seen that BullMQ Pro has the concept of groups (https://docs.bullmq.io/bullmq-pro/groups), which seems to be exactly what I need. However, I would like to know if there is an equivalent solution on Google Cloud.
Thank you in advance for your help!
I think Cloud Task is not the right service for this. As you describe your use case, I would say that PubSub with an ordering key would help you to deliver messages from the same ordering key (i.e. the user ID) in sequence: the next message is delivered only when the previous one is ack. But multiple keys can be processed in parallel