Edit:
OK, so, people said it is unclear what I am asking. I am asking for feedback on this design. Here is an example user story:
As a group admin on the website I want to be notified when a user in my group uploads a file to the group.
Easiest solution would be that in the code handling the upload, we just directly create an email message in there and send it. However, this seems like it isn’t really the appropriate level of separation of concerns, so instead we are thinking to have a separate worker process which does nothing but send notifications. So, the website in the upload code handles receiving the file, extracting some metadata from it (like filename) and writing this to the database. As soon as it is done handling the file upload it then does two things: Writes the details of the notification to be sent (such as subject, filename, etc…) to a dedicated “notification” table and also creates a message in a queue which the notification sending worker process monitors. The entire sequence is shown in the diagram below.
My questions are: Do you see any drawbacks in this design? Is there a better design? The team wants to use Azure Worker Roles, Queues and Table storage. Is it the right call to use these components or is this design unnecessarily complex? Quality attribute requirements are that it is easy to code, easy to maintain, easy to debug at runtime, auditable (history is available of when notifications were sent, etc…), monitor-able. Any other quality attributes you think we should be designing for?
Original:
We are creating a cloud application (in Azure) in which there are at least 2 components. The first is the “source” component (for example a UI / website) in which some action happens or some condition is met that triggers a second component or “worker” to perform some job. These jobs have details or metadata associated with them which we plan to store in Azure Table Storage. Here is the pattern we are considering:
Steps:
- Condition for job met.
- Source writes job details to table.
- Source puts job in queue.
Asynchronously:
- Worker accepts job from queue.
- Worker Records DateTimeStarted in table.
- Queue marks job marked as “in progress”.
- Worker performs job.
- Worker updates table with details (including DateTimeCompleted).
- Worker reports completion to queue.
- Job deleted from queue.
Please comment and let me know if I have this right, or if there is some better pattern. For example sake, consider the work to be “sending a notification” such as an email whose template fields are filled from the “details” mentioned in the pattern.
1
I can’t answer whether you should do this or not, but I can try to help with your decision-making process.
First, what you are proposing is a very standard mechanism for asynchronously processing data. So, you are doing something very conventional, and that is good. I use AWS and have done this a lot with SQS and SNS. I can’t speak to what Azure offers. AWS gives an example of this pattern for batch processing.
Here are some advantages to this approach:
-
The initial request can be faster because there is less work being done. There is some time for putting the message in the queue, but it is probably faster than saving to disk and sending an email.
-
You can make the notifications more fault tolerant, or at the least, not interfere with the initial request which prompted that notification. If your email service goes down for a day, you can keep the message in the queue and the worker can try it again the next day.
-
It will scale well – both in terms of users, and as new features arise.
There are a few downsides:
-
There is now additional complexity. It would be simpler to just send the email.
-
Correctly handling errors can more complicated. What do you do if you can’t send an email? Retry? Delete the message from the queue and forget? This error handling is important, but your use-case decides how sophisticated you need it to be.
-
You probably want another web service handling this, which adds another component. Though, on small-scale projects, I have put the worker thread in the same web service.
Separation of Concerns
You mentioned Separation of Concerns as one reason for your design. Regardless of how you end up architecting this, you should still separate concerns at a class level. Whatever class is receiving this data can call an “notifier” interface. You might have different implementations of this over time: send the email in the same thread, send the email in a different thread without a queue, put the email in a queue. Each time you change your architecture, you keep the servicing code the same.
3