I am tasked with creating a development, design and architecture guide for a large multi-year project. I have to dictate best design practices for a number of architectural perspectives. See the below diagram to demonstrate how I see this looking.
So based on other architectural mandate, MQ based asynchronous services are preferred because of guaranteed message delivery and because not every business critical component guarantees to be High Availability. My experience in developing against MQ interfaces was rather low scale and not in a transactional COA design. Typically I would develop multiple threads that would routinely check for new messages to process, both messages to push into a queue for requests, temporarily persisting the state, then another thread that will consume responses from another queue.
When considering a COA style architecture that has a clearly defined Data Access layer and Business Logic layer, how is asynchronous data sources represented or implemented here typically? It generally makes sense for a Data Access layer to provide an interface to Business Logic where the data source is a REST or SOAP based web service. These services are synchronous, they receive a request and immediately process it, then return a response while the client waits a predetermined amount of time to recieve it.
Asynchronous requests may not be processed right away, they also might not return a satisfactory response. MQ only guarantees the target component will eventually receive the request message, it does not guarantee it will return a response.
I feel like the best way to handle this is to have two seperate interfaces required by the Data Access layer, request interface and response interface. A scheduled batch job that is then using the Business Logic component might be able to deal with this effectively using multiple threads, however there may be another component that requires a real time synchronous interface of our Business Logic component where we are required to interface through MQ that is NOT real time.
I am confused in how to reconcile the two. Is this the right design pattern here? How can I effectively provide an interface to a component that requires real time when my component requires an asynchronous interface?
NOTES:
Some relevant articles I have been reading:
-
https://www.ibm.com/developerworks/community/blogs/WebSphere_Process_Server/entry/sync_over_async_switch?lang=en
-
http://en.wikipedia.org/wiki/Request%E2%80%93response
-
Link
It seems this situation is defined already as an Architectural Anti-Pattern called Sync over Async. It is clear to me that reading this people are saying it is an extraordinarily bad idea. Then others note that sometimes we do have no choice and we have to do this which I am finding myself in. I am not in a position to ask for a synchronous interface to lookup and search services for this one system, and I have no say with the others components requirement for a synchronous lookup and search service.
Maybe you can design your interface between business and data layer to be async? One component of the data layer would access the underlying ‘asyncness’ of MQ, the other component would be and async-over-sync accessing your Database through JDBC
I’m assuming your security layer is wrapped within the Generic Data Access component. The first thought that came to my mind when I saw your design was “why are trusted systems accessing the data layer in the same manner that untrusted systems are?”
The design you’re laying out looks like something an insurance, financial, or even retail environment would want to use.
Your trusted layer, which is the batch and real-time synchronous components, is generally used for the create, update, and delete operations that can affect the underlying data.
Your untrusted layer, which comprises the widgets, is generally restricted to report type operations.
As you call out, the complexity in the design is that you have requests coming in through synchronous and asynchronous methods. One benefit of using Websphere MQ as an intermediary for your asynchronous calls is that the WMQ clients can listen for specific messages on a return queue through a correlation ID. Other lighter-weight MQ servers generally don’t provide that feature.
Another nice aspect of WMQ is that it has a really robust dead letter queue handling system. Inevitably, you’re going to get responses that are delivered late. The DLQ system will help you track those and understand why they are happening.
I think you’re right to be worried about how the widgets are interfacing with the Generic Data Access component. The primary challenge is you’re setting yourself up for custom interfaces for each widget. It may be manageable now, but experience and hindsight says that there will be more and more and more of them.
To simplify things at the data access layer, I’d recommend having all widget requests be run through a WMQ based layer. All incoming requests are put on a queue, and responses are placed on a reply queue with a correlation ID.
The advantage here is that your Generic Data Access component only has 2 means of access. Trusted systems get direct access to the layer. Everybody else goes through the WMQ based queues. An advantage of this is it’s trivial to keep adding WMQ clients that can push requests on behalf of future widgets.
But wait! you might say. There are widgets that expect synchronous responses and running them through a queue will affect that. That’s true, but you’re going to have to deal with that communication protocol mismatch at some point in your design somewhere.
I recommend putting that impedance mismatch within your MQ clients as it keeps the logic within the Generic Data Access component simplified. Said another way, the data access layer isn’t responsible for handling the impedance mismatch. It’s the responsibility of the client performing the access.
While this may appear to complicate the API for the synchronous clients, it actually exposes something that may have otherwise been hidden. _Somewhere_ in your data access pathway, something could have gotten disconnected and the upstream clients need to be able to handle a timeout / no-response. By using an MQ client and explicitly stating within the API “this will timeout in 1 minute” then the upstream client has to acknowledge that situation and handle it accordingly.
As an aside, using a WMQ client for protocol mismatches is a pretty good way to go. All of the rectification code is encapsulated within a particular layer, and your ability to reuse that layer increases due to the abstraction it provides you. And even if you can’t completely re-use the WMQ code due to differing API calls, you can still re-use the communication protocol rectification code.
And as a final consideration, using a WMQ layer for all widget access makes it easier to scale up your data access layer’s response time. As the volume of requests increases, you can always add more queue handler threads within the data access layer.
To summarize…
Make all of the widgets go through a Websphere MQ queue based access layer in order to request data from your central store. Doing so abstracts out the external interface to your data access layer, and the decoupling that provides will make it easier for you to adjust to future changes.