I am working at a retailer with a deep catalog of products and we are taking stock of our architecture.

Currently, we use microservices, but all product information goes to a service that is getting bigger and bigger over time. It continuously ingests delta update feeds from the source system where all product information is maintained, and also receives a full feed every night. This is ingested and updates a database, and this database is then used directly (other microservices reading from the database) or indirectly (other microservices calling a product API which reads from the database) from a large and growing number of services handling other concerns – product feeds in various formats, integrations with other systems.

We are looking to change this since microservices shouldn’t read from each other’s databases, but the trouble I’m running into is the practical implications of the other solutions. There are above 200K products with many more coming, and the changed information needs to make its way to the other services quickly.

(I will use the term “downstream service” here; that’s just to say that it will feed off of the events produced by the catalog service. The idea is for them to be as decoupled as they can, but for them to need to consume and maintain their own database of basic information about a product (a different set for each service) at some point as part of doing what they do; at the very least they’d need to know that “part number 123” now exists or has gone up for sale, for instance.)

Problem A, data flow: The microservices approach to solving this in a decoupled way is to post the new information as an event on an event bus, which would then need to be a series of events. To keep the catalog ingesting service decoupled, it should do what it does being ignorant of what downstream services already know. Which means that every night it would post 200K messages about the products being updated, and everything downstream would have to process this. I buy into the idea of separation and every service owning the part that matters to it; this just sounds a bit heavy.

Problem B, bootstrapping: When a new downstream service comes online, how does it get access to the full catalog? Let’s say that in addition to posting messages, at the time of the full update, it also serializes a full dump of all products as known to it on a blob storage somewhere, and the downstream service can fetch it as a shortcut for bootstrapping.

If a new service comes online at 11 AM and does everything right: recognizes that it is starting from scratch, immediately starts listening to updates on the event bus, puts them on an internal queue, grabs the full dump first, ingests them and then ingests everything that came in from the event bus, it would still be ignorant of the updates that happened between the time the full dump was produced (let’s say at 6 AM) and the events that came in after it started listening at 11 AM.


The best workaround I am currently considering is allowing the downstream service to start listening, and also calling the catalog service and requesting a special bootstrapping dump snapshot to be written with the current data, which is then delivered via the claim check pattern through the event bus. But this does have them tightly coupled to each other, and also ties the catalog service up with doing a lot of work on behalf of the downstream service.

Problem C, architectural purity: If everything depends on downstream services ingesting a huge file produced by the catalog service, is that really less coupled? It’s asynchronous and won’t block the source system, sure. And we have separated the domain model used from the one used in the database, sure. But is this really the best we can do?

So the question is: What are appropriate solutions to these problem that minimize the issues and do not require us to “cut against the grain” of microservices?

I don’t think I’m the first to have these thoughts and issues, but I’m having a lot of trouble finding the material that no doubt exists about these kinds of integration issues. This is the first project I’m working on that is trying to fully embrace microservices, and I’m still lacking for experience in all areas, and know which “old” solutions to reach for but not the “new”. If there are any books or materials that are entirely about going into problems like this, I would be happy to hear about them.