I am using Hazelcast 5.3 and have set up a Hazelcast cluster with multiple nodes. I am using an IMap to distribute some data between my microservices. This IMap is backed by a MapStore with a write-behind configuration (write-delay-seconds greater than 0), but I assumed this wouldn’t affect the in-memory visibility of the data (or does it?). I experience stale date from time to time.

Here is the issue I’m facing:

From time to time, a scheduler in microservice A puts some data into the map and then calls microservice B. Microservice B should be able to see the data that was put into the map by microservice A before it was called. Sometimes this works as expected, but other times microservice B doesn’t see the new data and encounters stale data instead.

My understanding is that the write operation is performed on the partition key owner, and any subsequent reads should be directed to the same partition owner, ensuring the data read is always fresh. However, the observed behavior suggests otherwise.

Questions:

Is my understanding of how IMap works correct?
How can I achieve strong consistency in this scenario to ensure that microservice B always sees the fresh data put by microservice A?
Additional Information:

Any insights or solutions to ensure consistent data visibility would be greatly appreciated!