Consider a software system that consists of multiple web services (perhaps microservices) communicating with one another. In the course of a certain use case, a client sends a POST web request to an external service A, which in turn needs to send a POST web request to an internal service B.

In case service B is currently unavailable and, e.g., returns HTTP 503, service A will perform internal retries for some time to try and cope with the outage of its dependency.

Now, if the retries don’t fix the problem, what is the best practice for service A? Should it also return 503 to indicate unavailability? Or perhaps 502, as it “acted as a gateway”?

And what should the client do? This might easily cause a retry cascade. OTOH, many HTTP client libraries (e.g., Microsoft.Extensions.Http.Resilience) will retry on any 5xx error, so maybe it doesn’t really matter anyway?

I hope this question won’t be judged as being opinion-based – surely, with service-oriented and microservices architectures having been around for some time, there must be some standard patterns for this common scenario?

(Of course, service A should ideally not synchronously depend on service B – asynchronous messaging is king with regards to resilience. However, let’s just assume that this is a synchronous use case and it’s not feasible to remodel it as an asynchronous one. And as it’s POST requests with desired side effects, we cannot just throw caching or data replication at the problem, either.)