I am dealing with a scenario where, due to a misconfigured Spring profile value in my app, it started calling the dev upstream endpoint to validate the prod token received from its downstream using its prod credentials. As expected, this led to failures.

downstream --> my app --> upstream

The downstream app retains failed requests and retries them, along with new requests. Consequently, the number of requests keeps increasing. As these requests continue to fail with response code 401 due to credentials from a different environment, this leads to a kind of infinite loop. Eventually, the upstream starts rejecting requests by throwing exceptions with status code 429 on the requests (my app is using an upstream app as a dependency).

While I understand that monitoring/alerting can help identify and manually fix this issue, I want to avoid any resource overload on the server during such incidents.

I believe a circuit breaker can be useful here. After a certain number of 401 responses from the upstream, the circuit can open.

I need suggestions on the following points:

1. Is using a circuit breaker the correct approach to handle this scenario?
2. Should the circuit still open when credentials and environment configurations are correct and pointing to correct upstream but the token received from my downstream is not valid (occasionally)? In this case, I receive a status code 200 from my upstream with {status: "INVALID"} in the response body.
3. In the scenario where the circuit breaker opens due to incorrect Spring profile value in my app and 401 responses from the upstream, which response to the downstream is more appropriate from the fallback method – 401, 500, or 429 (too many requests)? I am unsure about using 401 since it’s not the downstream that is sending an incorrect token; it’s my app config validating the correct token with the correct client credentials against the wrong environment.

Thanks,