There are 3 Spring Boot services (Client, Proxy, API) – Client is calling API and request is sent to/back through proxy, approx 1% of responses are corrupted and client unable to read it – throwing java.io.IOException: Illegal character in chunk size: 123.

This means that response is chunked and every chunk has size.
Since the hex representation of the character “{” is “123” it means that in response chunk size value is missing and data starts with “{“ which is start of JSON object. Client receives header “Transfer-Encoding”: chunked – which is instruction to read the response by chunks, but as the chunk size/ partition size is missing Client unable to read response and throws exception.

Looking into response headers from API, which comes throw Proxy I can see there duplicate header “Transfer-Encoding”. It was found that API itself adding originally such header ONCE, which is normal behavior when response is dynamic (from DB) Spring/Tomcat doesn’t know content length and it chunks the response. But also when response goes through Proxy, then proxy adds one more “Transfer-Encoding” header. Looking in Proxy code – it converts received headers from API and forwards it (all headers are unique, no duplication). But proxy internal Tomcat is adding additionally “Transfer-Encoding” header to response.
It looks like Tomcat bug.
Upgrading Spring Boot/Tomcat version to the latest did not help.

This problem can be reproduced only during load – e.g 300 requests will give around 3-4 errors.
Removing manually “Transfer-Encoding” header (Content-Length will be added automatically) on API side solves the problem, but this doesn’t seem correct solution.

Please share your experience/ suggest a solution.