I am using Objectify, a Java data access API designed for Google Cloud Datastore, in my Java application running on Cloud Run. Over the past week, I have encountered around 400 occurrences of the following exception during random read operations for different entities.

com.google.cloud.datastore.DatastoreException: Non-protobuf error
That’s an error.The server encountered a temporary error and could not complete your request.
Please try again in 30 seconds.That’s all we know.HTTP status code was 502.

Observations:

1. The error occurs during random (mostly read) queries.
2. The frequency of the error has increased the last couple of weeks, with around 400 occurrences in the past week (in a total of around a million http requests).
3. Over the past months I had random occurrences of this error but in way lower numbers (less than 10 per month).
4. I don’t think I am exceeding any quotas (even during peak times, the application handles around 150 queries per second, based on the monitoring dashboards provided by Google.).

Google suggests to implement a retry logic (I have a retry mechanism with exponential backoff for my async google tasks handling exceptions like this one, but this is not something I can implement for all the customer facing APIs).

Any insights or suggestions would be greatly appreciated.
Thank you!