We migrated one of our services from CloudFuncitons to GCP CloudRun and started getting very weird errors during the first request to BigQuery API.
There are several typical errors:
com.google.cloud.bigquery.BigQueryException: Error writing request body to server
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:115)
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.create(HttpBigQueryRpc.java:236)
at com.google.cloud.bigquery.BigQueryImpl$5.call(BigQueryImpl.java:414)
at com.google.cloud.bigquery.BigQueryImpl$5.call(BigQueryImpl.java:399)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
at com.google.cloud.bigquery.BigQueryRetryHelper.run(BigQueryRetryHelper.java:86)
at com.google.cloud.bigquery.BigQueryRetryHelper.runWithRetries(BigQueryRetryHelper.java:49)
at com.google.cloud.bigquery.BigQueryImpl.create(BigQueryImpl.java:398)
at com.google.cloud.bigquery.BigQueryImpl.create(BigQueryImpl.java:363)
com.google.cloud.bigquery.BigQueryException: Unexpected end of file from server
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.translate(HttpBigQueryRpc.java:115)
at com.google.cloud.bigquery.spi.v2.HttpBigQueryRpc.create(HttpBigQueryRpc.java:236)
at com.google.cloud.bigquery.BigQueryImpl$5.call(BigQueryImpl.java:410)
at com.google.cloud.bigquery.BigQueryImpl$5.call(BigQueryImpl.java:399)
at com.google.api.gax.retrying.DirectRetryingExecutor.submit(DirectRetryingExecutor.java:103)
at com.google.cloud.bigquery.BigQueryRetryHelper.run(BigQueryRetryHelper.java:86)
at com.google.cloud.bigquery.BigQueryRetryHelper.runWithRetries(BigQueryRetryHelper.java:49)
at com.google.cloud.bigquery.BigQueryImpl.create(BigQueryImpl.java:398)
at com.google.cloud.bigquery.BigQueryImpl.create(BigQueryImpl.java:363)
Errors disappear after several minutes/attempts of calling BigQuery and all work well until the next restart/redeploy of the service.
We introduced Retry mechnanism using external retry library as suggested here https://cloud.google.com/bigquery/docs/error-messages#connecterrors
But maybe there is another approach to avoid this service “startup” problems?
Do you know what can be the problem?
Thanks in advance!