I have PubSub code as a spring boot application that when run locally on laptop, can consume messages from a topic that I create on a console.

I am using different credentials from the ones that I use when I deploy code to K8. When the pod comes up, it was crashing due to this check failing

2024-05-02T05:58:44.500Z  WARN 1 --- [nio-8080-exec-8] c.g.c.s.a.p.h.PubSubHealthIndicator      : Failed to connect to Pub/Sub APIs. Check your credentials and verify you have proper access to the service.

I added config to disable the default health check by doing management.health.pubsub.enabled=false
I did so because I read on a different random thread that the healthcheck might be failing due to wrong reason and some other bug. So I disabled the check.

The pod does not crash now, but of course the same code that is able to read from same topic when used locally, is not able to pull/poll any messages when running in the K8. The logs do indicate that I have the right permission. I see the below log

: Scopes in use by default credentials: [https://www.googleapis.com/auth/pubsub,

https://www.googleapis.com/auth/spanner.admin, 
https://www.googleapis.com/auth/spanner.data, 
https://www.googleapis.com/auth/datastore, 
https://www.googleapis.com/auth/sqlservice.admin, 
https://www.googleapis.com/auth/devstorage.read_only, 
https://www.googleapis.com/auth/devstorage.read_write, 
https://www.googleapis.com/auth/cloudruntimeconfig, 
https://www.googleapis.com/auth/trace.append, 
https://www.googleapis.com/auth/cloud-platform, 
https://www.googleapis.com/auth/cloud-vision, 
https://www.googleapis.com/auth/bigquery, 
https://www.googleapis.com/auth/monitoring.write]

Not very sure at this point how to even debug this now. Any thoughts will help