Question:

I am running InfluxDB on a K3s cluster using the official InfluxDB Docker image, but I am encountering the following error intermittently in the logs:

ts=2024-12-18T08:33:12.981696Z lvl=info msg=Unauthorized log_id=0tY3_WvW000 error="authorization not found"

Details of My Setup:

1. Platform: K3s (lightweight Kubernetes distribution)
2. InfluxDB Deployment:
   • I used a StatefulSet with a Persistent Volume Claim.
   • The container is based on the official InfluxDB Docker image.
   • INFLUXDB_HTTP_AUTH_ENABLED is set to true in the configuration.
3. Token and Authentication:
   • The INFLUXDB_TOKEN is stored in a Kubernetes secret and mounted into the InfluxDB StatefulSet.
   • The same token is used by a Kafka-to-InfluxDB application to write data.
   • The token has read and write permissions for the relevant bucket.
4. Services:
   • The InfluxDB service is exposed internally within the cluster at http://influxdb.<namespace>.svc.cluster.local:8086.
   • The Kafka-to-InfluxDB application uses this URL for communication.

What Works:

• The Kafka-to-InfluxDB application writes data successfully most of the time.
• Manual curl commands using the token work as expected:

  curl -XPOST "http://influxdb.<namespace>.svc.cluster.local:8086/api/v2/write?org=MyOrg&bucket=MyBucket&precision=s" 
  --header "Authorization: Token <my_token>" 
  --data-raw "test_measurement value=1"
  

What Doesn’t Work:

• Periodically, I see the error error="authorization not found" in the InfluxDB logs, and data fails to be written.

What I’ve Tried:

1. Verified the token’s permissions using the Influx CLI:

   influx auth list --org MyOrg
   

   The token has read-buckets, write-buckets, and other relevant permissions.

2. Restarted both the InfluxDB StatefulSet and the Kafka-to-InfluxDB Deployment to ensure fresh secrets are loaded.
3. Increased logging verbosity in InfluxDB by adding -log-level=debug but couldn’t find detailed information about the specific request causing the error.

Questions:

1. What could cause the authorization not found error when the token appears to work in most cases?
2. Is there any known issue with token rotation or caching in the official InfluxDB Docker image that might cause this intermittent behavior?
3. Are there additional configurations or debugging steps I should check to pinpoint the root cause?

Any insights or solutions would be greatly appreciated. Thank you!