I have installed confluent schema registry based on “https://packages.confluent.io/archive/7.6/confluent-7.6.0.zip” on my on-premise
Kubernetes cluster,and pointing it to _schemas topic in Confluent Cloud Kafka.
When I run a single pod instance of schema registry (SR), I am able to publish new schemas to the SR consistently.
But when I run more than one pod instance, and the request hits the follower instance,the request fails with the error message like below,complaining about the follower instance unable to forward the request to leader due to “Unknown host” exception.
Error logs:
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:883)ntat org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1034)ntat java.base/java.lang.Thread.run(Thread.java:840)nCaused by: io.confluent.kafka.schemaregistry.exceptions.SchemaRegistryRequestForwardingException: Unexpected error while forwarding the registering schema request to https://dp-schema-registry-7dc6cbdb5-kr2rk:1443ntat io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.forwardRegisterRequestToLeader(KafkaSchemaRegistry.java:1248)ntat io.confluent.kafka.schemaregistry.storage.KafkaSchemaRegistry.registerOrForward(KafkaSchemaRegistry.java:880)ntat io.confluent.kafka.schemaregistry.rest.resources.SubjectVersionsResource.register(SubjectVersionsResource.java:412)nt… 73 morenCaused by: java.net.UnknownHostException: dp-schema-registry-7dc6cbdb5-kr2rk
Basically the pods corresponding to the different SR instances in the same cluster are trying to reach each other using pod name and hence unable to resolve the target pod.If I manually try to connect to one pod from the other using the pod IP,they are able to connect.
My question is how do I let the pods interact with each other using pod Ip instead of pod name? I tried setting SCHEMA_REGISTRY_HOST_NAME to status.podIP in env variable section of pod,and it is able to set the variable to pod IP,but looks like the SR I is unable to understand this property and assign it to the pod host names.
Below are some of my settings in the schema registry properties:
listeners–> I have set it to “https://0.0.0.0:1443”
host.name → I have not set this
Please suggest how do I let me follower pod connect/forward to leader pod of SR in above scenario.