I am running Milvus 2.3.11 and using bitnami kafka for my log broker.

Due to many critical vulnerabilities in bitnami/kafka:3.1.0-debian-10-r52, I upgraded kafka to 3.7.0-debian-12-r6
After the upgrade datanode started throwing the following errors

FAIL|rdkafka#consumer-6| [thrd:GroupCoordinator]: GroupCoordinator: milvus-kafka-0.milvus-kafka-headless.milvus-database.svc.cluster.local:9092: Failed to │ │  resolve 'milvus-kafka-0.milvus-kafka-headless.milvus-database.svc.cluster.local:9092': Name or service not known (after 2ms in state CONNECT, 1 identical error(s) suppressed)

I believe it is not able to connect to kafka. I ran nslookup milvus-kafka-0.milvus-kafka-headless.milvus-database.svc.cluster.local on a pod in the same namespace and it was able to resolve the host. But I have not idea why my datanode cannot connect to kafka.

I reverted back to the previous deployment but I keep getting the same error. I have tried restarting both kafka and datanode pods but I get the same errors.

Now, I am not sure if the error started before my kafka upgrade but I noticed it after the upgrade.

Thanks in advance for your help.