I am running Kafka version 3.7 in Kraft mode with 3 controllers and 3 brokers as docker containers (running on separate/distinct hosts).
Relevant controller configurations from one of the controllers:
KAFKA_CFG_PROCESS_ROLES=controller
KAFKA_KRAFT_CLUSTER_ID=K2v8t-ugRYeLRARSmfGh0Q
KAFKA_CFG_NODE_ID=6
[email protected]:9097,[email protected]:9097,[email protected]:9097
KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=CONTROLLER:SSL,INSIDE_SSL:SSL
KAFKA_CFG_LISTENERS=CONTROLLER://10.87.170.6:9097
Relevant broker configuration from one of the brokers:
KAFKA_CFG_CONTROLLER_LISTENER_NAMES=CONTROLLER
[email protected]:9097,[email protected]:9097,[email protected]:9097
KAFKA_CFG_PROCESS_ROLES=broker
KAFKA_CFG_NODE_ID=3
KAFKA_HEAP_OPTS=-Xmx3686m -Xms3686m
KAFKA_CFG_LISTENER_SECURITY_PROTOCOL_MAP=INSIDE:PLAINTEXT,OUTSIDE:PLAINTEXT,INSIDE_SSL:SSL,OUTSIDE_SSL:SSL,CONTROLLER:SSL
KAFKA_CFG_LISTENERS=INSIDE://10.87.170.78:9094,OUTSIDE://10.87.170.78:9092,INSIDE_SSL://10.87.170.78:9093,OUTSIDE_SSL://10.87.170.78:9096
KAFKA_CFG_ADVERTISED_LISTENERS=INSIDE://10.87.170.78:9094,OUTSIDE://10.87.170.78:9092,INSIDE_SSL://10.87.170.78:9093,OUTSIDE_SSL://10.87.170.78:9096
My brokers are unable to connect to controllers with errors:
Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching node-kfkc3 found
and looks like controller does lookup for itself (snippet from controller with id=6):
Caused by: java.security.cert.CertificateException: No subject alternative DNS name matching node-controller6 found.
at java.base/sun.security.util.HostnameChecker.matchDNS(HostnameChecker.java:212)
at java.base/sun.security.util.HostnameChecker.match(HostnameChecker.java:103)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:458)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkIdentity(X509TrustManagerImpl.java:418)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:292)
at java.base/sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:144)
at java.base/sun.security.ssl.CertificateMessage$T13CertificateConsumer.checkServerCerts(CertificateMessage.java:1329)
... 19 more
The certificate does not match alternative name with DNS but since I am using IPs for communication, I allow IP SANs.
Anyone face similar issue? and have tips to disable reverse dns lookup for controllers?
Note:
I have KAFKA_CFG_INTER_BROKER_LISTENER_NAME=INSIDE_SSL
So inter broker communication is on SSL as well but I do not see any issue there.
Issue is only with the controllers