I already have a functioning Kafka KRaft cluster with mTLS. Now I’m trying to setup a cluster with SASL_SSL but I get the following error on the broker listener:

INFO [SocketServer listenerType=BROKER, nodeId=1] Failed authentication with /172.31.0.228 (channelId=172.31.21.224:9094-172.31.0.228:48788-0) (Unexpected Kafka request of type METADATA during SASL handshake.) (org.apache.kafka.common.network.Selector)

The cluster consists of three nodes, broker/controller combined. No consumers/producers are connected yet, the error is triggered by inter-broker communication. What’s interesting is the controller listener seems to be fine.

I tried many things already, but believe this config should be very close to what it should be. I’ve read that the error could point to a mismatch in protocol, but everything is configured to use SASL_SSL. The error is also not very descriptive and I am not sure how to properly troubleshoot.

# server.properties configuration

# Listeners
listeners=CONTROLLER://0.0.0.0:9093,BROKER://0.0.0.0:9094
advertised.listeners=BROKER://<BROKER_HOST>:9094
listener.security.protocol.map=CONTROLLER:SASL_SSL,BROKER:SASL_SSL

controller.listener.names=CONTROLLER
inter.broker.listener.name=BROKER

# SASL
sasl.enabled.mechanisms=PLAIN
sasl.mechanism.controller.protocol=PLAIN
sasl.mechanism.inter.broker.protocol=PLAIN

# JAAS
listener.name.controller.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required 
    username="admin" 
    password="test123" 
    user_admin="test123";

listener.name.broker.plain.sasl.jaas.config=org.apache.kafka.common.security.plain.PlainLoginModule required 
    username="admin" 
    password="test123" 
    user_admin="test123";

# SSL
security.protocol = SASL_SSL
ssl.enabled.protocols=TLSv1.3,TLSv1.2
ssl.keystore.type=PEM
ssl.keystore.location=/opt/kafka/certs/keystore.pem
ssl.key.password=<password>
ssl.truststore.type=PEM
ssl.truststore.location=/opt/kafka/certs/truststore.pem