In short
To repeat the issue, you may clone the project from:
https://github.com/ethanalef/event-driven-model
Then build both listener and publisher projects with maven, build the image with Dockerfile insider each project then run docker-compose.yml.
You may find the failed logs in listener project.
Background
I have a simple spring boot app running with rabbitMQ successfully within a container.
Then I add Kafka setting in the docker-compose.yml; and application.properties for the app such that the app will connect to both rabbitMQ and Kafka within the same container.
However, the app cannot connected to Kafka:
<code>org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-myId-1, groupId=myId] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Node may not be available.
<code>org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-myId-1, groupId=myId] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Node may not be available.
</code>
org.apache.kafka.clients.NetworkClient : [Consumer clientId=consumer-myId-1, groupId=myId] Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Node may not be available.
While the Kafka log shows:
<code>2024-12-16 17:38:08 [2024-12-16 09:38:08,360] INFO [Controller id=1, targetBrokerId=1] Node 1 disconnected. (org.apache.kafka.clients.NetworkClient)
2024-12-16 17:38:08 [2024-12-16 09:38:08,361] WARN [Controller id=1, targetBrokerId=1] Connection to node 1 (kafka/172.18.0.4:9092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
2024-12-16 17:38:08 [2024-12-16 09:38:08,362] WARN [RequestSendThread controllerId=1] Controller 1's connection to broker kafka:9092 (id: 1 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
2024-12-16 17:38:08 java.io.IOException: Connection to kafka:9092 (id: 1 rack: null) failed.
2024-12-16 17:38:08 at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:71)
2024-12-16 17:38:08 at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:299)
2024-12-16 17:38:08 at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:252)
2024-12-16 17:38:08 at org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:135)
<code>2024-12-16 17:38:08 [2024-12-16 09:38:08,360] INFO [Controller id=1, targetBrokerId=1] Node 1 disconnected. (org.apache.kafka.clients.NetworkClient)
2024-12-16 17:38:08 [2024-12-16 09:38:08,361] WARN [Controller id=1, targetBrokerId=1] Connection to node 1 (kafka/172.18.0.4:9092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
2024-12-16 17:38:08 [2024-12-16 09:38:08,362] WARN [RequestSendThread controllerId=1] Controller 1's connection to broker kafka:9092 (id: 1 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
2024-12-16 17:38:08 java.io.IOException: Connection to kafka:9092 (id: 1 rack: null) failed.
2024-12-16 17:38:08 at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:71)
2024-12-16 17:38:08 at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:299)
2024-12-16 17:38:08 at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:252)
2024-12-16 17:38:08 at org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:135)
</code>
2024-12-16 17:38:08 [2024-12-16 09:38:08,360] INFO [Controller id=1, targetBrokerId=1] Node 1 disconnected. (org.apache.kafka.clients.NetworkClient)
2024-12-16 17:38:08 [2024-12-16 09:38:08,361] WARN [Controller id=1, targetBrokerId=1] Connection to node 1 (kafka/172.18.0.4:9092) could not be established. Node may not be available. (org.apache.kafka.clients.NetworkClient)
2024-12-16 17:38:08 [2024-12-16 09:38:08,362] WARN [RequestSendThread controllerId=1] Controller 1's connection to broker kafka:9092 (id: 1 rack: null) was unsuccessful (kafka.controller.RequestSendThread)
2024-12-16 17:38:08 java.io.IOException: Connection to kafka:9092 (id: 1 rack: null) failed.
2024-12-16 17:38:08 at org.apache.kafka.clients.NetworkClientUtils.awaitReady(NetworkClientUtils.java:71)
2024-12-16 17:38:08 at kafka.controller.RequestSendThread.brokerReady(ControllerChannelManager.scala:299)
2024-12-16 17:38:08 at kafka.controller.RequestSendThread.doWork(ControllerChannelManager.scala:252)
2024-12-16 17:38:08 at org.apache.kafka.server.util.ShutdownableThread.run(ShutdownableThread.java:135)
application.properties:
<code>spring.kafka.bootstrap-servers=localhost:9092
spring.kafka.consumer.group-id=orderGroup
spring.kafka.consumer.auto-offset-reset=earliest
<code>spring.kafka.bootstrap-servers=localhost:9092
spring.kafka.consumer.group-id=orderGroup
spring.kafka.consumer.auto-offset-reset=earliest
</code>
spring.kafka.bootstrap-servers=localhost:9092
spring.kafka.consumer.group-id=orderGroup
spring.kafka.consumer.auto-offset-reset=earliest
Dockerfile:
<code>FROM openjdk:21-jdk-slim
COPY target/listener-0.0.1-SNAPSHOT.jar app.jar
ENTRYPOINT ["java","-jar","/app.jar"]
<code>FROM openjdk:21-jdk-slim
VOLUME /tmp
COPY target/listener-0.0.1-SNAPSHOT.jar app.jar
ENTRYPOINT ["java","-jar","/app.jar"]
</code>
FROM openjdk:21-jdk-slim
VOLUME /tmp
COPY target/listener-0.0.1-SNAPSHOT.jar app.jar
ENTRYPOINT ["java","-jar","/app.jar"]
docker-compose.yml:
image: rabbitmq:3-management
test: ["CMD", "rabbitmq-diagnostics", "check_port_connectivity"]
image: confluentinc/cp-zookeeper:latest
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
test: [ "CMD", "echo", "ruok", "|", "nc", "localhost", "2181", "|", "grep", "imok" ]
image: confluentinc/cp-kafka:latest
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
test: [ "CMD", "kafka-broker-api-versions", "--bootstrap-server", "localhost:9092" ]
condition: service_healthy
condition: service_healthy
<code>version: '3'
services:
rabbitmq:
image: rabbitmq:3-management
ports:
- "5672:5672"
- "15672:15672"
healthcheck:
test: ["CMD", "rabbitmq-diagnostics", "check_port_connectivity"]
interval: 5s
timeout: 5s
retries: 5
zookeeper:
image: confluentinc/cp-zookeeper:latest
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
healthcheck:
test: [ "CMD", "echo", "ruok", "|", "nc", "localhost", "2181", "|", "grep", "imok" ]
interval: 10s
timeout: 5s
retries: 5
kafka:
image: confluentinc/cp-kafka:latest
depends_on:
- zookeeper
ports:
- "9092:9092"
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
healthcheck:
test: [ "CMD", "kafka-broker-api-versions", "--bootstrap-server", "localhost:9092" ]
interval: 10s
timeout: 5s
retries: 5
listener:
build: ./listener
ports:
- "8081:8081"
depends_on:
rabbitmq:
condition: service_healthy
kafka:
condition: service_healthy
restart: unless-stopped
</code>
version: '3'
services:
rabbitmq:
image: rabbitmq:3-management
ports:
- "5672:5672"
- "15672:15672"
healthcheck:
test: ["CMD", "rabbitmq-diagnostics", "check_port_connectivity"]
interval: 5s
timeout: 5s
retries: 5
zookeeper:
image: confluentinc/cp-zookeeper:latest
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
healthcheck:
test: [ "CMD", "echo", "ruok", "|", "nc", "localhost", "2181", "|", "grep", "imok" ]
interval: 10s
timeout: 5s
retries: 5
kafka:
image: confluentinc/cp-kafka:latest
depends_on:
- zookeeper
ports:
- "9092:9092"
environment:
KAFKA_BROKER_ID: 1
KAFKA_ZOOKEEPER_CONNECT: zookeeper:2181
KAFKA_ADVERTISED_LISTENERS: PLAINTEXT://kafka:9092
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: PLAINTEXT:PLAINTEXT
KAFKA_INTER_BROKER_LISTENER_NAME: PLAINTEXT
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
healthcheck:
test: [ "CMD", "kafka-broker-api-versions", "--bootstrap-server", "localhost:9092" ]
interval: 10s
timeout: 5s
retries: 5
listener:
build: ./listener
ports:
- "8081:8081"
depends_on:
rabbitmq:
condition: service_healthy
kafka:
condition: service_healthy
restart: unless-stopped
I cannot identify what is wrong so please help.
I try to revise application.properties and docker-compose.yml with AI like copilot, cursor etc and I expect it should work like rabbitMQ. But no hints at the moment.