We are using kakfa for data exchange bitween micro-services for nest-js and java we have api-gateway in nestjs and user service in java, we are facing issue with kafka disconnecting after few days. what is the issue.
ERROR [ServerKafka] ERROR [Runner] The coordinator is not aware of this member, re-joining the group {"timestamp":"2024-07-25T10:45:02.565Z","logger":"kafkajs","groupId":"flights-consumer-server","memberId":"nestjs-consumer-server-3fc36120-7844-4785-9211-89fe2647291e","error":"The coordinator is not aware of this member"}
Here is my docker compose yaml file for kafka service.
services:
kafka-broker-1:
image: confluentinc/cp-kafka:latest
container_name: kafka-broker-1
environment:
KAFKA_BROKER_ID: 1
KAFKA_CONTROLLER_LISTENER_NAMES: CONTROLLER
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: "INTERNAL:PLAINTEXT,PLAINTEXT:PLAINTEXT,EXTERNAL:PLAINTEXT,CONTROLLER:PLAINTEXT"
KAFKA_LISTENERS: "INTERNAL://:19092,EXTERNAL://:9092,CONTROLLER://kafka-broker-1:9093"
KAFKA_INTER_BROKER_LISTENER_NAME: INTERNAL
KAFKA_CONTROLLER_QUORUM_VOTERS: 1@kafka-broker-1:9093
KAFKA_PROCESS_ROLES: broker,controller
KAFKA_NODE_ID: 1
CLUSTER_ID: 'lFakshjkdN5OsydWA3ynLHQ=='
KAFKA_LOG_DIRS: /var/lib/kafka/data
KAFKA_OFFSETS_TOPIC_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_REPLICATION_FACTOR: 1
KAFKA_TRANSACTION_STATE_LOG_MIN_ISR: 1
KAFKA_MIN_INSYNC_REPLICAS: 1
KAFKA_ADVERTISED_LISTENERS: "INTERNAL://kafka-broker-1:19092,EXTERNAL://10.31.1.31:9092"
KAFKA_MESSAGE_MAX_BYTES: 10485760 # 10 MB
KAFKA_MAX_REQUEST_SIZE: 10485760 # 10 MB
KAFKA_LOG_RETENTION_BYTES: 1073741824 # 1 GB
KAFKA_LOG_CLEANUP_POLICY: delete
volumes:
- /var/lib/kafka/data
ports:
- "9092:9092"
networks:
- my-network
kafdrop-1:
image: obsidiandynamics/kafdrop
restart: "no"
ports:
- "9000:9000"
environment:
KAFKA_BROKERCONNECT: "kafka-broker-1:19092" # Adjust ports if necessary
depends_on:
- kafka-broker-1
networks:
- my-network
networks:
my-network:
driver: bridge