Question: How to Configure Docker for Kafka MirrorMaker 2.0 to Replicate Messages Between Two Clusters?
I have two Kafka clusters: a source cluster and a destination cluster, each with 3 brokers. I want to replicate messages from the source cluster to the destination cluster using MirrorMaker 2.0, and I’m using the official Apache Kafka Docker image for this setup.
Could someone help me with the configuration of the docker-compose.yml
file?
- Should I run the MirrorMaker command each time the Docker container starts, or is there a better way to configure this for production use?
- Is it sufficient to run just one instance of MirrorMaker 2.0, or should I set up a dedicated MirrorMaker cluster with 3 nodes, similar to my Kafka clusters?
I’m looking for the best practices for running MirrorMaker 2.0 in a production environment using Docker.
I have tried to run it inside the docker container with below docker compose file
version: '3.8'
services:
mirrormaker2:
image: apache/kafka:latest
hostname: mirrormaker2
container_name: mirrormaker2
volumes:
- ./connect-mirror-maker.properties:/opt/kafka/config/connect-mirror-maker.properties:rw
network_mode: "host"
environment:
KAFKA_NODE_ID: 1
KAFKA_PROCESS_ROLES: 'broker,controller'
KAFKA_LISTENER_SECURITY_PROTOCOL_MAP: 'CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,PLAINTEXT_HOST:PLAINTEXT'
KAFKA_CONTROLLER_QUORUM_VOTERS: '1@:9093'
KAFKA_LISTENERS: 'PLAINTEXT://:19092,CONTROLLER://10.142.13.108:9093,PLAINTEXT_HOST://:9092'
KAFKA_INTER_BROKER_LISTENER_NAME: 'PLAINTEXT'
CLUSTER_ID: '4L6g3nShT-eMCtK--X86sw'
KAFKA_ADVERTISED_LISTENERS: 'PLAINTEXT://:19092,PLAINTEXT_HOST://:29092'
KAFKA_CONTROLLER_LISTENER_NAMES: 'CONTROLLER'
KAFKA_LOG_DIRS: '/tmp/kraft-combined-logs'
command: >
sh -c "cd /opt/kafka && ./bin/connect-mirror-maker.sh ./config/connect-mirror-maker.properties"