I setup MM2 to migrate the old data from the Kafka setup, which is very primitive consisting of one Zookeeper and one Kafka instances, to new KRaft cluster which has 3 controllers and 3 brokers. In the old setup, there are 44 topics and 2 of them have around 3.5m offsets. But the number of messages that is replicated to the new cluster is around 350k. I tried several configs but it didnt change the numbers. Other topics not fully replicated neither. One topic I’ve keeping my eye on is constantly replicating 8k message out of 50k in the old setup. I want know why the offsets not replicated alltogether. The whole idea was migrating the data but this kinda becomes a problem. This is the connector config;
{
"name": "zk",
"config": {
"connector.class": "org.apache.kafka.connect.mirror.MirrorSourceConnector",
"source.cluster.alias": "zk",
"target.cluster.alias": "kraft",
"source.cluster.bootstrap.servers": "192.168.2.18:29092",
"target.cluster.bootstrap.servers": "192.168.18.2:29093,192.168.18.2:29094,192.168.18.2:29095",
"target.cluster.security.protocol": "SASL_PLAINTEXT",
"target.cluster.sasl.mechanism": "PLAIN",
"cluster1->cluster2.enabled": true,
"target.cluster.sasl.jaas.config": "org.apache.kafka.common.security.plain.PlainLoginModule required username='admin' password='admin-secret' user_admin='admin-secret';",
"key.converter.class": "org.apache.kafka.connect.converters.ByteArrayConverter",
"value.converter.class": "org.apache.kafka.connect.converters.ByteArrayConverter",
"replication.policy.class": "org.apache.kafka.connect.mirror.IdentityReplicationPolicy",
"topics": ".*",
"refresh.topics.enabled": true,
"refresh.groups.enabled": true,
"emit.checkpoints.enabled": true,
"sync.group.offsets.enabled": true,
"consumer.auto.offset.reset": "earliest",
"offset-syncs.topic.location": "source"
}
}
Any help is much appreciated.