When I create a new topic in source cluster with few messages, the new topic will be replicated successfully to target cluster in few seconds. Then I create a consumer group in source, consume 1 message in source, this consumer group will also be replicated successfully to target cluster. Current offset in this new consumer group in both cluster will be 1 since I consumed 1 message.
Then I consume next message in source, the offset in consumer group in source cluster will be 2, but the offset in target cluster is still 1.
Detailed steps:
In source cluster
./bin/kafka-console-producer.sh –bootstrap-server xxx:xxx –topic cnx-test –producer.config /tmp/consumer.properties
topic cnx-test with data will be appear in both source and target.
In source cluster
./bin/kafka-console-consumer.sh –bootstrap-server xxx:xxx –topic cnx-test –group cnx-test-group –consumer.config /tmp/consumer.properties –max-messages 1
group cnx-test-group will appear in both source and target, with offset 1
In source cluster
./bin/kafka-console-consumer.sh –bootstrap-server connectcx-dev-kafka-external-bootstrap.connectcx-dev:9094 –topic cnx-test –group cnx-test-group –consumer.config /tmp/consumer.properties –max-messages 1
The output shows I consumed the second message. cnx-test-group in source cluster shows offset = 2, but cnx-test-group in target cluster will still be with offset = 1.
I’ve tried add offset.lag.max: 1 in sourceConnector, but it turns out that the consumer group cannot be replicated to the target at the beginning (when i firstly create consumer group and consume first message)