Looking forward to dive deep in most optimal design strategy for Kafka topic migration.
I am publishing to two topics. Topic-1(legacy) topic-2 (new topic)
I want my consumer to migrate to new topic topic-2. How to do this migration with no consumer downtime and no loss of data.
Topic 1 and topic-2 are not guaranteed to be in sync and publish at same time , will have same data flow though.
Strategy-1
Start consumer on topic-2 with reading from earliest message on Kafka.
- slow and too much lag can generate but complete solution if able to process enormous data
Reset offset validation and getting offset from consumer