I have a spring boot + kafka application, which consumes the messages from kafka, process and update DB. We have set “manual auto commit” and the max poll records as “500” and max poll interval is 1000ms and concurrency set as “100”.

The problem we are facing is multiple threads from different(or even from same pod as I checked in logs) pods (currently we have 4 pods running) consuming the messages and end up inserting duplicate records in DB. Kafka partitions are 4.

Some of the solutions seen in the internet and tried are

1. Make the application idempotent – not possible in my application
2. Use same groupId – I have given same groupId as we have deployed same code base.
3. reduce the maxpoll records and increase the poll interval – This I have tried not dont see much improvement.
4. partitions and pods should be same – we are maintaining the same.

Can someone please help, what is the best we can do ?. Is there any property in kafka or spring boot which will enable us to make the application to consume only once despite running multiple instances?