I currently have 2 instances of the same application running in 2 environments (lets call them instance 1 and instance 2). These instances consume messages from Kafka (different Kafka brokers). In logs for both instances I see the following error from ConsumerCoordinator
[ERROR] o.a.k.c.c.i.ConsumerCoordinator - [Consumer clientId=csp.hk, groupId=csp.hk] Asynchronous auto-commit of offsets failed: Offset commit cannot be completed since the consumer is not part of an active group for auto partition assign
ment; it is likely that the consumer was kicked out of the group.. Will continue to join group.
[WARN ] o.a.k.c.c.i.ConsumerCoordinator - [Consumer clientId=csp.hk, groupId=csp.hk] Asynchronous auto-commit of offsets /*many topic partitions*/ failed: Offset commit cannot be completed since the consumer is not part of an active group for auto partition assignment; it is likely that the consumer was kicked out of the group.
However in instance 1 I also see a retry attempt being made right after being kicked out.
[INFO ] o.a.k.c.c.i.ConsumerCoordinator - [Consumer clientId=csp.hk, groupId=csp.hk] Request joining group due to: need to re-join with the given member-id: csp.hk-0e3f111c-8a92-49c3-81dc-61c4c5956df7
[INFO ] o.a.k.c.c.i.ConsumerCoordinator - [Consumer clientId=csp.hk, groupId=csp.hk] (Re-)joining group
There is no attempt being made by instance 2. The consumer never re-joins the group and therefore all future messages are ignored/not consumed.
I cant seem to figure out why there is a difference in behavior. Both applications should be using same Kafka consumer properties. Could anyone help with this?
I am happy to provide more info if that helps.