Sometimes I run into a situation where I had to restart my API to consume messages from a datacenter
Here is the
We have 3 data centers (DC1, DC2 and DC3) and each datacenter has about 4 to 8 brokers. One Consumer (spring boot java api) having 10 instances(pods) consumes messages from a topic name say PROD-TOPIC that has 60 partitions and one consumer group say prod-consumer-group.
I am using @KafkaListerner annotation and do not have any customized polling logic. Also I am using default kafka settings.
On a normal situation, DC1 will have 5 consumer instances (12 partitions/pod), DC2 will have 3 consumer instances (20 Partitions/pod) and DC3 will have 2 consumer instances (30 partitions/pod) consuming messages.
Since DC3 has only 2 pods, due to issues when both pods having issues, api stops consuming message from that particular datacenter.
Issue –
Since the consumer taking longer than max.poll.interval.ms to process all the messages and particular pod will be considered inactive/dead and removed from the group
splunk log
[-consumer-group] g.g.k.c.c.internals.ConsumerCoordinator : [Consumer clientid=consumer-prod-consumer-group-1, groupld=prod-consumer-group] consumer poll timeout has expired. This means the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time processing messages. You can address this either by increasing max.poll.interval.ms or by reducing the maximum size of batches returned in poll() with max.poll.records.
Sometime I see the following splunk message as well. Not sure what is the root cause
splunk log
Request joining group due to: rebalance failed due to 'The group member needs to have a valid member id before actually entering a consumer group.' (MemberidRequired Exception)
My questions are,
-
Why doesn’t kafka rebalances the pods equally (almost) amoung each data centers say 3, 3 and 4 instead of 5, 3 and 2 (as in the case above).
-
In the above case, when 2 pods in DC3 goes down, why kafka wouldn’t rebalances among remaining 8 pods so that each data centers gets 3, 3 and 2 pods?
Often I see no active consumers in one of the datacenter (especially DC3 where only two consumers assigned) and messages are keep piling up in each partitions.
- Will there be a data center maintenance or applying patches that could permanently kick the consumer out of the group? I had to frequently restart all the pods so that rebalance happens and consumer starts consuming messages from all datacenter again
If I increase the number of pods to 12, Would kafka rebalances 4 pods with each datacenter ?
How about increasing the max.poll.interval.ms to 10 minutes to allow Consumer to process all fetched messages before the max.poll.interval.ms. Currently it is taking the default of 5 minutes (300000 ms)