I have to consume messages from a topic on a Kafka cluster with 10 partitions where messages are being evenly distributed accros them. I have a ASP.NET application with 10 background services where each service fires up a consumer (Confluent.Kafka 2.6.1) so that I can process messages from each partition in parallel. For now, I’m not doing any real processing so that is not affecting the time to consume new messages.

The issue that I’m having is that some of my consumers go way ahead with the number of messages consumed from their partition. For example, the consumer with the highest message count has consumed around 17000 messages, while the consumer with the lowest number of messages has consumed only 1500 in a similar time span of 3 minutes.

I’ve played with various settings on the consumer configuration, but without any noticeable difference. Here is my latest configuration try:

new ConsumerConfig
{
    ClientId = options.Operator,
    BootstrapServers = options.GetKafkaBootstrapServers(),
    GroupId = options.GetKafkaGroupId(),
    SecurityProtocol = SecurityProtocol.Ssl,
    SslCaLocation = options.Kafka.Ssl.CaLocation,
    SslCertificateLocation = options.Kafka.Ssl.CertificateLocation,
    SslKeyLocation = options.Kafka.Ssl.KeyLocation,
    EnableAutoOffsetStore = false,
    ConsumeResultFields = "none",
    PartitionAssignmentStrategy = PartitionAssignmentStrategy.CooperativeSticky
};

that produced these results:

partition: average (total message count / elapsed seconds)
--------------------
0: 16 (3000 / 180)
1: 14 (2600 / 185)
2: 89 (16900 / 189)
3: 28 (5300 / 187)
4: 24 (4500 / 185)
5: 56 (10400 / 185)
6: 8 (1500 / 180)
7: 21 (4100 / 194)
8: 28 (5200 / 181)
9: 12 (2200 / 178)

Is there anything on the consumer side that I can do to balance this out? I don’t have control over the kafka cluster, but I might suggest something to the company owning the cluster if that will help with this.

Thank you