I’m running 8 threads in parallel. Each thread is given a list of partitions to process and will process all messages between a given start and end timestamp.
While most of the threads run fine, one or two lag a lot. Others finish up in 20 minutes but the slow ones take 2 hours, even though the volume of data is almost same across all threads.
The below code takes 300 milliseconds to execute in one thread while in others it takes 10 millisecond. Since we do multiple pools, this difference adds up fast. Also, the number of records polled is almost same in both threads.
ConsumerRecords<String, String> records = consumer.poll(60000);
JSONArray jsonArray = new JSONArray();
for (ConsumerRecord<String, String> record : records) {
jsonArray.put(record.value())
}
I doubt adding to JSONArray will cause this issue, so I feel it’s an issue with the poll. I also have few additional findings :
-
This happens every time I run and can happen for any partition/thread, hence I don’t think it’s an issue with a particular partition or processing thread.
-
If I don’t run in parallel, all of them run fine.
-
All partitions being processed by thread are slow and continue to be slow even when the other jobs are finished. Again, if I quit the thread and re-run without anything else in parallel, it runs fine.
I am trying to find the source of this issue but cannot get any answers. Can someone please help me on this?