I am working with an Apache Flink application that reads data streams from Kafka. The application processes the streams, creating objects from the data and calculating the processing time by marking the start time at object creation and the end time after performing a series of transformations including keyBy and some calculations.
In most cases, the processing times recorded are between 0-1 ms. However, I’ve noticed that some results show an unusually high processing time of over 10 ms. This behavior seems to be intermittent and varies between runs, despite the following consistent setup:
- The Flink cluster is deployed in standalone mode.
- I restart the cluster before each run to ensure a consistent environment.
- The same JAR file is used for all the runs.
What could be causing these intermittent spikes in processing time? How can I diagnose the underlying issue? Or is this fluctuation normal?