I’m using Grafana + Loki for monitoring operation of an application that generates logs in logFmt format. The application outputs log entries with different text patterns when it starts and ends processing a request, let’s say, msg="Started processing request" and msg="Completed processing request". An instance is considered busy for the time interval between these two messages.

Logs from several application instances running in parallel are collected and aggregated to same Loki instance. Each application instance decorates its log entries with a unique per-instance instance.id label.

Metrics I need to collect and visualise in time series are the number of busy instances per second and the intervals of time during which each instance was busy processing a request.

How can I create the queries that return the busy intervals for each application instance, and the number of busy application instances per second?

BR