I’m using Loki as my datasource and want to compare number of errors in the logs on different days.

I have a panel with a query that counts the total number of errors today:

sum by(container) (count_over_time({cluster="$Env", level="ERROR"} [24h]))

then another one that offsets the calculation by a selected $Offset:

sum by(container) (count_over_time({cluster="$Env", level="ERROR"} [24h] offset $Offset))

then a third one that subtracts the offset query result from today’s:

sum by(container) (count_over_time({cluster="$Env", level="ERROR"} [24h]))
-
sum by(container) (count_over_time({cluster="$Env", level="ERROR"} [24h] offset $Offset))

This works fine if the $Offset selected is =< 48h. Over that it times out

Status: 504. Message: Get (http://…) context deadline exceeded (Client.Timeout exceeded while awaiting headers)

I understand there’s a large number of logs to process with that query, but I was expecting Loki to only look at 24h worth of data shifted by the $Offset, which should work OK if it works fine for the last 24 or 48h and the logs volumes don’t change much each day. Though it seems that all data from now till whatever $Offset is selected is read, resulting in timeout for larger offsets.

Am I doing something wrong with the query?