In my Kubernetes cluster I ship logs via Alloy to Loki and access them through Grafana (version 10.x). The log lines are JSON formatted and contain a logger
field. The loggers build a hierarchy separated by dots like module.sub.subsub
. I thinks this is a very common setup and nothing fancy.
My Grafana dashboard has a variable to select one or multiple loggers. The logs are filtered based on the selection. The only option to fill the drop downs from Loki is based on labels. To get this working I configured Alloy to extract the logger from the JSON and to add it as label. That works fine. The drop down is filled with data and the proper filter is applied.
But this filters on the exact label. If I select the logger module.sub
I’ll only get logs for exactly that logger. Not the ones for module.sub.subsub
. My goal is to be able to filter for module
and get all logs whose logger start with module
.
To do so, I need to get possible prefixes (module
, module.sub
, module.sub.subsub
) into the drop down. The information I found suggests that this is usually handled on the query level so that the data is prepared in the backend and Grafana only needs to display it.
Loki seems not to support that kind of data processing in queries. I could also not find a way to split the loggers in Alloy. That would probably work, but I understand the Loki docs that one should not do so because it would make queries less efficient.
I assume that this is a very common requirement, but I could not find a solution. So I assume that I’m missing something or are on the completely wrong track.
How is it possible to filter Loki logs by logger prefix?