I have fluentbit ingesting logs, shipping them to Loki, which we can then search in Grafana. I have a parser which extracts the severity level from the log (info/warn/debug/trace), but in Grafana the level is automatically set to debug, which doesn’t appear in either the logs or my configs. Ex:
fluent-bit parser:
[PARSER]
Name my_parser
Format regex
Regex /^[(?<time>(?:[1-9]d{3}-(?:(?:0[1-9]|1[0-2])-(?:0[1-9]|1d|2[0-8])|(?:0[13-9]|1[0-2])-(?:29|30)|(?:0[13578]|1[02])-31)|(?:[1-9]d(?:0[48]|[2468][048]|[13579][26])|(?:[2468][048]|[13579][26])00)-02-29)T(?:[01]d|2[0-3]):[0-5]d:[0-5]d(?:.d{1,9})?(?:Z|[+-][01]d:[0-5]d))(?:s*)(?<level>w*)(?:s*)(?<module>.*)(?:]s*)(?<message>.*)$/
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%LZ
raw log from app:
[2024-07-16T22:30:19.108Z INFO solana_core::replay_stage] new root 277953146
“raw log” as Loki sees it after parsing:
{"level":"INFO","module":"solana_core::replay_stage","message":"new root 277953146"}
log as displayed in Grafana:
You can see the time field has been extracted from the raw log and is displayed correctly in Grafana, but for some reason my level field is being set as debug instead of info
I was able to solve this by using Labels in my output. Utilizing the [PARSER] result in the initial post:
[OUTPUT]
Name loki
Match *
Labels level=$level
Related Fluentbit docs: https://docs.fluentbit.io/manual/pipeline/outputs/loki#labels