I need to add a custom field to all log entries that contain string “myapp.database – Statement returned”. All other logs that do not contain this string, should not be discarded/filtered out.
I managed to add the custom field by applying grep, parser and record_transformer filters, but at the cost of all other logs being filtered out or the log entries containing string “myapp.database – Statement returned” being duplicated. Neither of these results is acceptable – as i said, the log entries containing the string should get the new field and the remaining log entries should not be filtered out.
Eventually i tried to apply label routing but it resulted in log entries being filtered out and custom field not being added at all. Below is my fluentd config where i tried to apply labels.
<source>
@type tail
path C:dmskrnl.log
pos_file C:dmskrnl.log.pos
tag dmskrnl.raw
<parse>
@type none
</parse>
</source>
<filter dmskrnl.raw>
@type grep
<regexp>
key message
pattern /myapp.database - Statement returned/
</regexp>
@label @ENRICH
</filter>
<label @ENRICH>
<filter dmskrnl.raw>
@type parser
key_name message
reserve_data true
tag dmskrnl.enriched
<parse>
@type regexp
expression /myapp.database - Statement returned (?<returned_bytes>d+) bytes/
</parse>
</filter>
<filter dmskrnl.enriched>
@type record_transformer
<record>
returned_bytes ${record["returned_bytes"]}
</record>
</filter>
<match dmskrnl.enriched>
@type stdout
</match>
</label>
<match dmskrnl.raw>
@type stdout
</match>