The transformation stage is executed after the parsing stage. This makes the decolorize processing not useful in my opinion. I need a cleaned message before the parsing phase. To extract my fields in a regex I have to include the ANSI coloring noise, this is not good.

1. How to clean the message before the parsing?
2. I’m using rsyslog, can rsyslog be configured to clean the message before sending it to promtail syslog?

What I want to parse:

#033[90m2024-05-20T20:44:29Z#033[0m | INFO | #033[1mHTTP response headers: map[Cache-Control:[must-revalidate] Content-Length:[247] Content-Type:[application/json] Date:[Mon, 20 May 2024 20:44:28 GMT] Server:[CouchDB/3.3.3 (Erlang OTP/24)] X-Couch-Request-Id:[107da8>

My processing pipeline:

    - match:
        selector: '{application="couchdb"}'
        stages:
        - decolorize:
        - regex:
            expression: '#d+?[d+?m(?P<timestamp>[w:-]*?)#d+?[d+?ms+?|s+?(?P<level>[A-Z]+)s+?|s+?#d+?[d+?m(?P<message>.*)#?d*?[?d*?m?'
        - labels:
            level:
            facility:
        - timestamp:
            format: RFC3339
            source: timestamp
        - output:
            source: message