Stack

Opensearch + Opensearch-Dashboards + Fluent-Bit

Goal

Parse Apache Tomcat’s catalina.out log file

Configs

fluent-bit.conf:

[SERVICE]
    Flush                 5
    Log_Level             info
    Parsers_File          parsers.conf


[INPUT]
    Name   tail
    Tag   tomcat.access
    Path    /u01/tomcat/apache-tomcat/logs/access*.txt
    Parser   tomcat_access

[INPUT]
    Name tail
    Tag tomcat.catalina
    Path /u01/tomcat/apache-tomcat/logs/catalina.out

[FILTER]
    Name multiline
    Match tomcat.catalina
    multiline.key_content log
    multiline.parser tomcat_catalina_multiline, java

[FILTER]
    Name     parser
    Match    tomcat.catalina
    Key_Name log
    Parser   tomcat_catalina_spring
    Preserve_key On

[OUTPUT]
    Name file
    Match tomcat.catalina
    Path /u01/fluent-bit
    Format out_file

[OUTPUT]
    Name                  opensearch
    Match                 tomcat*
    Host                  <host>
    Port                  <port>
    Suppress_Type_Name    On
    Index                 tomcat-test

parsers.conf:

[PARSER]
    Name        tomcat_access
    Format      regex
    Regex       ^(?<host>S+) (?<ident>S+) (?<user>S+) [(?<time>.*)] "(?<method>S+) (?<url>S+) (?<protocol>S+)" (?<status>S+) (?<size>S+)$
    Time_Key    time
    Time_Format %d/%b/%Y:%H:%M:%S %z

[PARSER]
    Name        tomcat_catalina_spring
    Format      regex
    Regex       ^(?<time>d+-d+-d+ d+:d+:d+.d+)s+(?<level>S+) d+ --- [s*(?<thread>[^]]+)] (?<context>S+)s+: (?<message>.*)$
    Time_Key    time
    Time_Format %Y-%m-%d %H:%M:%S.%L

[MULTILINE_PARSER]
    name          tomcat_catalina_multiline
    type          regex
    flush_timeout 500
    #
    # rules |   state name    | regex pattern                             | next state
    # ------|-----------------|--------------------------------------------------------
    rule      "start_state"      "/^(d+-d+-d+ d+:d+:d+.d+)(.*)$/"  "empty_row"
    rule      "empty_row"        "/^$/"                                    "error_row"
    rule      "error_row"        "/^.*$/"                                  "stacktrace"
    rule      "stacktrace"       "/^(s*at .*|)$/"                         "stacktrace"

File OUTPUT section

I’ve configured an output to a file for debugging purposes.
The tomcat.catalina contents seem to be just fine:

tomcat.catalina: [1723676238.680000000, {"level":"ERROR","thread":"main","context":"com.zaxxer.hikari.pool.HikariPool","message":"HikariPool-1 - Exception during pool initialization."}]
tomcat.catalina: [1723665438.703756094, {"log":"java.sql.SQLRecoverableException: IO Error: The Network Adapter could not establish the connectionntat ]
... <enormous multiline log entry here ...>"]
tomcat.catalina: [1723665438.703751094, {"log":"n"}]
tomcat.catalina: [1723676238.682000000, {"level":"WARN","thread":"main","context":"ConfigServletWebServerApplicationContext","message":"Exception encountered during context initialization - cancelling refresh attempt: org.springframework.context.ApplicationContextException... <long message here>"}]
### ... ###
### some seem to be parsed exactly how I need entries, same as following... ###
tomcat.catalina: [1723665438.703843277, {"log":"n"}]
tomcat.catalina: [1723676238.694000000, {"level":"INFO","thread":"main","context":"ConditionEvaluationReportLoggingListener"}]
tomcat.catalina: [1723665438.703843277, {"log":"n"}]
tomcat.catalina: [1723676238.723000000, {"level":"ERROR","thread":"main","context":"o.s.boot.SpringApplication","message":"Application run failed"}]
### and so on... ###

The problem

When I go to my Opensearch-Dashboards Discover tab, the parsed single line entries (those with level, thread, context and message fields) are not there. Only multiline ones and these:

tomcat.catalina: [1723665438.703843277, {"log":"n"}]

(The least show up as empty log ofc, the n symbol emerges only in JSON view.)

The fields are present at tomcat-test index Mappings and also show up in Fields of tomcat-test index pattern.

I’ve refreshed the tomcat-test index pattern fields, deleted the tomcat-test index, restarted both Fluent-Bit and Tomcat to recreate the index and push some new entries there. No errors reported by either Opensearch-Dashboards or Fluent-Bit.