I still newbie in fluentbit and confuse parsing java.
So I created fluentbit using helm and created a filter to parse java logs in json form. I tried to use nginx and using nginx parse worked but I got stuck on java. I’ve tried adding it in the input filter but it doesn’t work.
<code>logLevel: debug
#image:
# tag: 2.2.3
luaScripts:
prefix_output.lua: |
function add_index_prefix_field(tag, timestamp, record)
retcode = 0
prefix = 'inf_staging' -- Your prefix for output index
namespace = record['kubernetes']['namespace_name'] -- Any record accessor
app_name = record['kubernetes']['container_name'] -- Any record accessor
if namespace ~= nil or app_name ~= nil then
index_prefix = prefix .. '_' .. namespace .. '_' .. app_name
if index_prefix ~= nil then
record['index_prefix'] = index_prefix
retcode = 2
end
end
return retcode, timestamp, record
end
## https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/configuration-file
config:
service: |
[SERVICE]
Daemon Off
Flush {{ .Values.flush }}
Log_Level {{ .Values.logLevel }}
#Parsers_File /fluent-bit/etc/parsers.conf
Parsers_File /fluent-bit/etc/conf/custom_parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port {{ .Values.metricsPort }}
Health_Check On
## https://docs.fluentbit.io/manual/pipeline/inputs
inputs: |
[INPUT]
Name tail
Tag kube.poc-log.<pod_name>.<container_name>.<container_id>
Path /var/log/containers/*_poc-log_*.log
Tag_Regex (?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-(?<container_id>[a-z0-9]{64}).log$
multiline.parser cri
DB /var/log/poc-log.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
[INPUT]
Name tail
Tag kube.streaming.<pod_name>.<container_name>.<container_id>
Path /var/log/containers/*_streaming_*.log
Tag_Regex (?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-(?<container_id>[a-z0-9]{64}).log$
multiline.parser cri
DB /var/log/streaming.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
## https://docs.fluentbit.io/manual/pipeline/filters
filters: |
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc:443
Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
Kube_Tag_Prefix kube.
Regex_Parser custom-tag
Merge_Log On
Merge_Log_Key log_processed
#Merge_Parser nginx
Keep_Log On
Labels Off
Annotations Off
K8S-Logging.Parser On
K8S-Logging.Exclude On
[Filter]
Name nest
Match kube.*
Operation lift
Nested_under kubernetes
Add_prefix kubernetes_
[FILTER]
Name modify
Match kube.*
Remove stream
Remove kubernetes_pod_id
Remove kubernetes_docker_id
Remove kubernetes_host
Remove kubernetes_container_hash
[Filter]
Name nest
Match kube.*
Operation nest
Wildcard kubernetes_*
Nest_under kubernetes
Remove_prefix kubernetes_
[FILTER]
Name lua
Match kube.*
script /fluent-bit/scripts/prefix_output.lua
call add_index_prefix_field
## https://docs.fluentbit.io/manual/pipeline/outputs
outputs: |
[OUTPUT]
Name es
Match kube.*
Host xxxxxxxxxxxxxxxxxxx
Port 9243
HTTP_User elk-log-stag
HTTP_Passwd xxxxxxxxxxxxxxx
Logstash_Format On
# Format index, cek filter prefix_output lua
Logstash_Prefix_Key $index_prefix
Logstash_DateFormat %Y-%m-%d
Time_Key @timestamp
Retry_Limit False
Replace_Dots On
tls On
tls.verify true
Suppress_Type_Name true
## https://docs.fluentbit.io/manual/pipeline/parsers
customParsers: |
[PARSER]
Name nginx
Format regex
Regex ^(?<remote_addr>[^ ]*) - (?<remote_user>[^ ]*) [(?<time>[^]]*)] "(?<method>S+)(?: +(?<path>[^"]*?)(?: +S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^"]*)" "(?<agent>[^"]*)")?(?: "(?<http_x_forwarded_for>[^"]*)")?$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name xixi
Format regex
Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) [(?<time>[^]]*)] "(?<method>S+)(?: +(?<path>[^ ]*) +S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^"]*)" "(?<agent>.*)")?$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name custom-tag
Format regex
Regex ^(?<namespace_name>[^_]+).(?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*).(?<container_name>.+).(?<container_id>[a-z0-9]{64})
[PARSER]
# http://rubular.com/r/tjUt3Awgg4
Name cri
Format regex
# XXX: modified from upstream: s/message/log/
Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<log>.*)$
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
[PARSER]
Name catchall
Format regex
Regex ^(?<message>.*)$
[PARSER]
Name json
Format json
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
</code>
<code>logLevel: debug
#image:
# tag: 2.2.3
luaScripts:
prefix_output.lua: |
function add_index_prefix_field(tag, timestamp, record)
retcode = 0
prefix = 'inf_staging' -- Your prefix for output index
namespace = record['kubernetes']['namespace_name'] -- Any record accessor
app_name = record['kubernetes']['container_name'] -- Any record accessor
if namespace ~= nil or app_name ~= nil then
index_prefix = prefix .. '_' .. namespace .. '_' .. app_name
if index_prefix ~= nil then
record['index_prefix'] = index_prefix
retcode = 2
end
end
return retcode, timestamp, record
end
## https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/configuration-file
config:
service: |
[SERVICE]
Daemon Off
Flush {{ .Values.flush }}
Log_Level {{ .Values.logLevel }}
#Parsers_File /fluent-bit/etc/parsers.conf
Parsers_File /fluent-bit/etc/conf/custom_parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port {{ .Values.metricsPort }}
Health_Check On
## https://docs.fluentbit.io/manual/pipeline/inputs
inputs: |
[INPUT]
Name tail
Tag kube.poc-log.<pod_name>.<container_name>.<container_id>
Path /var/log/containers/*_poc-log_*.log
Tag_Regex (?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-(?<container_id>[a-z0-9]{64}).log$
multiline.parser cri
DB /var/log/poc-log.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
[INPUT]
Name tail
Tag kube.streaming.<pod_name>.<container_name>.<container_id>
Path /var/log/containers/*_streaming_*.log
Tag_Regex (?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-(?<container_id>[a-z0-9]{64}).log$
multiline.parser cri
DB /var/log/streaming.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
## https://docs.fluentbit.io/manual/pipeline/filters
filters: |
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc:443
Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
Kube_Tag_Prefix kube.
Regex_Parser custom-tag
Merge_Log On
Merge_Log_Key log_processed
#Merge_Parser nginx
Keep_Log On
Labels Off
Annotations Off
K8S-Logging.Parser On
K8S-Logging.Exclude On
[Filter]
Name nest
Match kube.*
Operation lift
Nested_under kubernetes
Add_prefix kubernetes_
[FILTER]
Name modify
Match kube.*
Remove stream
Remove kubernetes_pod_id
Remove kubernetes_docker_id
Remove kubernetes_host
Remove kubernetes_container_hash
[Filter]
Name nest
Match kube.*
Operation nest
Wildcard kubernetes_*
Nest_under kubernetes
Remove_prefix kubernetes_
[FILTER]
Name lua
Match kube.*
script /fluent-bit/scripts/prefix_output.lua
call add_index_prefix_field
## https://docs.fluentbit.io/manual/pipeline/outputs
outputs: |
[OUTPUT]
Name es
Match kube.*
Host xxxxxxxxxxxxxxxxxxx
Port 9243
HTTP_User elk-log-stag
HTTP_Passwd xxxxxxxxxxxxxxx
Logstash_Format On
# Format index, cek filter prefix_output lua
Logstash_Prefix_Key $index_prefix
Logstash_DateFormat %Y-%m-%d
Time_Key @timestamp
Retry_Limit False
Replace_Dots On
tls On
tls.verify true
Suppress_Type_Name true
## https://docs.fluentbit.io/manual/pipeline/parsers
customParsers: |
[PARSER]
Name nginx
Format regex
Regex ^(?<remote_addr>[^ ]*) - (?<remote_user>[^ ]*) [(?<time>[^]]*)] "(?<method>S+)(?: +(?<path>[^"]*?)(?: +S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^"]*)" "(?<agent>[^"]*)")?(?: "(?<http_x_forwarded_for>[^"]*)")?$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name xixi
Format regex
Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) [(?<time>[^]]*)] "(?<method>S+)(?: +(?<path>[^ ]*) +S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^"]*)" "(?<agent>.*)")?$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name custom-tag
Format regex
Regex ^(?<namespace_name>[^_]+).(?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*).(?<container_name>.+).(?<container_id>[a-z0-9]{64})
[PARSER]
# http://rubular.com/r/tjUt3Awgg4
Name cri
Format regex
# XXX: modified from upstream: s/message/log/
Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<log>.*)$
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
[PARSER]
Name catchall
Format regex
Regex ^(?<message>.*)$
[PARSER]
Name json
Format json
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
</code>
logLevel: debug
#image:
# tag: 2.2.3
luaScripts:
prefix_output.lua: |
function add_index_prefix_field(tag, timestamp, record)
retcode = 0
prefix = 'inf_staging' -- Your prefix for output index
namespace = record['kubernetes']['namespace_name'] -- Any record accessor
app_name = record['kubernetes']['container_name'] -- Any record accessor
if namespace ~= nil or app_name ~= nil then
index_prefix = prefix .. '_' .. namespace .. '_' .. app_name
if index_prefix ~= nil then
record['index_prefix'] = index_prefix
retcode = 2
end
end
return retcode, timestamp, record
end
## https://docs.fluentbit.io/manual/administration/configuring-fluent-bit/classic-mode/configuration-file
config:
service: |
[SERVICE]
Daemon Off
Flush {{ .Values.flush }}
Log_Level {{ .Values.logLevel }}
#Parsers_File /fluent-bit/etc/parsers.conf
Parsers_File /fluent-bit/etc/conf/custom_parsers.conf
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port {{ .Values.metricsPort }}
Health_Check On
## https://docs.fluentbit.io/manual/pipeline/inputs
inputs: |
[INPUT]
Name tail
Tag kube.poc-log.<pod_name>.<container_name>.<container_id>
Path /var/log/containers/*_poc-log_*.log
Tag_Regex (?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-(?<container_id>[a-z0-9]{64}).log$
multiline.parser cri
DB /var/log/poc-log.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
[INPUT]
Name tail
Tag kube.streaming.<pod_name>.<container_name>.<container_id>
Path /var/log/containers/*_streaming_*.log
Tag_Regex (?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*)_(?<namespace_name>[^_]+)_(?<container_name>.+)-(?<container_id>[a-z0-9]{64}).log$
multiline.parser cri
DB /var/log/streaming.db
Mem_Buf_Limit 5MB
Skip_Long_Lines On
Refresh_Interval 10
## https://docs.fluentbit.io/manual/pipeline/filters
filters: |
[FILTER]
Name kubernetes
Match kube.*
Kube_URL https://kubernetes.default.svc:443
Kube_CA_File /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
Kube_Token_File /var/run/secrets/kubernetes.io/serviceaccount/token
Kube_Tag_Prefix kube.
Regex_Parser custom-tag
Merge_Log On
Merge_Log_Key log_processed
#Merge_Parser nginx
Keep_Log On
Labels Off
Annotations Off
K8S-Logging.Parser On
K8S-Logging.Exclude On
[Filter]
Name nest
Match kube.*
Operation lift
Nested_under kubernetes
Add_prefix kubernetes_
[FILTER]
Name modify
Match kube.*
Remove stream
Remove kubernetes_pod_id
Remove kubernetes_docker_id
Remove kubernetes_host
Remove kubernetes_container_hash
[Filter]
Name nest
Match kube.*
Operation nest
Wildcard kubernetes_*
Nest_under kubernetes
Remove_prefix kubernetes_
[FILTER]
Name lua
Match kube.*
script /fluent-bit/scripts/prefix_output.lua
call add_index_prefix_field
## https://docs.fluentbit.io/manual/pipeline/outputs
outputs: |
[OUTPUT]
Name es
Match kube.*
Host xxxxxxxxxxxxxxxxxxx
Port 9243
HTTP_User elk-log-stag
HTTP_Passwd xxxxxxxxxxxxxxx
Logstash_Format On
# Format index, cek filter prefix_output lua
Logstash_Prefix_Key $index_prefix
Logstash_DateFormat %Y-%m-%d
Time_Key @timestamp
Retry_Limit False
Replace_Dots On
tls On
tls.verify true
Suppress_Type_Name true
## https://docs.fluentbit.io/manual/pipeline/parsers
customParsers: |
[PARSER]
Name nginx
Format regex
Regex ^(?<remote_addr>[^ ]*) - (?<remote_user>[^ ]*) [(?<time>[^]]*)] "(?<method>S+)(?: +(?<path>[^"]*?)(?: +S*)?)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^"]*)" "(?<agent>[^"]*)")?(?: "(?<http_x_forwarded_for>[^"]*)")?$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name xixi
Format regex
Regex ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) [(?<time>[^]]*)] "(?<method>S+)(?: +(?<path>[^ ]*) +S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^"]*)" "(?<agent>.*)")?$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
[PARSER]
Name custom-tag
Format regex
Regex ^(?<namespace_name>[^_]+).(?<pod_name>[a-z0-9](?:[-a-z0-9]*[a-z0-9])?(?:.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*).(?<container_name>.+).(?<container_id>[a-z0-9]{64})
[PARSER]
# http://rubular.com/r/tjUt3Awgg4
Name cri
Format regex
# XXX: modified from upstream: s/message/log/
Regex ^(?<time>[^ ]+) (?<stream>stdout|stderr) (?<logtag>[^ ]*) (?<log>.*)$
Time_Key time
Time_Format %Y-%m-%dT%H:%M:%S.%L%z
[PARSER]
Name catchall
Format regex
Regex ^(?<message>.*)$
[PARSER]
Name json
Format json
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z
I still working on it, hope you can help me guys.
Thank you.