sample log entry –
2024-06-14 06:51:58.906 | INFO | main:main:30 – All indicators and predictions:
[
{
“datetime”:1718308800000,
“key1”:66701.7292561551,
“key2”:67014.6836462447,
“key3”:66529.3890236763,
“key4”:66880.5703857987,
“key5”:1132.28416,
“key6”:744.0276710022,
“key7”:0,
“key8”:0.0585026458,
“key9”:0,
“key10″:””,
“key11”:null
}
]
2024-06-14 06:51:58.907 | INFO | main:main:31 – datetime key1 key2 key3 key4 key5 … key6 key7 key8 key9 key10 key11
994 2024-06-13 20:00:00 66701.729256 67014.683646 66529.389024 66880.570386 1132.28416 … 744.027671 0 0.058503 0 NaN
995 2024-06-13 22:00:00 66880.569752 66991.561215 66630.729675 66808.863462 788.89703 … 680.327224 0 0.036088 0 NaN
996 2024-06-14 00:00:00 66808.863251 66929.160405 66530.243225 66807.067821 1427.33212 … 660.197075 0 -0.004118 0 NaN
997 2024-06-14 02:00:00 66807.067750 66994.600135 66627.601075 66765.022607 1277.60587 … 648.589025 0 0.036305 0 NaN
998 2024-06-14 04:00:00 66765.022583 67145.746712 66681.040358 66956.667536 1253.85572 … 640.019675 0 0.151628 0 NaN

[5 rows x 24 columns]
2024-06-14 06:51:58.935 | INFO | main:main:66 – No new signals found
2024-06-14 06:51:58.935 | INFO | main:main:68 – Time taken: 0.6224438566714525

I am unable to parse them properly.
Trying to format them in order –
timestamp | level | message

my fluent-bit conf file –
[SERVICE]
Flush 1
Log_Level info
Parsers_File multiline_parsers.conf
Daemon Off
HTTP_Server On
HTTP_Listen 0.0.0.0
HTTP_Port 2020
Health_Check On

[FILTER]
Name parser
Match jarvis.*
Key_Name log
Parser jarvis_parser

[INPUT]
Name tail
Path /path/to/log/log-file.log
Parser multiline-log
Tag jarvis.archi
Read_From_Head true
Buffer_Chunk_Size 1M
Buffer_Max_Size 5M

[OUTPUT]
Name es
Match jarvis.archi
Host IP
Port PORT
HTTP_User USER
HTTP_Passwd PASSWD
Retry_Limit False
Suppress_Type_Name On
Logstash_Format On
Logstash_Prefix index-name-

multiline_parsers.conf –
[MULTILINE_PARSER]
Name multiline-log
Type regex
Flush_Timeout 1000
Rule “start_state” “/^d{4}-d{2}-d{2} d{2}:d{2}:d{2}.d{3} | [A-Z]+ | .*/” “cont”
Rule “cont” “/^s+/” “cont”

Using it to parse python logs.
Tried multiple regexp –
^(?d{4}-d{2}-d{2} d{2}:d{2}:d{2}.d{3}) | (?[A-Z]+) | (?[^:]+):(?[^:]+):(?d+) – (?.*)
I also tried using a lua script to extract the log message but that too didn’t work.

What all changes should i make i am getting logs in different lines only.