I frequently receive application logs from journald. The application itself logs with jsonlines, but sometimes I got invalid lines, if…:

• something is written to stderr like Listening for transport dt_socket at address: 5045 or something like JProfiler > ....
• systemd adds some logs itself like Started my.service - Service Description.
• my log lines exceed the maximum journald line length, they are split up into multiple lines.

Long story short, I want to do some preprocessing of the shared log files where in a first step I only keep valid jsonlines and put all other lines into a dedicated second file for later review and processing.

My first attempt was to pipe the logfile through a bash script with running jq -e . >/dev/null 2>&1 <<< "$1" on each line. That turned out to be way too slow as my log files usually are about 2GB in size. So I would like to have a solution where I call jq only once and let it process all the loglines which would be way faster than calling jq once for each line.

I found the try catch syntax from jq but I don’t get a fully working solution with regards to the file containing the non jsonlines messages. Here are my best attempts so far:

cat input.log | jq -c -r -R '. as $line | try fromjson catch "($line)n" | stderr' > good.jsonl 2> bad.txt

This nicely keeps only good jsonlines, but it is horribly slow. Further,the “bad.txt” is all in one line and literally prints “n” as 2-char-string instead of newline character. Don’t know why.

I tried another formatting of that output string like so:

cat input.log | jq -c -r -R '. as $line | try fromjson catch ($line+="n" | $line | stderr) ' > good.jsonl 2> bad.txt

That runs much faster and produces proper jsonl good.jsonl but results in an unusable bad.txt like:

jq: error (at <stdin>:18126): Invalid path expression with result "Started my.service ...

where it literally prints only the first few characters of my broken message and not the entire line, hence I can’t reconstruct it.

The best solution I found so far is:

cat input.log | jq -c -r -R '. as $line | if (fromjson) then . else $line | stderr end' > good.jsonl 2> bad.txt

This leaves my good.json with the JSONL as expected and produces a “bad.txt” like:

jq: error (at <stdin>:18126): Invalid numeric literal at line 1, column 8 (while parsing 'Started my.service - Service Description.')
jq: error (at <stdin>:18127): Invalid numeric literal at line 1, column 10 (while parsing 'Listening for transport dt_socket at address: 5045')

If I find no better solution, I’ll go for the last one and post-edit the jq error message with some regex or similar to only keep the original line without the error details, but I wonder if I can somehow simply tell jq to directly output my original line without the error details? For me it looks like the catch clause has some issues with the jq string formatting options..

FYI: I am currently using jq 1.6 (installed on Ubuntu 22.04)