I have JSON-like records being written by sensors.
The files are not valid because they look like concatenated records like the following:
{
"timestamp": "2024-07-17T12:31:12Z",
"source_ip": "10.0.0.2",
"source_port": 5678,
"destination_ip": "192.168.1.1",
"destination_port": 443,
"protocol": "UDP",
"bytes_sent": 800
}
{
"timestamp": "2024-07-17T12:31:12Z"
}
{
"timestamp": "2024-07-17T12:31:12Z"
}
There are no leading and trailing square brackets and there are no commas between the records.
I can clean this up with jq easily enough if I can detect the issue. Not all files are broken like this, other files are properly formatted. Also, these files can be large (hundreds of Gb) so I am using ijson. They are customer files, I have no control over their formatting or content, I can only deal with it.
What’s the ijson way to determine if a file is invalid per the above or not? This seems to work but it feels like I am killing an ant with a sledgehammer:
def test_file_for_array_markers(file_path):
with open(file_path, 'r') as file:
try:
for record in ijson.items(file, 'item'):
return isinstance(record, dict)
except ijson.JSONError as e:
return False
While the above may work, I don’t think it is the best way. Running with what Rodrigo said, I am now trying to read the files using:
for record in ijson.items(file, 'item', multiple_values=True):
If I remove the multiple_values flag I get the “trailing garbage” error which is fine. But when I add it, I get nothing at all. I never enter the for loop. I amended the question above to include an actual example of what I am reading.
7
for only parses first element and then returns. Check if all are dicts.
You script return None when there are no elements. No idea if this is correct or not.
def test_file_for_array_markers(file_path):
with open(file_path, 'r') as file:
try:
return all(isinstance(record, dict) for record in ijson.items(file, 'item'))
except ijson.JSONError as e:
pass
return False
2
The “ijson way” to determine if a file is invalid per the above (i.e., whether it contains one or more top-level elements) would be to try to process it fully and check whether errors arise (you can use the basic_parse function since it’s the fastest and you’re not interested in the actual results, only whether it parses or not). If you get no errors, the file is valid as-is.
If you get errors, try again with multiple_values=True. If you get no errors, then it was invalid due to there being multiple top-level values. If you get errors, then it’s invalid because of something else (e.g., wrong syntax somewhere else within each document).
Going through the file twice though is obviously not great. Assuming you not only want to check the validity of the files, but also actually process their data, you’d want to restructure your program in a way that a single pass suffices.
4