I have multiple analysis results in different json files in different folders. The experiment implied that 80 subjects underwent 7 tests, and the results of each test (between 5 to 15 values per test per subject) are recorded in one json file.

I have to analyze them with either Python or R.

1. I do not know if it is best to combine them in one big json, or to manipulate them in list (or in other ways)?
2. And if so, what is the most straightforward way to extract variables in these many json files?

I tried to merge all json files with json.dump but I realized that the merged json file is a mess. Some relevant informations, such as the ID of the subjects are not contained in the json, but in the filepath (e.g. ID of the subjects and folder name), which adds a problem.

The json files have the same structure when they report the result of the same test (i.e., all TEST1 json files are similar). But they differ between the test. And as you can see, they are no ID of the subject and no info of the name of the test in it. I was thinking that maybe it would be useful to extract it from the path and to add it in the json file as a key, to “group” the information of all the json for one subject, and then group all the subjects? I’m not sure if it is useful.

*_*result/S01/S01_TEST1.json

_result/S01/S01_TEST2.json

etc.

I tried (the stuff in comment was an attempt to include the subjects and test name in the merged file but it failed!):

import sys
import json
import glob
from pathlib import Path
import tools



def merge_json_files(directory_path):
    merged_data = []
    file_paths = glob.glob(directory_path + '/*/*.json')
    for path in file_paths:
        # SUBJECT = os.path.split(os.path.split(path)[0])[1]
        # FILENAME = Path(path).stem
        # SESSION_NAME=FILENAME[4::]
        # ID_INFO = {'SUJ':SUBJECT,'SESSION':SESSION_NAME}
        # tools.append_to_json_file(path,  ID_INFO)
        with open(path, 'r') as file:
            data = json.load(file)

            merged_data.append(data)
    return merged_data
 
 
directory_path = "/_results/S01"
output_file = "merged.json"
merged_data = merge_json_files(directory_path)
with open(output_file, 'w') as outfile:
    json.dump(merged_data, outfile)
print(merged_data)

a part of json file example here for Test1:

[
    {
        "variable1": 97.5618361882855,
        "variable2": 97.34192871353984,
    },
    {
        "SVM_acc_cross_valid": {
        "All": 0.9166666666666666,
        "Right": 0.9166666666666666        
        }
    }
]