Here is my current code in bash (based on https://redis.io/docs/latest/develop/use/patterns/bulk-loading):
head=$(redis-cli -h $redis_server get .git-head)
if [[ ! $head ]]; then
redis-cli -h $redis_server flushdb
for fileOrFolder in $(ls -1); do
time {
find $fileOrFolder -type f |
LC_ALL=C xargs -n1 bash -c 'echo -e *3\r\n$3\r\nSET\r\n$${#0}\r\n$0\r\n$$(stat -c%s $0)\r && cat $0 && echo -e \r' |
redis-cli -h $redis_server --pipe
}
done
redis-cli -h $redis_server set .git-head $(git rev-parse HEAD)
fi
And it works, but there is a problem – the values are of the type string, not JSON. At least this is what Redis Insights tells me.
My question is – how can I modify the code to make sure the stored values are of the JSON type and keep the bulk load performance? Or even – should I modify it at all? Maybe the string type is good enough and would not limit our query abilities.
EDIT 1
To emphasize – the files are all JSON files already. To store them as JSON would mean to store them exactly as they are, only indicate to Redis that the values are valid JSONs.
If you don’t need to perform complex queries on the data itself, then using strings is fine.
If you still want to store in JSON type, you can use JSON.SET command instead of SET
You can modify your code as below:
time {
find $fileOrFolder -type f |
LC_ALL=C xargs -n1 bash -c 'file_content=$(cat $0 | jq -Rs .);
file_size=$(stat -c%s $0);
redis-cli -h $redis_server json.set $0 . "{"file":"$0","size":"$file_size","content":$file_content}"'
}
5