I have a large JSON file that is an object of objects, which I would like to split into separate files name after object keys. Is it possible to achieve this using jq or any other off-the-shelf tools?
The original JSON is in the following format
{ "item1": {...}, "item2": {...}, ...}
Given this input I would like to produce files item1.json, item2.json etc.
This should give you a start:
for f in `cat input.json | jq -r 'keys[]'` ; do
cat input.json | jq ".$f" > $f.json
done
or when you insist on more bashy syntax like some seem to prefer:
for f in $(jq -r 'keys[]') ; do
jq ".[\"$f\"]" < input.json > "$f.json"
done < input.json
cat file | command is anti-pattern, backticks is another one
jq --arg f "$f" '.[$f]' would be a further improvement, preventing arbitrary code injection (granted, not a big security issue until jq gets a system() call, or a way to open files for write, or other facilities that it doesn't currently have) by ensuring that the substituted value can only ever be treated as a literal string.jq -n --argjson f "$f" '$f' will also guarantee a good parsing; -n is important if your not pipingHere's a solution that requires only one call to jq:
jq -cr 'keys[] as $k | "\($k)\n\(.[$k])"' input.json |
while read -r key ; do
read -r item
printf "%s\n" "$item" > "/tmp/$key.json"
done
It might be faster to pipe the output of the jq command to awk, e.g.:
jq -cr 'keys[] as $k | "\($k)\t\(.[$k])"' input.json |
awk -F\\t '{ print $2 > "/tmp/" $1 ".json" }'
Of course, these approaches will need to be modified if the key names contain characters that cannot be used in filenames.
Is it possible to achieve this using jq or any other off-the-shelf tools?
It is. The command-line JSON-parser xidel can help out.
Let's assume 'input.json':
{
"item1": {
"a": 1
},
"item2": {
"b": 2
},
"item3": {
"c": 3
}
}
You could use Bash, as the other answers lay out...
$ for f in $(xidel -s "input.json" -e '$json()'); do
xidel -s "input.json" --variable f="$f" -e '$json($f)' > "$f.json"
done
$ xidel -s "input.json" -e '$json() ! (.,serialize($json(.),{"method":"json"}))' |
while read -r key ; do read -r item; printf "%s\n" "$item" > "$key.json"; done
...but with the integrated EXPath File Module Xidel can also do this very efficiently:
$ xidel -s "input.json" -e '
$json() ! file:write(`{.}.json`,$json(.),{"method":"json"})
'
$ xidel -s item[123].json -e '$raw'
{"a":1}
{"b":2}
{"c":3}
$ xidel -s "input.json" -e '
$json() ! file:write(`{.}.json`,$json(.),{"method":"json","indent":true()})
'
$ xidel -s item[123].json -e '$raw'
{
"a": 1
}
{
"b": 2
}
{
"c": 3
}
