8

I am trying to merge all json files in a directory (can be many).

I know that if there are only 2 files, I can use jq -s . file1.json file2.json. But when I tried to use the a wild card like jq -s . file*.json, it fails. Any idea?

2

5 Answers 5

10

Try this:

jq -s add *.json > merged.json
Sign up to request clarification or add additional context in comments.

1 Comment

This only merges the top level, see the next answer
6

From what I've seen, jq -s add *.json merges objects only at the top level (i.e. not recursively), and jq -s '.' *.json creates an array with each element being one of the JSON files.

The command Ouss wrote just hung for me, so I don't know about that.

I ended up using the following, which simple enough and worked for me to perform a recursive merge of multiple JSON files:

jq -s 'reduce .[] as $obj ({}; . * $obj)' *.json

1 Comment

This should be the correct answer. It works perfectly with jq-1.7.
2

My JSON files contain JSON object. jq -s add *.json did not work, but the following works

jq -s '.' *.json > merged.json

Comments

1

The answer Here is the jq answer... for more see the explanation below:

s="*.json"; jq -s "`x=-1; n=$(ls $s | wc -l); while [ $((x++)) -lt $(($n-2)) ]; do printf ".[$x] * " ; done; printf ".[$(($n-1))]";`" $s

Step by step explanation

To merge two json files using jq use this command:

jq -s ".[0] * .[1]" file1.json file2.json

Now to merge three json files:

jq -s ".[0] * .[1] * .[2]" file1.json file2.json file3.json

Let's follow the same logic and generalise for n files using wildcards. The difficult part is to generate the string .[0] * .[1] ... .[n-1].

First let's find n. n is the number of json files in the folder. so:

ls *.json | wc -l

now let's write a loop that will build the string ".[1] * .. * .[a] * .[a+1] * .. * .[n-2] *". The kernel of this string is .[a] * so:

x=-1; n=$(ls *.json | wc -l); while [ $((x++)) -lt $(($n-2)) ]; do printf ".[$x] * " ; done;

ok and now let's add the last part .[n-1]:

x=-1; n=$(ls *.json | wc -l); while [ $((x++)) -lt $(($n-2)) ]; do printf ".[$x] * " ; done; printf ".[$(($n-1))]";

now let's pass that string to the jq command... and let's parameterise *.json so that we can replace it with whatever we want with ease:

s="*.json"; jq -s "`x=-1; n=$(ls $s | wc -l); while [ $((x++)) -lt $(($n-2)) ]; do printf ".[$x] * " ; done; printf ".[$(($n-1))]";`" $s

et voilà!

Comments

0

While waiting a jq's solution, let me address the question "... or any tool?"

Different people mean different things by "merging", thus assuming here a plain recursive JSON merging, then with jtc (another tool) the solution would be like this (for an arbitrary number of files):

  • read all the JSONs into one super JSON array
  • merge into the 1st child (the first read JSON) the rest of read JSONs
  • output the resulting 1st JSON child in the array
jtc -J / -w[0] -mi[1:] / -w[0] *.json 

to illustrate the solution, here's how it works with couple arbitrary JSON files:

bash $ jtc -tc file1.json 
{
   "days": {
      "2020-05-18": { "seconds": 60 },
      "2020-05-19": { "seconds": 30 },
      "2020-05-20": { "seconds": 1400 }
   },
   "total": { "seconds": 1490 }
}
bash $ jtc -tc file2.json 
{
   "days": {
      "2020-05-20": { "seconds": 95 },
      "2020-05-21": { "seconds": 80 },
      "2020-05-22": { "seconds": 120 }
   },
   "total": { "seconds": 295 }
}
bash $ jtc -J / -w[0] -mi[1:] / -w[0] -tc file*.json
{
   "days": {
      "2020-05-18": { "seconds": 60 },
      "2020-05-19": { "seconds": 30 },
      "2020-05-20": {
         "seconds": [ 1400, 95 ]
      },
      "2020-05-21": { "seconds": 80 },
      "2020-05-22": { "seconds": 120 }
   },
   "total": {
      "seconds": [ 1490, 295 ]
   }
}
bash $ 

-tc flag is used only to display JSONs here in a compact format

PS. It so happens that I'm the creator of jtc tool - unix JSON processing utility.

14 Comments

I am sorry that I did not post my question clearly. The jsons are all with the same format, just with different data. I need to run the tool on windows as our servers are all windows.
Does jtc have a windows version? I could not find it so far.
The commands of some of the "answers" from the various threads here in stackoverflow or stackexchange do not work. For example, I used command jq -s '{ attributes: map(.attributes[0]) }' *.json. But jq throws an error "This application has requested the Runtime to terminate it in an unusual way. Please contact the application's support team for more information."
If I used the simple command "jq.exe -s . *.json", I get exactly the same error thrown by jq. Not sure if jq supports wildcard on windows platform? If I only passed 2 files, jq does work properly.
No, there's no native version of jtc for widows yet. But it's easily compilable under cygwin environment
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.