Update one JSON file values with values from another JSON using JQ (on all levels)

Question

I have two JSON files:

source.json:

{
  "general": {
    "level1": {
      "key1": "x-x-x-x-x-x-x-x",
      "key3": "z-z-z-z-z-z-z-z",
      "key4": "w-w-w-w-w-w-w-w"
    },
    "another" : {
      "key": "123456",
      "comments": {
        "one": "111",
        "other": "222"
      }
    }
  },
  "title": "The best"
}

and the

target.json:

{
  "general": {
    "level1": {
      "key1": "xxxxxxxx",
      "key2": "yyyyyyyy",
      "key3": "zzzzzzzz"
    },
    "onemore": {
      "kkeeyy": "0000000"
    }
  },
  "specific": {
    "stuff": "test"
  },
  "title": {
    "one": "one title",
    "other": "other title"
  }
}

I need all the values for keys which exist in both files, copied from source.json to target.json, considering all the levels.
I've seen and tested the solution from this post. It only copies the first level of keys, and I couldn't get it to do what I need. The result from solution in this post, looks like this:

{
  "general": {
    "level1": {
      "key1": "x-x-x-x-x-x-x-x",
      "key3": "z-z-z-z-z-z-z-z",
      "key4": "w-w-w-w-w-w-w-w"
    },
    "another": {
      "key": "123456",
      "comments": {
        "one": "111",
        "other": "222"
      }
    }
  },
  "specific": {
    "stuff": "test"
  },
  "title": "The best"
}

Everything under the "general" key was copied as is.
What I need, is this:

{
  "general": {
    "level1": {
      "key1": "x-x-x-x-x-x-x-x",
      "key2": "yyyyyyyy",
      "key3": "z-z-z-z-z-z-z-z"
    },
    "onemore": {
      "kkeeyy": "0000000"
    }
  },
  "specific": {
    "stuff": "test"
  },
  "title": {
    "one": "one title",
    "other": "other title"
  }
}

Only "key1" and "key3" should be copied.
Keys in target JSON must not be deleted and new keys should not be created.

Can anyone help?

Good that you have shown your research and attempt to solve the problem. But you should include the actual code you have tried to solve your problem. Linking to another Q is OK for context, but Qs should be self contained. Good luck.\ — shellter
– shellter, Commented Oct 17, 2018 at 15:49

Jeff Mercado · Accepted Answer · 2018-10-17 17:17:16Z

1

One approach you could take is get all the paths to all scalar values for each input and take the set intersections. Then copy values from source to target from those paths.

First we'll need an intersect function (which was surprisingly difficult to craft):

def set_intersect($other):
    (map({ ($other[] | tojson): true }) | add) as $o
    | reduce (.[] | tojson) as $v ({}; if $o[$v] then .[$v] = true else . end)
    | keys_unsorted
    | map(fromjson);

Then to do the update:

$ jq --argfile s source.json '
reduce ([paths(scalars)] | set_intersect([$s | paths(scalars)])[]) as $p (.;
    setpath($p; $s | getpath($p))
)
' target.json

answered Oct 17, 2018 at 17:17

Jeff Mercado

136k34 gold badges268 silver badges282 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

peak · Accepted Answer · 2018-10-18 09:57:54Z

1

[Note: this response answers the original question, with respect to the original data. The OP may have had paths in mind rather than keys.]

There is no need to compute the intersection to achieve a reasonably efficient solution.

First, let's hypothesize the following invocation of jq:

jq -n --argfile source source.json --argfile target target.json -f copy.jq

In the file copy.jq, we can begin by defining a helper function:

# emit an array of the distinct terminal keys in the input entity
def keys: [paths | .[-1] | select(type=="string")] | unique;

In order to inspect all the paths to leaf elements of $source, we can use tostream:

($target | keys) as $t
| reduce ($source|tostream|select(length==2)) as [$p,$v]
    ($target;
     if $t|index($p[-1]) then setpath($p; $v) else . end)

Alternatives

Since $t is sorted, it would (at least in theory) make sense to use bsearch instead of index:

 bsearch($p[-1]) > -1

Also, instead of tostream we could use paths(scalars).

Putting these alternatives together:

($target | keys) as $t
| reduce ($source|paths(scalars)) as $p
    ($target;
     if $t|bsearch($p[-1]) > -1 
     then setpath($p; $source|getpath($p))
     else . end)

Output

{
  "general": {
    "level1": {
      "key1": "x-x-x-x-x-x-x-x",
      "key2": "yyyyyyyy",
      "key3": "z-z-z-z-z-z-z-z"
    },
    "onemore": {
      "kkeeyy": "0000000"
    }
  },
  "specific": {
    "stuff": "test"
  }
}

edited Oct 18, 2018 at 9:57

answered Oct 18, 2018 at 0:20

peak

119k21 gold badges185 silver badges218 bronze badges

2 Comments

wsa Over a year ago

I updated the question with better JSON examples, to include cases which were not working well with this solution, leading to deleted keys or new keys being added to the target file

peak Over a year ago

Your update suggests your requirements should refer to paths rather than keys. Please clarify.

peak · Accepted Answer · 2018-10-18 11:48:10Z

0

The following provides a solution to the revised question, which is actually about "paths" rather than "keys".

([$target|paths(scalars)] | unique) as $paths
| reduce ($source|paths(scalars)) as $p
    ($target;
     if $paths | bsearch($p) > -1 
     then setpath($p; $source|getpath($p))
     else . end)

unique is called so that binary search can be used subsequently.

Invocation:

jq -n --argfile source source.json --argfile target target.json -f program.jq

edited Oct 18, 2018 at 11:48

answered Oct 18, 2018 at 11:26

peak

119k21 gold badges185 silver badges218 bronze badges

Collectives™ on Stack Overflow

Update one JSON file values with values from another JSON using JQ (on all levels)

3 Answers 3

Comments

Alternatives

Output

2 Comments

Invocation:

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Alternatives

Output

2 Comments

Invocation:

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related