Merging multiple JSON Lines files into a single JSON object

As @peak has pointed out, some aspects are underspecified. For instance, what should happen with .customerId and .id? Are they always the same across all files (as suggested by the sample files provided)? Do you want the items of the .emails array just thrown into one large array, or do you want to have them "merged" by some criteria (e.g. by a common value in their .address field)? Here are some stubs to start from:

• Simply concatenate the .emails arrays and take all other parts from the first file:

jq 'reduce inputs as $in (.; .emails += $in.emails)' file*.json

# or simpler
jq '.emails += [inputs.emails[]]' file*.json

Demo Demo

{
  "emails": [
    {
      "address": "[email protected]"
    },
    {
      "address": "[email protected]",
      "customType": "",
      "type": "custom"
    },
    {
      "address": "[email protected]"
    },
    {
      "address": "[email protected]",
      "primary": true
    },
    {
      "address": "[email protected]"
    },
    {
      "address": "[email protected]"
    },
    {
      "address": "[email protected]",
      "primary": true
    },
    {
      "address": "[email protected]"
    }
  ],
  "customerId": "xx",
  "id": "654"
}

• Merge the objects in the .emails array by a common value in their .address field, with latter values overwriting former values for other fields with colliding names, and discard all other parts from the files:

jq -n 'reduce inputs.emails[] as $e ({}; .[$e.address] += $e) | map(.)' file*.json

Demo

[
  {
    "address": "[email protected]"
  },
  {
    "address": "[email protected]",
    "customType": "",
    "type": "custom"
  },
  {
    "address": "[email protected]"
  },
  {
    "address": "[email protected]",
    "primary": true
  },
  {
    "address": "[email protected]"
  }
]

• If you are only interested in a list of unique field names for a given address, regardless of the counts and values used, you can also go with:

jq -n '
  reduce inputs.emails[] as $e ({}; .[$e.address][$e | keys_unsorted[]] = 1)
  | map_values(keys)
'

Demo

{
  "[email protected]": [
    "address"
  ],
  "[email protected]": [
    "address",
    "customType",
    "type"
  ],
  "[email protected]": [
    "address"
  ],
  "[email protected]": [
    "address",
    "primary"
  ],
  "[email protected]": [
    "address"
  ]
}

The structure of the objects contained in file1.json is unknown, so the solution must be agnostic of any keys/values and the solution must not assume any structure or depth.

You can use the --stream flag to break down the structure into an array of paths and values, discard the values part and make the paths unique:

jq --stream -nc '[inputs[0]] | unique[]' file*.json

["customerId"]
["emails"]
["emails",0,"address"]
["emails",0,"customType"]
["emails",0,"primary"]
["emails",0,"type"]
["emails",1,"address"]
["emails",2]
["emails",2,"address"]
["emails",2,"primary"]
["emails",3]
["emails",3,"address"]
["id"]

Trying to build a representation of this, similar to any of the input files, comes with a lot of caveats. For instance, how would you represent in a single structure if one file had .emails as an array of objects, and another had .emails as just an atomic value, say, a string. You would not be able to represent this plurality without introducing new, possibly ambiguous structures (e.g. putting all possibilities into an array).

Therefore, having a list of paths could be a fair compromise. Judging by your desired output, you want to focus more on the object structure, so you could further reduce complexity by discarding the array indices. Depending on your use case, you could replace them with a single value to retain the information of the presence of an array, or discard them entirely:

jq --stream -nc '[inputs[0] | map(numbers = 0)] | unique[]' file*.json

["customerId"]
["emails"]
["emails",0]
["emails",0,"address"]
["emails",0,"customType"]
["emails",0,"primary"]
["emails",0,"type"]
["id"]

jq --stream -nc '[inputs[0] | map(strings)] | unique[]' file*.json

["customerId"]
["emails"]
["emails","address"]
["emails","customType"]
["emails","primary"]
["emails","type"]
["id"]

The following program meets these two key requirements:

1. "all keys from input objects must be present in output object";
2. "the solution must be agnostic of any keys/values and the solution must not assume any structure or depth."

The approach is the same as one suggested by @pmf, and for the example given in the Q, produces results that are very similar to the one that is shown:

jq -n --stream '
  def squish: map(select(type == "string"));
  reduce (inputs | select(length==2)) as [$p, $v] ({};
    setpath($p|squish; $v))
'

With the given input, this produces:

{
  "customerId": "xx",
  "emails": {
    "address": "[email protected]",
    "customType": "",
    "type": "custom",
    "primary": true
  },
  "id": "654"
}