jq - can't append objects to array in while loop

Question

I've got this service-accounts-list.txt file with service accounts:

serviceAccount:[email protected]
serviceAccount:[email protected]
serviceAccount:[email protected]
serviceAccount:[email protected]
serviceAccount:[email protected]

and this results.json file which I'd like to append data to:

{
  "access": [
    {
      "role": "WRITER",
      "specialGroup": "projectWriters"
    },
    {
      "role": "OWNER",
      "specialGroup": "projectOwners"
    },
    {
      "role": "READER",
      "specialGroup": "projectReaders"
    }
  ]
}

Expected result: I'd like to add an object for each line in service-accounts-list.txt, like so:

{
  "access": [
    {
      "role": "WRITER",
      "specialGroup": "projectWriters"
    },
    {
      "role": "OWNER",
      "specialGroup": "projectOwners"
    },
    {
      "role": "READER",
      "specialGroup": "projectReaders"
    },
    {
      "role": "WRITER",
      "userByEmail": "serviceAccount:[email protected]"
    },
    {
      "role": "WRITER",
      "userByEmail": "serviceAccount:[email protected]"
    },
    {
      "role": "WRITER",
      "userByEmail": "serviceAccount:[email protected]"
    },
    {
      "role": "WRITER",
      "userByEmail": "serviceAccount:[email protected]"
    },
    {
      "role": "WRITER",
      "userByEmail": "serviceAccount:[email protected]"
    }
  ]
}

My bash script so far looks like this:

while read p;
do
  cat result.json | jq --arg email "$p" '.access += [{"role": "WRITER", "userByEmail": $email}]' > result.json
done < service-accounts-list.txt

The resulting result.json file is actually empty. If I redirect the output to for example > result2.json it correctly adds the last service account, like so:

{
  "access": [
    {
      "role": "WRITER",
      "specialGroup": "projectWriters"
    },
    {
      "role": "OWNER",
      "specialGroup": "projectOwners"
    },
    {
      "role": "READER",
      "specialGroup": "projectReaders"
    },
    {
      "role": "WRITER",
      "userByEmail": "serviceAccount:[email protected]"
    }
  ]
}

so it seems the jq syntax is correct. I've tried adding the --unbuffered flag to no avail.

What am I missing?

For your convenience, ShellCheck automatically detects this and other common problems. — that other guy
– that other guy, Commented May 11, 2020 at 19:04
Thanks a lot for the tip. That looks useful for a bash scripting newbie like me — LundinCast
– LundinCast, Commented May 12, 2020 at 18:36

peak · Accepted Answer · 2020-05-12 06:41:16Z

3

Maybe the simplest efficient solution would be:

jq -nR --argfile result result.json '
  $result 
  | .access += [{role: "WRITER", userByEmail: inputs}] 
' service-accounts-list.txt

Notice -- no slurping, no splitting, no shell scripting. If the use of --argfile bothers you, feel free to use --slurpfile but "unslurp" $result by writing $result[0].

edited May 12, 2020 at 6:41

answered May 11, 2020 at 21:58

peak

119k21 gold badges185 silver badges218 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

anubhava · Accepted Answer · 2020-05-11 19:33:08Z

While obvious problem is that you are reading and redirecting to same file which you cannot do in bash and this answer has many useful explanations and work around for that.

But a better solution would be to totally avoid invoking jq multiple times in a loop and get it done in a single execution of jq.

# array to hold new values in json format
narr=()

# loop through input file and append json formatted values to array
while read -r line; do
    narr+=('{"role": "WRITER", "userByEmail": "'$line'"}')
done < service-accounts-list.txt

# call jq only once using -n option
jq -n 'input | .access += [inputs]' results.json <(printf '%s\n' "${narr[@]}")

Output:

{
  "access": [
    {
      "role": "WRITER",
      "specialGroup": "projectWriters"
    },
    {
      "role": "OWNER",
      "specialGroup": "projectOwners"
    },
    {
      "role": "READER",
      "specialGroup": "projectReaders"
    },
    {
      "role": "WRITER",
      "userByEmail": "serviceAccount:[email protected]"
    },
    {
      "role": "WRITER",
      "userByEmail": "serviceAccount:[email protected]"
    },
    {
      "role": "WRITER",
      "userByEmail": "serviceAccount:[email protected]"
    },
    {
      "role": "WRITER",
      "userByEmail": "serviceAccount:[email protected]"
    },
    {
      "role": "WRITER",
      "userByEmail": "serviceAccount:[email protected]"
    }
  ]
}

Inian · Accepted Answer · 2020-05-11 20:09:25Z

2

You can use jq itself without any shell processing. Process the plain text file using raw input mode -R over null input and start forming the resultant JSON from scratch.

The reduce() function iteratively runs over each entry int the text file and creates a object with role name and appends it to the access array.

jq -Rn \
   --slurpfile res result.json \
   '[ inputs | split("\n") | add ] as $data 
    | $res[] 
    | reduce $data[] as $d (.; .access += [{ role: "WRITER", userByEmail: $d  }] )' \
service-accounts-list.txt

answered May 11, 2020 at 20:09

Inian

87k15 gold badges166 silver badges183 bronze badges

Collectives™ on Stack Overflow

jq - can't append objects to array in while loop

3 Answers 3

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related