Attempting to use JQ to process json data : Elasticsearch bulk index api

Question

This question arrises from the following post:

jq -c -r ".[]" C:\setting-es.json | while read line; do echo '{"index":{}}'; echo $line; done > bulk.json

The above jq shell command is throwing error "Missing statement body in do loop"

I have tried to change syntax around but still it is not working. I am trying to write a shell script to transform the following data for elasticsearch's bulk api:

[{
    "codeId": "111",
    "association": [{
        "associationId": 123,
        "businessUnitsAssociationId": 1,
        "financialBusinessUnits": "DCS",
        "businessApprovalLimit": [{
            "businessApprovalLimitApprovalLimitId": 1,
            "itemMinAmount": "0.00",
            "itemMaxAmount": "0.00"
        }, {
            "businessApprovalLimitApprovalLimitId": 2,
            "itemMinAmount": "0.00",
            "itemMaxAmount": "0.00"
        }, {
            "businessApprovalLimitApprovalLimitId": 3,
            "itemMinAmount": "0.00",
            "itemMaxAmount": "0.00"
        }]
    }]
}]

I am trying to transform it to the following:

{"index":{}}
[{"codeId":"111","association":[{"associationId":123,"businessUnitsAssociationId":1,"financialBusinessUnits":"DCS","businessApprovalLimit":[{"businessApprovalLimitApprovalLimitId":1,"itemMinAmount":"0.00","itemMaxAmount":"0.00",},{"businessApprovalLimitApprovalLimitId":2,"itemMinAmount":"0.00","itemMaxAmount":"0.00",},{"businessApprovalLimitApprovalLimitId":3,"itemMinAmount":"0.00","itemMaxAmount":"0.00",}]}]

values True and False spelled as invalid JSON values (in JSON those spelled lower case entirely) intentionally or mistakenly? — Dmitry
– Dmitry, Commented Nov 17, 2019 at 20:38
@Dmitry i have fixed it but regardless this is just dummy data — cluis92
– cluis92, Commented Nov 17, 2019 at 20:59
@Dmitry how does jtc know how each json data is brokern up? In my dummy data, I am having five data points, how does it know how each is being separated? I ask this because my actual dataset has nested values with each data point separated.. I can share if needed — cluis92
– cluis92, Commented Nov 17, 2019 at 21:02
@cluis92, in such case, the same jtc would work (but -a option could be removed, it's redundant as now it's a single JSON). — Dmitry
– Dmitry, Commented Nov 17, 2019 at 21:27
@cluis92, right, but it's not needed, showing a snippet which would suffice explaining the input concept and allowing building a correct solution is enough. — Dmitry
– Dmitry, Commented Nov 17, 2019 at 21:41

peak · Accepted Answer · 2019-11-17 23:08:40Z

1

Here's an answer to the revised question, after correcting the invalid JSON (i.e., after removing two superfluous commas).

There is still no need for a shell loop.

At a bash or bash-like prompt:

jq -c '.[] | ({"index":{}}, [.])'  input.json

At a Powershell prompt, it might be easier to place the jq program into a file, and invoke jq with the -f FILENAME option.

edited Nov 17, 2019 at 23:08

answered Nov 17, 2019 at 21:31

peak

119k21 gold badges185 silver badges218 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Dmitry Over a year ago

there's more than superfluous commas in the sample json, there's missing curly and square brackets. I have fixed it, but it's pending a review.

cluis92 Over a year ago

@peak so would this jq command be run from a shell script, such as windows powershell? Here is what I tried jq -c '.[] | ({"index":{}}, [.])' jq -c '.[] | ({"index":{}}, [.])' @C:\Users\chris\dummy-json.json and it is saying 'Could not open file jq' in powershell

cluis92 Over a year ago

thank you @peak marked as accepted.. this was what worked for me jq -c '.[] | ({"index":{}}, [.])' activity-es-jq.json > bulk-activity.json

peak · Accepted Answer · 2019-11-17 21:20:35Z

0

[This response was based on the original question.]

There's no need for any shell loop:

$ jq -c '{"index":{}},.' input.json
{"index":{}}
{"str field":"some string","int field":12345,"bool field":true}
{"index":{}}
{"str field":"another string","int field":42,"bool field":false}
{"index":{}}
{"str field":"random string","int field":3856452,"bool field":true}
{"index":{}}
{"str field":"string value","int field":11111,"bool field":false}
{"index":{}}
{"str field":"last string","int field":54321,"bool field":true}

answered Nov 17, 2019 at 21:20

peak

119k21 gold badges185 silver badges218 bronze badges

1 Comment

cluis92 Over a year ago

so the reason I am needing a loop is because my data is much larger than the above and it is not having {"index":{}} as headers between each data point (its technically not valid json, but this is the only way elasticsearch will accept the data..)

Collectives™ on Stack Overflow

Attempting to use JQ to process json data : Elasticsearch bulk index api

2 Answers 2

3 Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related