0

I have this command line:

curl https://token-api/42 | awk '{ print $0 }'

which produces this output:

{"id":"PX-12345","expiresAtUtc":"2020-07-05T06:55:52","createdUtc":"2019-07-15T06:55:52"}

The awk '{ print $0 }' is there to ensure there is a newline appended at the end of the output.

Now I want to do the same but for a csv list of user ids (users.csv) instead of a hard-coded 42 by using xargs. So I tried this and a number of variations:

awk -F, '{print "curl https://token-api/$0 | awk \'''\''{ print $0 }\'''\''"}' < users.csv | xargs -P1 -I {} sh -c '{}'

This produces the following error:

awk: cmd. line:1: warning: escape sequence `\'' treated as plain `''
awk: cmd. line:1: {
awk: cmd. line:1:  ^ unexpected newline or end of string

If I remove the xargs part, I get the following output:

curl https://token-api/$0 | awk '{ print $0 }'

which is passed to xargs. I guess my problem is with the $0 that is used twice but I have no idea to escape this.

If there is another (better) way to append a newline to the output of curl, that might help too because without that it's working fine. However, then I get a long list of unseparated JSON documents as output like this:

{"id":"PX-12345","expiresAtUtc":"...","createdUtc":"..."}{"id":"PX-12346","expiresAtUtc":"...","createdUtc":"..."}{"id":"PX-12347","expiresAtUtc":"...","createdUtc":"..."}
3
  • 1
    what do you want to achieve exactly? just append a line to the output of curl? { curl http://foo.com/bar; echo; }. Commented Jul 15, 2019 at 7:20
  • What does your users.csv look like? Commented Jul 15, 2019 at 7:31
  • @mosvy the 'trick' with echo works, I was making it too complex I guess. Thanks!! Commented Jul 15, 2019 at 9:00

1 Answer 1

1

Rather than executing a command for every line of the csv file, it might be easier to take advantage of curls multiple URL feature.

You can specify multiple URLs or parts of URLs by writing part sets within braces as in: http://site.{one,two,three}.com

I'm not sure what your csv file looks like, seems like it's all in a single line? If so you could just cat the file and combine with the curl command.

curl "https://token-api/{$(cat users.csv)}"

Still needs a bit of cleaning up.

curl "https://token-api/{$(cat users.csv)}" 2>/dev/null | sed '1d;s/--_curl_--.*$//'; echo

The sed command deletes the first line since that will just be the first URL, then removes the rest of the URLs, which leaves a useful newline between entries. Finally just echo to end with a newline.

2
  • As usual, I made my example code too simple. The request is actually a POST request where the data posted comes from the csv file. So I can't use curls multiple URL feature unfortunately. Commented Jul 15, 2019 at 13:07
  • That makes more sense, I think if it's getting that complex it might be better making a shell script rather than trying to get it all into a single command. Maybe something like while IFS=, read -r USERID DATA1 _ DATA2 _; do; curl "https://token-api/$USERID" -d "data1=$DATA1&data2=$DATA2"; echo; done < users.csv Commented Jul 16, 2019 at 3:58

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.