AWK: execute CURL on each line and parse result

Question

given an input stream with following lines:

I would like to call

curl -s http://foo.bar/some.php?id=xxx

with xxx being the number for each line, and everytime let an awk script fetch some information from the curl output which is written to the output stream. I am wondering if this is possible without using the awk "system()" call in following way:

cat lines | grep "^[0-9]*$" | awk '
    {
        system("curl -s " $0 \
        " | awk \'{ #parsing; print }\'")
    }'

I find this question interesting in situations where the input come from a "real" stream, not a file. For instance a sensing device that generate data (lines) periodically e.g. on /dev/ttyUSB0. In such cases, the proposed answers (reading from a file) are applicable only if storing data from the stream a temporary file and periodically process the file (what if data arrive while processing ?). But a more direct approach (using a pipe) might be better... — Rémi
– Rémi, Commented Nov 19, 2014 at 12:21
Note the cat | grep | awk can be simplified to just awk '/^[0-9]+$/ {}' file. Also, what kind of parsing do you want to do? Maybe it is not necessary to use awk at all and a while read loop suffices. — fedorqui
– fedorqui, Commented Jun 24, 2015 at 8:37

Foo Bah · Accepted Answer · 2011-09-11 05:36:19Z

1

You can use bash and avoid awk system call:

grep "^[0-9]*$" lines | while read line; do
    curl -s "http://foo.bar/some.php?id=$line" | awk 'do your parsing ...'
done

answered Sep 11, 2011 at 5:36

Foo Bah

26.4k5 gold badges58 silver badges82 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Tony · Accepted Answer · 2011-07-18 21:23:29Z

0

A shell loop would achieve a similar result, as follows:

#!/bin/bash
for f in $(cat lines|grep "^[0-9]*$"); do 
    curl -s "http://foo.bar/some.php?id=$f" | awk '{....}'
done

Alternative methods for doing similar tasks include using Perl or Python with an HTTP client.

answered Jul 18, 2011 at 21:23

Tony

171 bronze badge

Comments

user4401178 · Accepted Answer · 2016-03-31 17:40:21Z

0

If your file gets dynamically appended the id's, you can daemonize a small while loop to keep checking for more data in the file, like this:

while IFS= read -d $'\n' -r a || sleep 1; do [[ -n "$a" ]] && curl -s "http://foo.bar/some.php?id=${a}"; done < lines.txt

Otherwise if it's static, you can change the sleep 1 to break and it will read the file and quit when there is no data left, pretty useful to know how to do.

answered Mar 31, 2016 at 17:40

user4401178

Collectives™ on Stack Overflow

AWK: execute CURL on each line and parse result

3 Answers 3

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related