append output of multiple curl requests to a file from shell script

Question

I'm trying to fetch the JSON output by an internal API and add 100 to a parameter value between cURL requests. I need to loop through because it restricts the maximum number of results per request to 100. I was told to "increment and you should be able to get what you need".

Anyway, here's what I wrote:

#!/bin/bash

COUNTER=100
until [ COUNTER -gt 30000 ]; do
    curl -vs "http://example.com/locations/city?limit=100&offset=$COUNTER" >> cities.json
    let COUNTER=COUNTER+100
done

The problem is that I get a bunch of weird messages in the terminal and the file I'm trying to redirect the output too still contains it's original 100 objects. I feel like I'm probably missing something terrifically obvious. Any thoughts? I did use a somewhat old tutorial on the until loop, so maybe it's a syntax issue?

Thank you in advance!

EDIT: I'm not opposed to a completely alternate method, but I had hoped this would be somewhat straightforward. I figured my lack of experience was the main limiter.

Why would it not contain its original content? You're only ever appending. — Charles Duffy
– Charles Duffy, Commented Mar 23, 2015 at 20:22
With respect to "weird messages to the terminal" -- everything going to curl's stderr is going to be printed to the terminal, and because you're using curl -v, that's the behavior you're asking for. If you want different messages printed by curl, I'd suggest deciding what level of verbosity you actually want. — Charles Duffy
– Charles Duffy, Commented Mar 23, 2015 at 20:25
@CharlesDuffy I think I was a little imprecise in my original comment. I meant that nothing had been appended when I halted the process - the contents were still the original 100 objects in the file from the first time I used cURL. I stopped the script after a few minutes because the requests were hanging. I also was wrong about the options I was using (I thought -s was supposed to suppress some of the stdout & stderr), so that explains messages in the terminal. I have no formal training, so this is sort of new for me. — jonathanopenshaw
– jonathanopenshaw, Commented Mar 23, 2015 at 22:30

Charles Duffy · Accepted Answer · 2015-03-23 20:23:33Z

2

If you want to overwrite the file's content only once, for your entire loop...

#!/bin/bash
# ^-- NOT /bin/sh, as this uses bash-only syntax

for (( counter=100; counter<=30000; counter+=100 )); do
    curl -vs "http://example.com/locations/city?limit=100&offset=$counter"
done >cities.json

This is actually more efficient than putting >>cities.json on each curl command, as it only opens the output file once, and has the side effect (which you appear to want) of clearing the file's former contents when the loop is started.

answered Mar 23, 2015 at 20:23

Charles Duffy

299k43 gold badges441 silver badges497 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Mark Setchell · Accepted Answer · 2015-03-23 23:03:14Z

1

You might find you can do this faster, and pretty easily with GNU Parallel:

parallel -k curl -vs "http://example.com/locations/city?limit=100\&offset={}" ::: $(seq 100 100 30000) > cities.json

answered Mar 23, 2015 at 23:03

Mark Setchell

210k32 gold badges310 silver badges504 bronze badges

4 Comments

jonathanopenshaw Over a year ago

I had no experience with parallel until now - thank you! Is this more efficient for loops in your experience? Also, any downsides to using parallel instead of a shell script?

Mark Setchell Over a year ago

GNU Parallel is good when you have lots of things to do in parallel, like stripping the metadata from 60,000 images or when you have lots of things to do that have high latency - like curl or ping where you can hide the latency of 7 things behind the latency of the 8th and get a better performance because all those latencies happen in parallel. For essentially sequential things, it is no better. It is an invaluable tool for your toolbox though and pretty quick to give it a try for many problems - so suck it and see !

Mark Setchell Over a year ago

You may find that you can add -j 16 after the -k to run 16 curls in parallel...

Charles Duffy Over a year ago

BTW, usually I tend to argue that xargs -P is a better choice than parallel (simpler, fewer moving parts, better separation of concerns), but this is a case where that's not true since you need to keep the output collated -- can't have part of one json file mixed with part of another.

Collectives™ on Stack Overflow

append output of multiple curl requests to a file from shell script

2 Answers 2

Comments

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related