Using CURL to download file and view headers and status code

Question

I'm writing a Bash script to download image files from Snapito's web page snapshot API. The API can return a variety of responses indicated by different HTTP response codes and/or some custom headers. My script is intended to be run as an automated Cron job that pulls URLs from a MySQL database and saves the screenshots to local disk.

I am using curl. I'd like to do these 3 things using a single CURL command:

Extract the HTTP response code
Extract the headers
Save the file locally (if the request was successful)

I could do this using multiple curl requests, but I want to minimize the number of times I hit Snapito's servers. Any curl experts out there?

Or if someone has a Bash script that can respond to the full documented set of Snapito API responses, that'd be awesome. Here's their API documentation.

Thanks!

You could parse the output, but I guess this is easier in python/ruby/php/... — Karoly Horvath
– Karoly Horvath, Commented Aug 6, 2012 at 21:38

WA Hunt · Accepted Answer · 2012-08-06 21:39:05Z

33

Use the dump headers option: curl -D /tmp/headers.txt http://server.com

answered Aug 6, 2012 at 21:39

WA Hunt

5854 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

curtisdf Over a year ago

Ah, yes. I had overlooked the -D option. Here's what I'm using now: curl -s -L -D /dev/stdout -o $IMAGE_FILE $SNAPITO_URL Thanks!

Darren Bishop Over a year ago

curl -s -L -D - -o $IMAGE_FILE $SNAPITO_URL

Shakiba Moshiri · Accepted Answer · 2018-09-30 07:15:50Z

13

Use curl -i (include HTTP header) - which will yield the headers, followed by a blank line, followed by the content.

You can then split out the headers / content (or use -D to save directly to file, as suggested above).

There are three options -i, -I, and -D

> curl --help | egrep '^ +\-[iID]'
 -D, --dump-header FILE  Write the headers to FILE
 -I, --head          Show document info only
 -i, --include       Include protocol headers in the output (H/F)

edited Sep 30, 2018 at 7:15

Shakiba Moshiri

24.7k3 gold badges42 silver badges50 bronze badges

answered Aug 6, 2012 at 21:40

Guy Adini

5,5245 gold badges35 silver badges35 bronze badges

3 Comments

curtisdf Over a year ago

I think you were thinking of the -I option, not -i.

Guy Adini Over a year ago

Nope. -I is headers only, -i is headers + content.

curtisdf Over a year ago

I see, you're right. Using -i would have given me another problem though: Parsing the output to separate out the headers from the binary image data. I got it working nicely with the -D option routed to stdout (and captured by a $() wrapper), and the -o option for saving the file. Thanks though! I'll remember to look into -i next time.

Collectives™ on Stack Overflow

Using CURL to download file and view headers and status code

2 Answers 2

2 Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related