curl: (3) URL using bad/illegal format or missing URL in bash Windows

Question

I am trying to download PDF files from a list of URLs in a .txt file, with one URL per line. ('urls.txt')

When I use the following command, where the URL I used is an exact copy-paste of the first line of the .txt file:

$ curl http://www.isuresults.com/results/season1617/gpchn2016/gpchn2016_protocol.pdf -o 'test.pdf'

the pdf downloads perfectly. However when I use this command:

xargs -n 1 curl -O < urls.txt

Then I receive a 'curl: (3) URL using bad/illegal format or missing URL' error x times the amount of URLs listed in the .txt file. I have tested many of the URLS individually, and they all seem to download properly.

How can I fix this?

Edit - the first three lines of urls.txt reads as follows:

http://www.isuresults.com/results/season1718/gpf1718/gpf2017_protocol.pdf
http://www.isuresults.com/results/season1718/gpcan2017/gpcan2017_protocol.pdf
http://www.isuresults.com/results/season1718/gprus2017/gprus2017_protocol.pdf

SOLVED: As per the comment below, the issue was that the .txt file was in DOS/Windows format. I converted it using the line:

$ dos2unix urls.txt

and then the files downloaded perfectly using my original line of code. See this thread for more info: Are shell scripts sensitive to encoding and line endings?

Thank you to all who responded!

It sounds like your urls.txt file is in DOS/Windows format, which will confuse unix tools. See "Are shell scripts sensitive to encoding and line endings?" (especially Anthony Geoghegan's answer). — Gordon Davisson
– Gordon Davisson, Commented Jan 11, 2022 at 2:27

Ian Kenney · Accepted Answer · 2022-01-11 03:33:12Z

1

Try using

xargs -n 1 -t -a urls.txt curl -O

here the -a option reads list from a file rather than standard input

EDIT:

As @GordonDavisson mentioned, it looks like you may have a file with DOS line endings, you can potentially clean these up using sed before passing to xargs

sed 's/\r//' < urls.txt | xargs -n 1 -t curl -O

edited Jan 11, 2022 at 3:33

answered Jan 11, 2022 at 1:49

Ian Kenney

6,4561 gold badge27 silver badges45 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Nadezh Over a year ago

curl -O 'isuresults.com/results/season1617/gpchn2016/…'\r' curl: (3) URL using bad/illegal format or missing URL

Nadezh Over a year ago

x times the number of URLs in the text file

Ian Kenney Over a year ago

can you update the question with (say the first 3) example urls from urls.txt.

Collectives™ on Stack Overflow

curl: (3) URL using bad/illegal format or missing URL in bash Windows

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related