Using a Bash variable in URL results in "curl: (3) Illegal characters found in URL"

Question

Problem

I'm trying to use a variable when calling a cURL command but it's including the the literal $line instead of the actual value.

while read line; do
  curl "https://x.com/v1/PhoneNumbers/$line?Type=carrier" -u "x:x" 
done < "${1:-/dev/stdin}"

Context

I'm passing a list of numbers to the script trying to read them line by line.

What output do you expect? $line to be translated to a shell variable value or something else? — Chen A.
– Chen A., Commented Aug 27, 2017 at 11:57
if you go on twilio.com/lookup you see the command there I'm simply trying to make it read from a file a bunch of phone numbers — Linda Roesch
– Linda Roesch, Commented Aug 27, 2017 at 12:14

Mateusz Piotrowski · Accepted Answer · 2017-08-27 13:29:09Z

4

From the comments we know that you're getting the following error:

curl: (3) Illegal characters found in URL

If formatted this way:

while IFS="$IFS"$'\r' read line; do
  curl "https://x.com/v1/PhoneNumbers/$line?Type=carrier" -u "x:x" 
done < "${1:-/dev/stdin}"

your command should work.

The problem is that your appending \r at the end of your input lines (so that every line of your input ends with a \r\n sequence). By default read does not strip trailing r. If we want read to trim those characters we have to add this character to the IFS environmental variable for read like this: IFS="$IFS"$'\r')" read ....

Here's a great comment from Charles Duffy:

Personally, I'd suggest IFS=$' \t\n\r', not referring to the old $IFS value -- why make your code's behavior contextually dependent?

Another valuable comment; this time from chepner:

Granted, a valid line probably isn't going to contain a \r, but conceptually, you don't want to treat a carriage return as whitespace; you just want to strip the \r that is part of the \r\n line ending. Instead of modifying IFS, read the line normally, then strip it with line=${line%$'\r'} before calling curl.

answered Aug 27, 2017 at 11:58

Mateusz Piotrowski

9,39311 gold badges61 silver badges83 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

chepner Over a year ago

Granted, a valid line probably isn't going to contain a \r, but conceptually, you don't want to treat a carriage return as whitespace; you just want to strip the \r that is part of the \r\n line ending. Instead of modifying IFS, read the line normally, then strip it with line=${line%$'\r'} before calling curl.

Mateusz Piotrowski Over a year ago

@chepner True. I'll add that to the answer.

Chen A. · Accepted Answer · 2017-08-27 12:09:53Z

0

You try to address a variable and a text right after it, you should use ${VAR} syntax:

curl "https://x.com/v1/PhoneNumbers/${line}?Type=carrier" -u "x:x" done < "${1:-/dev/stdin}"

The reason is bash tries to expand the parameter $line?Type .. instead of $line.

For example

[root@ ~]# line=1234
[root@ ~]# echo $line567

[root@ ~]# echo ${line}567
1234567

edited Aug 27, 2017 at 12:09

answered Aug 27, 2017 at 11:55

Chen A.

11.5k4 gold badges55 silver badges75 bronze badges

4 Comments

Linda Roesch Over a year ago

Unfortunately I'm getting curl: (3) Illegal characters found in URL

Mateusz Piotrowski Over a year ago

This is not the case as ? cannot be a part of the variable name so it's perfectly OK to omit {} (at least in this case).

Chen A. Over a year ago

Can you add echo statement of the string before executing curl and show it here? echo https://x.com/v1/PhoneNumbers/$line?Type=carrier

Charles Duffy Over a year ago

echo isn't nearly as useful for showing nonprintable characters as printf '%q\n' "https://x.com/v1/PhoneNumbers/$line?Type=carrier" (which will properly quote and escape a trailing carriage return, f/e, which is what the accepted answer proposes)

Collectives™ on Stack Overflow

Using a Bash variable in URL results in "curl: (3) Illegal characters found in URL"

2 Answers 2

2 Comments

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related