Can I put strings divided by newlines into an array using the read builtin?

Question

I have a variable:

var='/path/to/filename.ext
     /path/to/filename2.ext
     /path/to/filename3.ext'

I want to put all strings separated by a newline in an array:

declare -a arr

Based on numerous posts here on StackOverflow, I found a couple of ways:

# method 1: while loop
while read line; do
    arr+=($line)
done <<< "$var"

# method 2: readarray
readarray -t arr <<< "$var"

# method 3:
IFS=$'\n'
arr=("$var")

However, before I learned all these methods, I was using another one, namely:

# method 4 (not working in the current situation)
IFS=$'\n'
read -a arr <<< "$var"

This is not working, because it will only store the first string of var in arr[0]. I don't understand why it doesn't work in situations where the delimiter is a newline, while it does work with other delimiters, e.g.:

IFS='|'
strings='/path/to/filename.ext|/path/to/filename2.ext|'
read -a arr <<< "$strings"

Is there something that I'm missing?

Edit

Removed my own answer that argued you cannot use read for this purpose. Turns out you can.

Maybe it's the $ in front of the '\n'?

Danny Daglas
– Danny Daglas

2015-02-09 19:12:52 +00:00
Commented Feb 9, 2015 at 19:12 — Danny Daglas
– Danny Daglas, Commented Feb 9, 2015 at 19:12

gniourf_gniourf · Accepted Answer · 2015-02-09 19:30:25Z

9

It turns out that your answer is wrong. Yes, you can! you need to use the -d switch to read:

-d delim

The first character of delim is used to terminate the input line, rather than newline.

If you use it with an empty argument, bash uses the null byte as a delimiter:

$ var=$'/path/to/filename.ext\n/path/to/filename2.ext\n/path/to/filename3.ext'
$ IFS=$'\n' read -r -d '' -a arr < <(printf '%s\0' "$var")
$ declare -p arr
declare -a arr='([0]="/path/to/filename.ext" [1]="/path/to/filename2.ext" [2]="/path/to/filename3.ext")'

Success. Here we're using a process substitution with printf that just dumps the content of the variable with a trailing null byte, so that read is happy and returns a success return code. You could use:

IFS=$'\n' read -r -d '' -a arr <<< "$var"

In this case, the content of arr is the same; the only difference is that the return code of read is 1 (failure).

As a side note: there's a difference between

$ IFS=$'\n'
$ read ...

and

$ IFS=$'\n' read ...

The former sets IFS globally (i.e., IFS will retain this value for the remaining part of the script—until you modify it again, of course): you very likely don't want to do that!

The latter only sets IFS for the command read. You certainly want to use it that way!

Another side note: about your method 1. You're missing quotes, you're not unsetting IFS, and you're not using the -r flag to read. This is bad:

while read line; do
    arr+=($line)
done <<< "$var"

This is good:

while IFS= read -r line; do
    arr+=( "$line" )
done <<< "$var"

Why?

without unsetting IFS, you'll get leading and trailing spaces removed.
Without -r, some backslashes will be understood as escaping backslashes (\', trailing \, \, and maybe others).
Without properly quoting ( "$line" ), you'll get word splitting and filename expansion turned on: you don't want that in case your input contains spaces or glob characters (like *, [, ?, etc.).

edited Feb 9, 2015 at 19:30

answered Feb 9, 2015 at 19:23

gniourf_gniourf

47.4k10 gold badges105 silver badges113 bronze badges

Sign up to request clarification or add additional context in comments.

8 Comments

Stefan van den Akker Over a year ago

Awesome, great explanation, I've learned ten new things from your answer!

Stefan van den Akker Over a year ago

Question: why is the return value of read 0 (success) in the first case (process substitution), but 1 (error) in the second case (using unadorned "$var")? Does it have to do with that read doesn't encounter EOF in the first case (strings ends with null character), but does encounter it in the second case?

gniourf_gniourf Over a year ago

@Neftas: in Bash, a variable cannot contain a null byte. You can't, however hard you try, put a null byte in there. You may try: var=$'test\0test, but declare -p var shows that var contains test only; you can try printf -v var 'test\0test', but the same happens. You can try var=$(printf 'test\0test'), and here declare -p var shows that var contains testtest. So you really can't put a null byte in a Bash variable!

seh Over a year ago

And perhaps to save someone else a few hours of madness, note that read's "-d" flag requires an intervening space before its argument. That is, read -d '' works, but read -d'' does not. That's with version 3.2.57(1)-release on macOS.

gniourf_gniourf Over a year ago

@seh indeed! with -d'' (no space), Bash (any version, and in fact any shell) will parse it as just -d (that's because of the quote removal step of the parser).

|

Collectives™ on Stack Overflow

Can I put strings divided by newlines into an array using the read builtin?

Edit

1 Answer 1

8 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

Edit

1 Answer 1

8 Comments

Your Answer

Sign up or log in

Post as a guest

Related