bash: how to update an associative array in an implicit subshell?

Question

The problem: I cannot update an array in a while loop. An illustration (not the actual problem):

declare -A wordcounts
wordcounts["sentinel"]=1000
ls *.txt | while read f; do
  # assume that that loop runs multiple times
  wordcounts[$f]=$(wc -w  $f)
  echo ${wordcounts[$f]}  # this prints actual data
done
echo ${!wordcounts[@]}  # only prints 'sentinel'

This does not work, because the loop after the pipe runs in a subshell. All the changes that the loop does to variable wordcounts are only visible inside the loop.

Saying export wordcounts does not help.

Alas, I seem to need the pipe and the while read part, so ways to rewrite the code above using for is not what I'm looking for.

Is there a legitimate way to update an associative array form within a loop, or a subshell in general?

Cookyt · Accepted Answer · 2013-10-03 16:26:52Z

7

Since you have a complex command pipe you're reading from, you can use the following:

while read f; do
    # Do stuff
done < <(my | complex | command | pipe)

The syntax <(command) runs a command in a subshell and opens its stdout as a temporary file. You can use it any place where you would normally use a file in a command.

Further, you can also use the syntax >(command) to open stdin as a file instead.

answered Oct 3, 2013 at 16:26

Cookyt

1,6271 gold badge12 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

chepner · Accepted Answer · 2013-10-03 17:26:09Z

3

If you are using bash 4.2, you can set the lastpipe shell option to allow the while loop, as the last element in the pipeline, to run in the current shell instead of a subshell.

A simple demonstration:

$ echo foo | read word
$ echo $word

$ set +m  # Only needed in an interactive shell to disable job control
$ shopt -s lastpipe
$ echo foo | read word
$ echo $word
foo

answered Oct 3, 2013 at 17:26

chepner

538k77 gold badges594 silver badges746 bronze badges

Comments

devnull · Accepted Answer · 2013-10-03 16:08:17Z

2

Is there a legitimate way to update an associative array form within a loop, or a subshell in general?

You could avoid a subshell by saying:

while read f; do
  ...
done < *.txt

That said, you sample code has problems otherwise. The loop would read the file line by line, so saying

wordcounts[$f]=$(wc -w  $f)

wouldn't really make much sense. You probably wanted to say:

wordcounts[$f]=$(wc -w <<< $f)

EDIT:

Alas, I seem to need the pipe ...

Quoting from the manual:

Each command in a pipeline is executed in its own subshell (see Command Execution Environment).

edited Oct 3, 2013 at 16:08

answered Oct 3, 2013 at 16:01

devnull

124k33 gold badges247 silver badges234 bronze badges

4 Comments

9000 Over a year ago

Unfortunately, I need the pipe; actually I have a more complex thing than ls on its left side. Updated the question to clarify. And no, while read f does read filenames from the pipe, not file lines from f.

Cookyt Over a year ago

You can still use the same technique with a complex command. At the end, instead of done < *.txt, use done < <(my | complex | command | pipe)

9000 Over a year ago

@Cookyt: please make your comment into an answer, since it's what actually solves my problem and I'd like to accept it.

devnull Over a year ago

@9000 You could use a named pipe. mkfifo myp; command1 | command2 > myp &; while read ... done < myp

josch · Accepted Answer · 2020-11-27 11:22:03Z

0

I think the best solution is the one by Cookyt:

while read f; do
    # Do stuff
done < <(my | complex | command | pipe)

For me, that didn't work because in my environment I don't have /proc mounted because the <(cmd) construct needs /dev/fd/XXX and /dev/fd is a symlink to /proc/self/fd. In those cases, the solution by by chepner works:

shopt -s lastpipe
my | complex | command | pipe | while read f; do
    # Do stuff
done

If you also don't have bash, there is a third solution that works with POSIX shells (and thus also with bash):

set -- $(my | complex | command | pipe)
while [ -n "$1" ]; do
    f="$1"
    shift
    # Do stuff
done

answered Nov 27, 2020 at 11:22

josch

7,3094 gold badges46 silver badges56 bronze badges

Collectives™ on Stack Overflow

bash: how to update an associative array in an implicit subshell?

4 Answers 4

Comments

Comments

4 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

Comments

4 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related