Run block of commands parallelly & save output to a file sequentially

Question

I'm looking for a Linux Shell script where I can Run a block of commands in parallel & save output to a file sequentially.

For example:

for each line in file.txt
{
    Execute A
    Execute B
    Execute C > Save output to output.txt
}

How do we make sure the output.txt has sequential output & the block of commands can be run in parallel.

Please post bash code here, if you have a question about bash, not some pseudocode. — user1934428
– user1934428, Commented Nov 26, 2021 at 12:47
Why so cryptic? What are you actually trying to do? Executing 3 processes for each line of large files is rarely a good way to proceed... — Mark Setchell
– Mark Setchell, Commented Nov 26, 2021 at 23:50

Renaud Pacalet · Accepted Answer · 2021-11-29 14:09:49Z

4

The wait bash builtin waits for all background jobs. So redirecting their output to a temporary file, waiting until they end, and concatenating the temporary files should be about what you want. Example where we replace Execute A by ( echo "A"; sleep 5 )`:

if tA=$(mktemp); then ( echo "A"; sleep 5 ) > "$tA" & fi
if tB=$(mktemp); then ( echo "B"; sleep 5 ) > "$tB" & fi
if tC=$(mktemp); then ( echo "C"; sleep 5 ) > "$tC" & fi
wait
cat "$tA" "$tB" "$tC" > output.txt
rm -f "$tA" "$tB" "$tC"

As noted by Léa Gris, when creating temporary files with mktemp it is always better to ensure that they are properly deleted even if the process is terminated before it explicitly deletes them. So the following is cleaner:

unset tA tB tC
trap 'rm -f -- "$tA" "$tB" "$tC"' EXIT TERM
if tA=$(mktemp); then ( echo "A"; sleep 5 ) > "$tA" & fi
if tB=$(mktemp); then ( echo "B"; sleep 5 ) > "$tB" & fi
if tC=$(mktemp); then ( echo "C"; sleep 5 ) > "$tC" & fi
wait
cat "$tA" "$tB" "$tC" > output.txt

edited Nov 29, 2021 at 14:09

answered Nov 26, 2021 at 11:47

Renaud Pacalet

30.7k3 gold badges42 silver badges60 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Léa Gris Over a year ago

Before creating the temp files, register a cleanup trap: unset tA tB tC; trap 'rm -f -- "$tA" "$tB" "$tC"' EXIT TERM

Renaud Pacalet Over a year ago

@LéaGris Good point. I added this.

Collectives™ on Stack Overflow

Run block of commands parallelly & save output to a file sequentially

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related