-1

So, i already know how to successfully read every N lines in parallel, and run a command on each of those lines:

while read -r i && read -r a && read -r b && read -r c && read -r d && read -r e && read -r f && read -r g && read -r h && read -r j && read -r k && read -r l && read -r m && read -r n && read -r o && read -r p && read -r q && read -r r && read -r s && read -r t && read -r u && read -r v && read -r w && read -r x && read -r z && read -r aa && read -r bb && read -r cc && read -r dd && read -r ee && read -r ff && read -r gg && read -r hh && read -r ii && read -r jj; do

                    dosomething "$i" &
                    dosomething "$a" &
                    dosomething "$b" &
                    dosomething "$c" &
                    dosomething "$d" &
                    dosomething "$e" &
                    dosomething "$f" &
                    dosomething "$g" &
                    dosomething "$h" &
                    dosomething "$j" &
                    dosomething "$k" &
                    dosomething "$l" &
                    dosomething "$m" &
                    dosomething "$n" &
                    dosomething "$o" &
                    dosomething "$p" &
                    dosomething "$q" &
                    dosomething "$r" &
                    dosomething "$s" &
                    dosomething "$t" &
                    dosomething "$u" &
                    dosomething "$v" &
                    dosomething "$w" &
                    dosomething "$x" &
                    dosomething "$z" &
                    dosomething "$aa" &
                    dosomething "$bb" &
                    dosomething "$cc" &
                    dosomething "$dd" &
                    dosomething "$ee" &
                    dosomething "$ff" &
                    dosomething "$gg" &
                    dosomething "$hh" &
                    dosomething "$ii" &
                    dosomething "$jj" &
                    wait
done < somefile

Where dosomething is just an example function/application that do something with the specific line (which is represented by the variable next to it).

Essentially this work fine. Just wanted to make this work/look better by using arrays instead, but not sure how to format this so it use arrays...

Where it would generate variable names (only alpha, no number as that might create problem with the usual numbered variable, such as $1 etc) for N amount of job.

To prevent confusion: each read in the script above does "one line" each, so each read = one line. Meaning each iteration there are reading one line X amount of read. The wait at the end of each iteration wait for all jobs to finish.

Example:

  • This answer does a similar thing that I'm doing, except it read from two file, two lines (one for each) at the same time. Mine does way more.

PS: I'm aware i could do something better using parallel, xargs or something else, but prefer to only use bash/POSIX if possible.

Any input appreciated.

Improve this question
9
  • 1
    from your script i have no idea what you're trying to do... Why i, a, b, c ... etc. How do you know how many times read you need? Why not one read per iteration and a wait after the loop? Commented Mar 22, 2021 at 14:33
  • because then it would only read "one line" per "iteration"...here it does as many lines per iteration as there are "read" so, i guess it's in parallel as i described in my post above :) @pLumo Commented Mar 22, 2021 at 14:40
  • This seems extremely complicated... whats wrong with a regular loop with read? You can still execute your commands in the background (what you call "in parallel") Commented Mar 22, 2021 at 14:48
  • because then it'll only do a single line per iteration?? this wouldn't be fast enough, i mean, i believe i didn't leave any explanation of the code above (do tell me if i did so i can edit it in) @Panki Commented Mar 22, 2021 at 14:49
  • And your approach somehow does not? I am confused, to say the least. What do you define as "iteration"? I thought the goal here is to run a command with every line in the file? So iterations = linecount Commented Mar 22, 2021 at 14:50

1 Answer 1

Reset to default
1

does not use arrays, but keep a counter to determine when to wait.

count=0
while read -r line; do
    dosomething "$line" &
    ((++count % 35 == 0)) && wait
done < file

Parenthetically, do you not like the variable y?

One big issue with your code: you have to successfully read all 35 lines to dosomething with them. while processing the file, suppose you read a and b but hit EOF while attempting to read c -- read returns non-zero, the while condition fails and the loop breaks without processing $a and $b.

With an array, not simpler:

mapfile -t lines < file
len=${#lines[@]}
i=0
while ((i < len)); do
    for _ in {1..35}; do
        dosomething "${lines[i]}" &
        ((i++))
        ((i == len)) && break
    done
    wait
done
Improve this answer
2
  • haha, i didn't noticed i forgot to use the variable y :) I actually don't have anything against it, the more the merrier (for the number of jobs that is). Commented Mar 22, 2021 at 16:10
  • Interesting. I guess i didn't thought as far as i thought when i came up with the array method hmm. Although, i do roughly recall getting to the end of a huge file, and the loop didn't entirely fail when there was less than 35 lines (it did fail for individual lines but not the for the rest of first couple of lines that catch the rest of the file), unless i checked wrong... Commented Mar 22, 2021 at 16:33

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.