Bash forked for loop with while loop control

Question

I have a for loop that will cycle through all the files in a directory. I want to run a process on each file (in this example echo file name and sleep 5). However I want to be able to run this 5 files at a time (in the background with &). The problem is I can't figure out how to iterate $f within the while loop so that I don't end up processing the same file five times instead of five different files at the same time.

#!/bin/bash

maxjobs=5

for f in `ls /home/user/`
do
 jobsrunning=0
  while [ $jobsrunning -lt $maxjobs ]
  do
    echo "Converting file"$f 
    sleep 5 & #wait for 5 seconds 
    jobsrunning=$((jobsrunning + 1))
    echo $jobsrunning
  done
wait
done

you never reset jobsrunning to 0. But a much better solution exists to "feed" files to a process. Search here for find print0 xargs. Unfortuantely the final term might be multiple, jobs, -n or a few others. — shellter
– shellter, Commented Jun 16, 2014 at 13:53
no need for ls ... subshell with for. "for f in /home/user/*" suffices - difference with ls is that names will be fully qualified, therefore not a drop-in replacement. May be preferable in your case because actions on those files needed to know the implied "/home/user" location as your current solution requires — Deleted User
– Deleted User, Commented Jun 16, 2014 at 14:01
"so that I don't end up processing the same file five times instead of five different files at the same time" make not a lot of sense. why would you process the same file five times? Why would, when printed to screen, the same file name appear 5 times? — Deleted User
– Deleted User, Commented Jun 16, 2014 at 14:03

kojiro · Accepted Answer · 2014-06-16 14:12:08Z

3

You really just need to reset jobsrunning, and I don't think you need the inner loop at all. It's just a condition.

#!/bin/bash

maxjobs=5
jobsrunning=0
for f in /home/user/*; do
    if (( jobsrunning >= maxjobs )); then
        wait
        jobsrunning=0
    fi
    echo "converting"
    sleep 5 & # wait for how many seconds?
    (( jobsrunning++ ))
    echo "$jobsrunning"
done
wait

That said, this sounds like a job for parallel.

edited Jun 16, 2014 at 14:12

answered Jun 16, 2014 at 13:56

kojiro

77.8k20 gold badges151 silver badges217 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

crysis405 Over a year ago

This doesn't quite work, the jobsrunning needs to be defined before the if statement. I'm getting loads of errors when trying to run it.

kojiro Over a year ago

@crysis405 Well, explicit may be better, but an undefined name evaluates to 0 in shell arithmetic context, so this should work fine. However, if you just used [ ] as in your original code, then yes, you need to define it. Why not switch to modern arithmetic expressions, though?

crysis405 Over a year ago

I'm in Ubuntu 12.04 and I can't seem to get this to work. Do I need to update something?

crysis405 Over a year ago

./test_par.sh: 2: ./test_par.sh: jobsrunning: not found converting ./test_par.sh: 8: ./test_par.sh: jobsrunning++: not found test_par.sh is the file I pasted this into (assuming that's correct)

kojiro Over a year ago

@crysis405 you can't invoke a bash script as sh. Those are two different languages, and that is why you are having that problem.

|

Collectives™ on Stack Overflow

Bash forked for loop with while loop control

1 Answer 1

7 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

7 Comments

Your Answer

Sign up or log in

Post as a guest

Related