Error in attempting to parallel task of a bash script

Question

I am trying to parallel the task of rpw_gen_features in the following bash script:

#!/bin/bash
maxjobs=8
jobcounter=0
MYDIR="/home/rasoul/workspace/world_db/journal/for-training"
DIR=$1
FILES=`find $MYDIR/${DIR}/${DIR}\_*.hpl -name *.hpl -type f -printf "%f\n" | sort -n -t _ -k 2` 
for f in $FILES; do
  fileToProcess=$MYDIR/${DIR}/$f
  # construct .pfl file name
  filebasename="${f%.*}"
  fileToCheck=$MYDIR/${DIR}/$filebasename.pfl
  # check if the .pfl file is already generated
  if [ ! -f $fileToCheck ];
  then        
    echo ../bin/rpw_gen_features -r $fileToProcess &
    jobcounter=jobcounter+1
  fi
  if [jobcounter -eq maxjobs]
    wait
    jobcounter=0
  fi
done

but it generates some error at runtime:

line 20: syntax error near unexpected token `fi'

I'm not an expert in bash programming, so please feel free to comment on the whole code.

Ole Tange · Accepted Answer · 2014-01-05 02:12:54Z

2

I am curious why you don't just use GNU Parallel:

MYDIR="/home/rasoul/workspace/world_db/journal/for-training"
DIR=$1
find $MYDIR/${DIR}/${DIR}\_*.hpl -name *.hpl -type f |
  parallel '[ ! -f {.}.pfl ] && echo ../bin/rpw_gen_features -r {}'

Or even:

MYDIR="/home/rasoul/workspace/world_db/journal/for-training"
parallel '[ ! -f {.}.pfl ] && echo ../bin/rpw_gen_features -r {}' ::: $MYDIR/$1/$1\_*.hpl

It seems to be way more readable, and it will automatically scale when you move from an 8-core to a 64-core machine.

Watch the intro video for a quick introduction: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1

Walk through the tutorial (man parallel_tutorial). You command line with love you for it.

edited Jan 5, 2014 at 2:12

answered Jan 5, 2014 at 1:54

Ole Tange

34.2k9 gold badges93 silver badges112 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Boris the Spider · Accepted Answer · 2014-01-04 20:34:35Z

1

You are missing a then, spaces and ${} around the variables:

  if [jobcounter -eq maxjobs]
    wait
    jobcounter=0
  fi

Should be

  if [ ${jobcounter} -eq ${maxjobs} ]; then
    wait
    jobcounter=0
  fi

Further, you need to double check your script as I can see many missing ${} for example:

jobcounter=jobcounter+1

Even if you use the variables correctly this still will not work:

jobcounter=${jobcounter}+1

Will yield:

1
1+1
1+1+1

And not what you expect. You need to use:

jobcounter=`expr $jobcounter + 1`

With never versions of BASH you should be able to do:

(( jobcounter++ ))

edited Jan 4, 2014 at 20:34

answered Jan 4, 2014 at 20:27

Boris the Spider

61.4k6 gold badges111 silver badges177 bronze badges

2 Comments

Boris the Spider Over a year ago

@Rasoul You were also missing spaces and variable declarations.

Rasoul Over a year ago

You are right! I just fixed it. What do you think about the whole code? Is this the right way to parallel the task?

Collectives™ on Stack Overflow

Error in attempting to parallel task of a bash script

2 Answers 2

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related