How to run on BASH a python script that takes multiple arguments using GNU parallel?

Question

I have a python script which I normally execute from BASH shell like this:

pychimera $(which dockprep.py) -rec receptor1.pdb -lig ligand1.mol -cmethod gas -neut

As you see some of the arguments need an input (e.g. -rec) while others don't (e.g. -neut). I must execute this script 154 times with different inputs. How is it possible to run 8 threads in parallel using GNU parallel script?

pychimera $(which dockprep.py) -rec receptor1.pdb -lig ligand1.mol -cmethod gas -neut
pychimera $(which dockprep.py) -rec receptor2.pdb -lig ligand2.mol -cmethod gas -neut
pychimera $(which dockprep.py) -rec receptor3.pdb -lig ligand3.mol -cmethod gas -neut
...

You need to create a file with all commands in it. You can build that file automatically (and we can help with enough information to build that file). Then module load parallel && parallel < commands.txt — user6435535
– user6435535, Commented Apr 6, 2019 at 12:29

Mark Setchell · Accepted Answer · 2019-04-06 18:56:21Z

3

I think you want this:

parallel 'pychimera $(which dockprep.py) -rec receptor{}.pdb -lig ligand{}.mol -cmethod gas -neut' ::: {1..154}

If you have other than 8 CPU cores, and specifically want 8 processes at a time, use:

parallel -j8 ...

If you want to see the commands that would be run without actually running anything, use:

parallel --dry-run ...

edited Apr 6, 2019 at 18:56

answered Apr 6, 2019 at 13:36

Mark Setchell

210k32 gold badges310 silver badges504 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

tevang Over a year ago

The --dry-run is very informative. But the input files do not necessarily follow the simplified format with indices that I used. They may vary completely.

Mark Setchell Over a year ago

The command I suggested works exactly the same as the answer you have accepted! I can only help you as well as your question describes your problem - if your problem is harder please alter your question and I, or maybe Ole (the author of GNU Parallel) will assist further.

Ole Tange Over a year ago

@MarkSetchell I cannot help further: Your answer is literally character for character what I would have written.

user6435535 · Accepted Answer · 2019-04-06 12:39:42Z

1

Example commands.txt generator script:

#!/usr/bin/env bash

if [ "$#" -ne 1 ]; then
    echo "missing parameter: n"
    exit 1
fi

rm commands.txt 2> /dev/null 
dockp=$(which dockprep.py)

for((i=1;i<=$1;i++)); do
  echo "pychimera $dockp -rec receptor$i.pdb -lig ligand$i.mol -cmethod gas -neut" >> commands.txt
done

If you save above bash script as cmdgen.sh you can run it as:

bash cmdgen.sh 100

if you need n to be 100.

To run commands in parallel:

$ module load parallel
$ parallel < commands.txt

answered Apr 6, 2019 at 12:39

user6435535

1 Comment

tevang Over a year ago

Yes, it works great! I can control the number of threads like this "parallel -j8 < commands.txt".

Collectives™ on Stack Overflow

How to run on BASH a python script that takes multiple arguments using GNU parallel?

2 Answers 2

3 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related