Launch python commands in parallel using shell

Question

I have a command in python this way:

Python my_prog in1.fa ins.fa out1.fa
Python my_prog in2.fa ins.fa out2.fa
Python my_prog in3.fa ins.fa out3.fa

I used the parallel command of GNU parallel and I assemble the files in1.fa, in2.fa and in3.fa in the one file IN.fa. My problem is I do not know how to put another agument or more in the parallel command. here is my command:

cat IN.fa | parallel -j 20 --cat --pipe --block 3M --recstart '>' time python my_prog.py

How can I make several arguments in the command Parallel please?

Ole Tange · Accepted Answer · 2016-12-29 01:43:15Z

1

Let us assume that my_prog can read from stdin and send output to stdout and that it takes a single argument (ins.fa):

parallel --pipepart -a in.fa --block 3M Python my_prog ins.fa > out.fa

If my_prog cannot read from stdin, but from a named pipe (fifo) this will work:

parallel --fifo --pipepart -a in.fa Python my_prog {} ins.fa > out.fa

If my_prog cannot read from a fifo, but only an actual file, this will work:

parallel --cat --pipepart -a in.fa Python my_prog {} ins.fa > out.fa

If my_prog cannot output to stdout, but can output to a fifo you can often use:

parallel --cat --pipepart -a in.fa Python my_prog {} ins.fa {#}.out /dev/stdout  > out.fa

Or:

parallel --cat --pipepart -a in.fa Python my_prog {} ins.fa {#}.out '>(cat)' > out.fa

If my_prog cannot output to a fifo, you need to have it output to a uniquely named file, which you can then cat and remove. Here we use the sequence number to make a unique file.

parallel --cat --pipepart -a in.fa Python my_prog {} ins.fa {#}.out '; cat {#}.out; rm {#}.out'  > out.fa

You really should consider walking through the tutorial. It will answer this and so many other questions: man parallel_tutorial

edited Dec 29, 2016 at 1:43

answered Dec 28, 2016 at 15:14

Ole Tange

34.1k9 gold badges93 silver badges111 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

midox Over a year ago

no! it's not my question. I want to partition my file IN.fa into subfiles in block 3M since they compare with ins.fa and each time gives me a result in a separate file.

Collectives™ on Stack Overflow

Launch python commands in parallel using shell

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related