How to run tasks in parallel in bash without GNU parallel

Question

I would like to process 2000 files on a 64 core machine. I have a python script foo.py which I run like this:

cat file0000.txt|./foo.py > out0000.txt

Ideally I would to split the 2000 files file0000.txt to file01999.txt into forty sets each of size 50 and run foo.py on each set in parallel. For sets 1 to 4 out of 40 that would be the equivalent of the following:

cat file00[0-4][0-9] |./foo.py > outfile1.txt &
cat file00[5-9][0-9] |./foo.py > outfile2.txt &
cat file01[0-4][0-9] |./foo.py > outfile3.txt &
cat file01[5-9][0-9] |./foo.py > outfile4.txt &

Sadly the system I am running this on doesn't have parallel so I have to do this without that very useful tool.

Bash script processing commands in parallel looks similar but the most popular answer is not directly relevant and the second most popular answer uses parallel which I don't have access to.

@Alper That could be the answer but I have never used it. How would you use it for my problem? — Simd
– Simd, Commented Jul 29, 2016 at 7:55
Something like ls -1 | xargs -I{} -P 5 sh -c "cat {} | ./foo.py > out{}.txt", Note: ls -1 should be listing your input files and change -P 5 as you like. — Alper
– Alper, Commented Jul 29, 2016 at 8:37
Can you elaborate on why you don't install GNU Parallel? As per oletange.blogspot.dk/2013/04/why-not-install-gnu-parallel.html — Ole Tange
– Ole Tange, Commented Jul 29, 2016 at 16:42
@OleTange It's the installing part I am not allowed to do. I see from your very useful link that one option is to download it and run it as a normal user :) — Simd
– Simd, Commented Jul 29, 2016 at 17:52

Ole Tange · Accepted Answer · 2016-07-30 07:39:33Z

1

As per the comments: Do a personal installation of GNU Parallel which you are allowed to do if you are allowed to run your own scripts:

./configure --prefix=$HOME && make && make install

And then:

ls | ~/bin/parallel 'cat {} | ./foo.py > {= s/file/out/ =}'

answered Jul 30, 2016 at 7:39

Ole Tange

34.2k9 gold badges93 silver badges112 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to run tasks in parallel in bash without GNU parallel

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related