Running an executable over multiple cores using a bash script

Question

Sorry if this is a repeat question and I know there are a lot of similar questions out there, but I am really struggling to find a simple answer that works.

I want to run an executable many times eg.

seq 100 | xargs -Iz ./program

However I would like to run this over multiple cores on my machine (currently on a macbook pro so 4 cores) to speed things up.

I have tried using gnu parallel by looking at other answers on here as that seems to be what I want and have it installed but I can't work out how parallel works and what arguments I need in what order. Non of the readme is helping as it is trying to do much more complicated things than I want to.

Could anyone help me? Thanks

It helps if you edit your question and show the first 3 commands that you would want GNU Parallel to run... then we can work out the incantation! — Mark Setchell
– Mark Setchell, Commented Apr 27, 2020 at 13:10
I just want to run the same executable in parallel rather than sequence. It is exactly the same every time it just takes around 6 hours to do 100 runs using the above code. — Charlie
– Charlie, Commented Apr 27, 2020 at 13:42
So you want run ./program then ./program then ./program all with no arguments and in parallel? — Mark Setchell
– Mark Setchell, Commented Apr 27, 2020 at 13:44
Yeah. The program already has all its input arguments. I want to evaluate a variable in the program and how it generates each time so do to that I need to run the whole thing lots and lots of times. — Charlie
– Charlie, Commented Apr 27, 2020 at 13:58

Mark Setchell · Accepted Answer · 2020-04-27 14:00:32Z

2

So, in order to run ./program 100 times, with GNU Parallel all you need is:

parallel -N0 ./program ::: {1..100}

If your CPU has 8 cores, it will keep 8 running in parallel till all jobs are done. If you want to run, say 12, in parallel:

parallel -j 12 -N0 ./program ::: {1..100}

answered Apr 27, 2020 at 14:00

Mark Setchell

210k32 gold badges310 silver badges505 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Charlie Over a year ago

That's perfect. I knew it would be simple I just could work out which arguments i needed and in which order and all the examples were trying to do more complicated things! If I wanted to limit it so that it only ran on 3 out of 4 cores so I could carry on working what would I change?

Mark Setchell Over a year ago

You could use parallel -j 3 -N0 ... or parallel -j 75% -N0 ... for 3 cores out of 4.

Collectives™ on Stack Overflow

Running an executable over multiple cores using a bash script

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related