Skip to main content
Mod Moved Comments To Chat
Add warning notice
Source Link
AdminBee
  • 23.6k
  • 25
  • 56
  • 77

GNU parallel has several options to limit resource usage when starting jobs in parallel.

The basic usage for two nested loops would be

parallel python sim -a {1} -p {2} ::: 1 2 3 4 5 ::: 17.76 20.01 21.510 23.76

If you want to launch at most 5 jobs at the same time, e.g., you could say

parallel -j5 python <etc.>

Alternatively, you can use the --memfree option to start new jobs only when enough memory is free, e.g. at least 256 MByte

parallel --memfree 256M python <etc.>

Note that the last option will kill the most recently started job if the memory falls below 50% of the "reserve" value stated (but it will be re-qeued for catch-up automatically).

GNU parallel has several options to limit resource usage when starting jobs in parallel.

The basic usage for two nested loops would be

parallel python sim -a {1} -p {2} ::: 1 2 3 4 5 ::: 17.76 20.01 21.510 23.76

If you want to launch at most 5 jobs at the same time, e.g., you could say

parallel -j5 python <etc.>

Alternatively, you can use the --memfree option to start new jobs only when enough memory is free, e.g. at least 256 MByte

parallel --memfree 256M python <etc.>

GNU parallel has several options to limit resource usage when starting jobs in parallel.

The basic usage for two nested loops would be

parallel python sim -a {1} -p {2} ::: 1 2 3 4 5 ::: 17.76 20.01 21.510 23.76

If you want to launch at most 5 jobs at the same time, e.g., you could say

parallel -j5 python <etc.>

Alternatively, you can use the --memfree option to start new jobs only when enough memory is free, e.g. at least 256 MByte

parallel --memfree 256M python <etc.>

Note that the last option will kill the most recently started job if the memory falls below 50% of the "reserve" value stated (but it will be re-qeued for catch-up automatically).

Source Link
AdminBee
  • 23.6k
  • 25
  • 56
  • 77

GNU parallel has several options to limit resource usage when starting jobs in parallel.

The basic usage for two nested loops would be

parallel python sim -a {1} -p {2} ::: 1 2 3 4 5 ::: 17.76 20.01 21.510 23.76

If you want to launch at most 5 jobs at the same time, e.g., you could say

parallel -j5 python <etc.>

Alternatively, you can use the --memfree option to start new jobs only when enough memory is free, e.g. at least 256 MByte

parallel --memfree 256M python <etc.>