2

Suppose there are 4 CUDA devices (0,1,2,3) on my computer and there are 10 tasks to run, each tasks is a script named run01.sh, run02.sh, ..., run10.sh.

The problem is, each task use only 1 GPU, I want to write a bash script to run those 10 tasks at the same time to make best use of the 4 CUDA devices. How can I make it?

Update:

To address @RenaudPacalet's question

Please edit your question and explain how you specify which device to use for a given job.

There are 4 workers (4 CUDA devices), what I want is to find out a solution in bash which ensure:

  1. 1 worker can only handle 1 task at a time.
  2. there should be no free workers if there are remaining tasks.
Improve this question
6
  • Your question is not really a programming question and it somehow asks for recommendations, which is out of scope here. Did you look at GNU parallel? Commented Dec 20, 2024 at 14:58
  • @RenaudPacalet I don't think parallel or xargs can deal with this as there are only 4 CUDA devices. for 10 tasks. And parallel/xargs cannot allocate limited resources to mutliple tasks. I would like to know is there any useful bash technique can be used to solve this issue. Commented Dec 20, 2024 at 16:53
  • there are only 4 CUDA devices. for 10 tasks That's what parallel or xargs are made for: launch a given number of jobs (4 in your case), no more, keep this workload by starting a new job each time a job completes, until all jobs complete (10 in your case). See the -P option. Commented Dec 20, 2024 at 17:07
  • GPU are massively parallel devices, and they can execute tasks of multiple process without any problem as long as you have enough memory for all the tasks (note that each task is already parallel but may not always saturate the GPU). In fact, besides memory, the overall execution can often be faster with more task running in parallel because GPU can overlap latency stall with other tasks. And yes, xargs can be used for that. You can also use a basic bash script and count concurrent jobs with jobs -r | wc -l in a loop running the tasks. Commented Dec 20, 2024 at 17:21
  • 1
    @link89 Please edit your question and explain how you specify which device to use for a given job. Commented Dec 21, 2024 at 6:26

2 Answers 2

Reset to default
3

To make a job use specific CUDA GPUs you can set the environment variable CUDA_VISIBLE_DEVICES.

Now, if you need to limit the number of jobs that can run at the same time on a single GPU then a possible solution is to use a fifo queue with the list of available GPUs; the idea is that each job will have to "take" a GPU from the fifo queue, and after said job completes, it writes the GPU it's been using to the fifo so that a new job can use it.

#!/bin/bash

mkfifo free_gpus
exec 3<>free_gpus

# write the GPUs you want to use (0,1,2,3) into the fifo.
# for 2 jobs per GPU, write each GPU twice.
printf '%s\n' 0 1 2 3 >&3 &

for exe in run*.sh
do
    read gpu # allocate a GPU (blocks until there's one available)
    { CUDA_VISIBLE_DEVICES="$gpu" "$exe"; echo "$gpu" >&3; } &
done <&3
wait
Improve this answer
Sign up to request clarification or add additional context in comments.

2 Comments

Cool solution. I don't get it until I read explanation of chatgpt.
chatGPT beaks it down nicely indeed
2

With GNU Parallel it looks like this:

parallel -j4 CUDA_VISIBLE_DEVICES='$(({%} - 1))' {} ::: run*.sh

GNU parallel is a tool for executing jobs in parallel, combined with some shell scripting to manage GPU resources. Here's a detailed breakdown:

parallel: parallel is a command-line driven tool that allows you to execute multiple jobs in parallel. It's particularly useful for running the same command with different arguments or for distributing tasks across multiple cores or nodes.

-j4: This flag tells parallel to use 4 jobs simultaneously. In other words, it will run up to 4 processes at the same time.

CUDA_VISIBLE_DEVICES: This environment variable controls which GPUs are visible to CUDA-enabled applications.

{%} is a special replacement string in parallel. It represents the job slot number, starting from 1.

$(( )) is arithmetic expansion in bash, which evaluates the expression inside.

{%} - 1 subtracts 1 from the job slot number, because CUDA device indices typically start at 0.

So, this sets each job to use a different GPU, based on its job slot number. For example:

  • Job 1 uses GPU 0
  • Job 2 uses GPU 1
  • Job 3 uses GPU 2
  • Job 4 uses GPU 3
  • Job 2 finishes first freeing GPU 1
  • Job 5 uses GPU 1
  • Job 3 finishes freeing GPU 2
  • Job 6 uses GPU 2
  • Job 1 finishes freeing GPU 0
  • Job 7 uses GPU 0

{} is a placeholder for the argument(s) that parallel will substitute with actual filenames or data.

::: run * .sh specifies that parallel should run the command for each file matching run*.sh in the current directory. This means if you have files like run1.sh, run2.sh, etc., each of these scripts will be executed.

Improve this answer

1 Comment

That looks awesome! Can you explain how it works in your answer?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.