I've a requirement of processing a file which contains some 100 shell (bash) commands; each line has a separate shell command. I have to execute these commands parallely (like 10 commands in parallel or 20, let the CPU decide how to do that in parallel). I honestly don't know how to accomplish it so I took a code somewhere around here only; below is the same:
from subprocess import PIPE
import subprocess
import time
def submit_job_max_len(job_list, max_processes):
sleep_time = 0.1
processes = list()
for command in job_list:
print 'running process# {n}. Submitting {proc}.'.format(n=len(processes),
proc=str(command))
processes.append(subprocess.Popen(command, shell=False, stdout=None, stdin=PIPE))
while len(processes) >= max_processes:
time.sleep(sleep_time)
processes = [proc for proc in processes if proc.poll() is None]
while len(processes) > 0:
time.sleep(sleep_time)
processes = [proc for proc in processes if proc.poll() is None]
cmd = 'cat runCommands.sh'
job_list = ((cmd.format(n=i)).split() for i in range(5))
submit_job_max_len(job_list, max_processes=10)
I don't understand the last 3 lines as to what actually it's doing. My test runs show that the number in range(n) will execute ever line that many times. So if the number is 5, then every line is executed 5 times. I don't want that. Can someone throw some light on this please. And again, please excuse my ignorance here.
