execute multiple commands in Linux using python in the same time

Question

I need to execute multiple commands in Linux using python in the same time. I don't need to run it command by command.

I try to write this code but i can't understand how to execute multiple commands in the same time using python also i read about python multithreading but i don't know how to use it.

Code:

#  -*- coding: utf-8 -*-

import os

commands = ['ping www.google.com', 'ping www.yahoo.com', 'ping www.hotmail.com']
count = 0
for com in commands:
    print "Start execute commands.."
    os.system(com)
    count += 1
    print "[OK] command "+str(count)+" runing successfully."
else:
    print "Finish.."

Please how i can do that with python and execute multiple commands in the same time??

You want subprocess.Popen, system blocks

Padraic Cunningham
– Padraic Cunningham

2016-05-07 19:39:59 +00:00
Commented May 7, 2016 at 19:39 — Padraic Cunningham
– Padraic Cunningham, Commented May 7, 2016 at 19:39

HoldOffHunger · Accepted Answer · 2020-10-23 16:41:03Z

Looks like a typical producer-consumer problem

import threading
import os

commands = ['ping www.google.com', 'ping www.yahoo.com', 'ping www.hotmail.com']

def worker_func():
    while commands:   # Checks if the list is not-empty. Loop exits when list is becomes empty
        com = commands.pop(0)
        print "Start execute commands.."
        os.system(com)
        count += 1
        print "[OK] command "+str(count)+" runing successfully."
        
workers = [threading.Thread(target=worker_func, args=tuple(), name='thread_'+str(i))  for i in range(5) ]  # Create 5 workers (consumers)
[worker.start() for worker in workers]  # Start working
[worker.join() for worker in workers]   # Wait for all workers to finish

Here I have created the 5 worker threads. These threads will run function worker_func.
worker_func Will pick up one element from the list and preform the job. When list becomes empty the function returns (exits).

Note: Read about Global Interpreter Lock to understand where python multithreading should not be used.
In this case the GIL (Global Interpreter Lock) should not affect you because the worker_func call a subprocess and wait for it complete. While the thread is waiting GIL is released to other threads.

Shark · Accepted Answer · 2016-05-07 20:04:07Z

I am suggesting two solutions but there are many

Simple Solution:

Use & at the end of your commands to run them in background:

commands = ['ping www.google.com &', 'ping www.yahoo.com &', 'ping www.hotmail.com &']
for com in commands:
    os.system(com) # now commands will run in background

threading + Queue solution with a control over maximum threads to spawn:

from Queue import Queue, Empty
import threading, os

def worker_func():
    while not stopped.is_set():
        try:
            # use the get_nowait() method for retrieving a queued item to
            # prevent the thread from blocking when the queue is empty
            com = q.get_nowait()
        except Empty:
            continue
        try:
            os.system(com)
        except Exception as e:
            print "[-] Error running command %s" %(str(e))
        finally:
            q.task_done()

commands = ['ping www.google.com', 'ping www.yahoo.com', 'ping www.hotmail.com']

thread_count = 4 # maximum parallel threads 
stopped = threading.Event()
q = Queue()
print "-- Processing %s tasks in thread queue with %s thread limit" %(str(len(commands)), str(thread_count))

for item in commands:
    q.put(item)

for i in range(thread_count):
     t = threading.Thread(target=worker_func)
     # t.daemon = True #Enable to run threads as daemons
     t.start()
q.join()       # block until all tasks are done
stopped.set()

robyschek · Accepted Answer · 2016-05-07 20:18:05Z

2

My solution doesn't starts extra threads.
I use subprocess.Popen to run a command, store Popen objects in a list in the first loop, and wait till the subprocesses finish in the second

from subprocess import Popen, PIPE

commands = ['ping www.google.com', 'ping www.yahoo.com', 'dir']
count = 0
processes = []
for com in commands:
    print "Start execute commands.."
    processes.append(Popen(com, shell=True))
    count += 1
    print "[OK] command "+str(count)+" running successfully."
else:
    print "Finish.."

for i, process in enumerate(processes):
    process.wait()
    print "Command #{} finished".format(i)

answered May 7, 2016 at 20:18

robyschek

2,04515 silver badges19 bronze badges

Comments

Hafees Kazhunkil · Accepted Answer · 2016-05-07 19:36:03Z

import threading
import os


def ping_url(number):

    os.system(number)

thread_list = []
commands = ['ping www.google.com', 'ping www.yahoo.com', 'ping www.hotmail.com']
for url in commands:
    # Instantiates the thread

    t = threading.Thread(target=print_number, args=(url,))
    # Sticks the thread in a list so that it remains accessible
    thread_list.append(t)

# Starts threads
for thread in thread_list:
    thread.start()

# This blocks the calling thread until the thread whose join() method is called is terminated.
# From http://docs.python.org/2/library/threading.html#thread-objects
for thread in thread_list:
    thread.join()

# Demonstrates that the main process waited for threads to complete
print "Done"

Collectives™ on Stack Overflow

execute multiple commands in Linux using python in the same time

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related