Please correct me if I am wrong here. A goal: to copy a file by spawning into another process (so actual copying is not "locking" the process that calls it).
cmd = ['cp', '/Users/username/Pictures/2Gb_ImageFile.tif', '/Volume/HugeNetworkDrive/VerySlow/Network/Connection/Destination.tif']
def copyWithSubprocess(cmd):
proc = subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
copyWithSubprocess(cmd)
Popen(cmd, stdout=PIPE, stderr=PIPE) won't "lock" your parent process.
cmd may stall itself if it generates enough output due to the full pipe buffers. If you want to discard subprocess' output then use DEVNULL instead of PIPE:
import os
from subprocess import Popen, STDOUT
DEVNULL = open(os.devnull, 'wb') #NOTE: it is already defined in Python 3.3+
p = Popen(cmd, stdout=DEVNULL, stderr=STDOUT)
# ...
if you want to process the output without blocking the main thread then you could use several approaches: fcntl, select, named pipes with iocp, threads. The latter is a more portable way:
p = Popen(cmd, stdout=PIPE, stderr=PIPE, bufsize=-1)
bind(p.stdout, stdout_callback)
bind(p.stderr, stderr_callback)
# ...
where bind() function:
from contextlib import closing
from functools import partial
from threading import Thread
def bind(pipe, callback, chunksize=8192):
def consume():
with closing(pipe):
for chunk in iter(partial(pipe.read, chunksize), b''):
callback(chunk)
t = Thread(target=consume)
t.daemon = True
t.start()
return t
You don't need an external process to copy a file in Python without blocking the main thread:
import shutil
from threading import Thread
Thread(target=shutil.copy, args=['source-file', 'destination']).start()
Python can release GIL during I/O so the copying happens both concurrently and in parallel with the main thread.
You could compare it with a script that uses multiple processes:
import shutil
from multiprocessing import Process
Process(target=shutil.copy, args=['source-file', 'destination']).start()
If you want to cancel the copying when your program dies then set thread_or_process.daemon attribute to True.
callback(chunk) in the code? For example: def stdout_callback(chunk): print("Got %d bytes on stdout" % len(chunk))DEVNULL.The easiest way to handle complicated asynchronous processes in Python is to use the multiprocessing library, which was designed specifically to support such tasks and has an interface that closely parallels that of the threading module (indeed I have written code that can switch between multi-threading and multi-processing operations mostly by importing one library or the other, but this required fairly rigorous limits on which parts of the modules were utilized).
[Edit: removed spurious advice about threading and made my opening assertion less bombastic]
Popen is "asynchronous". It is the way to call external processes in Python.
comunnicateor some other method that must wait ...