IPython: Running commands asynchronously in the background

Question

Say I have a Python command or script that I want to run from IPython asynchronously, in the background, during an IPython session.

I would like to invoke this command from my IPython session and be notified when it is done, or if something fails. I don't want this command to block my IPython prompt.

Are there any IPython magics that support this? If not, what is the recommended way of running asynchronous jobs/scripts/commands (that run locally) on IPython?

For example, say I have a function:

def do_something():
   # This will take a long time
   # ....
   return "Done"

that I have in the current namespace. How I can I run it to the background and be notified when it is done?

Is it a standalone Python script (.py file) and not a function from currently imported modules? In that case you could just spawn a new OS process using Python subprocess module. — Mikko Ohtamaa
– Mikko Ohtamaa, Commented Mar 18, 2014 at 13:02

Aron Ahmadia · Accepted Answer · 2015-09-30 20:35:01Z

10

Yes, try (in a cell):

%%script bash --bg --out script_out

sleep 10
echo hi!

The script magic is documented along with the other IPython magics. The necessary argument here is -bg to run the below script in the background (asynchronously) instead of the foreground (synchronously).

GitHub Issue #844 is now resolved.

edited Sep 30, 2015 at 20:35

answered Sep 26, 2015 at 20:18

Aron Ahmadia

2,4152 gold badges20 silver badges23 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

David Ferenczy Rogožan Over a year ago

Could you add some explanation and for example link to a documentation, please?

Aron Ahmadia Over a year ago

@DawidFerenczy, this what you were looking for?

David Ferenczy Rogožan Over a year ago

Yes, it's much better now, thank you. I didn't want it for me, but rather for others who will have the same problem.

jminuscula · Accepted Answer · 2014-03-18 13:06:44Z

6

There used to be a magic function in iPython that would let you do just that: https://github.com/ipython/ipython/wiki/Cookbook:-Running-a-file-in-the-background

However, it seems that it was removed and is still pending to come back in newer versions: https://github.com/ipython/ipython/issues/844

It still provides a library to help you achieve it, though: http://ipython.org/ipython-doc/rel-0.10.2/html/api/generated/IPython.background_jobs.html

answered Mar 18, 2014 at 13:06

jminuscula

5522 silver badges10 bronze badges

Comments

Amelio Vazquez-Reina · Accepted Answer · 2014-03-18 13:32:32Z

4

The most general way would be to use the Multiprocessing Module. This should allow you to call functions in your current script in the background (completely new process).

Edit This might not be the cleanest way, but should get the job done.

import time
from multiprocessing import Process, Pipe
ALONGTIME = 3

def do_something(mpPipe):
    # This will take a long time
    print "Do_Something_Started"
    time.sleep(ALONGTIME)
    print "Do_Something_Complete"
    mpPipe.send("Done")

if __name__ == "__main__":
    parent_conn, child_conn = Pipe()
    p = Process(target=do_something, args=(child_conn,))
    p.start()
    p.join() # block until the process is complete - this should be pushed to the end of your script / managed differently to keep it async :)
    print parent_conn.recv() # will tell you when its done.

edited Mar 18, 2014 at 13:32

Amelio Vazquez-Reina

97.2k142 gold badges376 silver badges585 bronze badges

answered Mar 18, 2014 at 13:06

Adam Lewis

7,3077 gold badges49 silver badges63 bronze badges

2 Comments

Amelio Vazquez-Reina Over a year ago

Thanks! This is a great pointer. I thought multiprocessing would help, I need to look into it, and it's great to have a specific simple example. What is the purpose of p.join() above? Do you know if the %bg magic in IPython was based on this module?

Adam Lewis Over a year ago

The p.join() blocks until the process is complete - not all the useful to your exact need, but required for the example. You'll need to manage the connections some how to make sure you don't orphan your other processes. I have used iPython in the past, but have no idea about that magic.

Collectives™ on Stack Overflow

IPython: Running commands asynchronously in the background

3 Answers 3

3 Comments

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

3 Comments

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related