1

I wish to:

  1. launch an async task (with something like django-celery or Gevent),
  2. wait a finite amount of time for task to finish (1 second),
  3. and then continue without killing the task.

I noticed this related thread https://stackoverflow.com/a/13001153/226800; however, in the accepted answer, when 'g' is destroyed does the task also get destroyed? I want to allow the task to run in the background even after the creator of the task has already returned, after waiting a while.

9
  • I can't see what is missing from my answer; I can guarantee that it's factually correct—but you don't believe me? Commented Oct 5, 2013 at 23:24
  • I think it needs cleaned up to be a 'correct' answer. The code you pasted in gist is actually more correct than your answer. I could not get anything (even after adding a while sleep) after task timed out but is continuing from the task in your example. Commented Oct 7, 2013 at 15:15
  • also, I want to clarify the question in that I really do want to completely return from the caller, If something like Gevent is used in context of a request/response cycle in Django, what happens to these events? Are they registered globally somehow? I really think stackoverflow.com/questions/9034091/… might be my answer. Do you agree? Commented Oct 7, 2013 at 15:18
  • Are you talking about events or greenlets? if greenlets: then yes, greenlets are "registered" with the Gevent Hub and continue living indefinitely, until killed or until they terminate normally or because of an exception, or the entire process is killed. As to if Django might kill off the entire thread or process where the Gevent Hub resides in—yes, that might happen, but you need to Google "django gevent" and make sure you set everything up properly. Commented Oct 7, 2013 at 15:27
  • Also, my Gist contains the exact same code as my answer, except in the Gist I'm using another greenlet to spawn the sub-greenlet. In my answer, the main greenlet is used implicitly, but it's still a greenlet, just like the main thread is a thread. And I didn't get the sentence that starts with " I could not get anything (even after adding a while sleep) after ..." Commented Oct 7, 2013 at 15:28

1 Answer 1

1

EDIT: yes, if you call g.kill(), the task "contained" by g is also killed ASAP (i.e. at the first cooperative context switch) ...and, well, that's kind of the point, if you think of it.

In Gevent, this is how you'd do it:

import gevent

def task():
    for _ in range(10):
        print "processing"
        gevent.sleep(1.0)
    return 'result'

task_g = gevent.spawn(task)
res = task_g.join(timeout=3.0)
if res is None:
    print "task timed out but is continuing"
else:
    print "got result: ", res

If you instead prefer an exception based flow, you can use with_timeout; this also has the advantage that you can return None from your task and not have it confused with a timeout:

import gevent
from gevent.timeout import Timeout, with_timeout

def task():
    ...

task_g = gevent.spawn(task)

try:
    result = with_timeout(3.0, task_g.join)
except Timeout:
    print "timeout"
    result = None
else:
    print "got result:", result

Later you can still kill the task if it "totally" times out by task_g.kill().

Sign up to request clarification or add additional context in comments.

6 Comments

I thought when my program finishes the lifetime of my greenlet (wrapped by gevents) will also be terminated regardless of results. Not sure how this works in context of a running django process. do you know?
Using subprocess? I think it should get killed if the parent process gets killed.
@brianray: Can I somehow improve/amend my answer for it to be acceptable?
@brianray: Cannot open the link. But I'm quite sure I'm right: greenlets continue to be referenced by the Gevent main loop (known as Hub), so even if their creator is garbage collected, they keep running; see gist.github.com/eallik/6798544 for proof.
here it is again pastebin.com/Pu5Vk5i9 But in your example the return from spawn does not fall off the stack
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.