django: creating a background process using os.fork?

Question

I wish to run a long-running script in the background upon receiving a request. I read about subprocess but I require that the call is nonblocking so that the request can complete in time.

def controlCrawlers(request):

    if request.method == 'POST' and 'type' in request.POST and 'cc' in request.POST:

        if request.POST['type'] == '3':
            if request.POST['cc'] == '1':
                    try: #temp solution checking socket is occupied by trying to connect
                        s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
                        s.connect(('localhost',DISCOVERY_SOCKET))
                        s.close()

                        return HttpResponse(simplejson.dumps({'success':0,'message': 'Socket is occupied. Possible crawler is already running'}), \
                                        mimetype='application/json')
                    except:
                        pid = os.fork()

                        if pid == 0:
                            #f = open('/home/foo/django','a')
                            #f.write('abc')
                           # f.close()
                            path = os.path.join(os.path.dirname(__file__), 'blogcontentReader/blogpost_crawler.py')
                            os.system("python %s" %path)
                            os._exit(0)

                        return HttpResponse(simplejson.dumps({'success':1,'message': 'Running...'}), \
                                    mimetype='application/json')

I used os.fork as suggested from another post but apparently control does not flow into my if pid == 0 portion. Is this the correct method to do this?

Try if request.method == 'POST' and request.POST.get('type') == '3' and request.POST.get('cc') == '1': and save yourself two indentation levels. Also, since you're returning in try, just do except: pass and move that stuff out an indentation level too. It will make everything easier to read. — agf
– agf, Commented Aug 11, 2011 at 4:12
I'm not sure why subprocess.Popen won't work. It doesn't stop execution from continuing? — agf
– agf, Commented Aug 11, 2011 at 4:21
@agf , yea, from what i see from the link in my post, it blocks until execution of the child process is complete. — goh
– goh, Commented Aug 11, 2011 at 5:52
Are you on OSX? That post is specifically about problems on OSX. — agf
– agf, Commented Aug 11, 2011 at 6:24

madth3 · Accepted Answer · 2013-06-24 16:24:34Z

5

Yeah, don't do this, use celery instead. It makes running asynchronous tasks a lot easier, more reliable.

edited Jun 24, 2013 at 16:24

madth3

7,33712 gold badges53 silver badges74 bronze badges

answered Aug 11, 2011 at 3:55

Zach Kelling

54.1k15 gold badges112 silver badges108 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

James Khoury Over a year ago

@amateur You could always consider it a good learning task as you'll probably want to use it again in another project?

goh Over a year ago

hmm u guys are probably right. I should consider picking up celery.

Zach Kelling Over a year ago

Yes. If you don't want to setup a broker you can use your database as a backend: docs.celeryproject.org/en/latest/tutorials/…

Artem Mezhenin Over a year ago

@goh, python-rq(python-rq.org) is also an option. It's a bit easier to start with.

user916367 Over a year ago

The question asked how to Fork a process safely. It doesn't matter if this 'isn't generally done' with django. Sometimes people's requirements live outside of that is considered best practice.

Torsten Engelbrecht · Accepted Answer · 2011-08-11 04:06:25Z

1

If you don't want to use asynchronous task queues with something like celery you can always just run a python script via cron. There are several options to do this. An example:

create a model which save the values which are needed by your process
write a standalone python/django script which get the values from the model, executee the task and remove the database entries
set up a cronjob to run your script

answered Aug 11, 2011 at 4:06

Torsten Engelbrecht

13.5k4 gold badges50 silver badges49 bronze badges

Collectives™ on Stack Overflow

django: creating a background process using os.fork?

2 Answers 2

5 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related