Executing Python Script as Celery Task

Question

I am trying to run a python script as a celery task with Django. The issue I am having is that the task thinks it is complete as soon as the script begins running. I initially used subprocess.popen() in the tasks.py file, but realized this would mean the task would be complete as soon as the popen() command was issued. I modified my tasks.py code to call a function in my python script, which runs the script; however, this still executes as though the task is immediately complete. I am confused because in flower it says the task is complete, but in the celery log it is outputting the log data defined in the script I am running. I found the following related post. I believe I am following its suggestion to execute a python function from tasks.py.

tasks.py:

def exe(workDir, cancelRun):
    sys.path.append(workDir)
    import run

    if cancelRun=='True':
        task_id=exe.request.id
        revoke(task_id,terminate=True)
    else:
        run.runModel(workDir)
        task_id=exe.request.id
        return task_id

runModel function code:

def runModel(scendir):
    fullpath=scendir+'/run.py'
    os.chdir(scendir)
    p=Process(target=myMain,args=(scendir,))
    p.start()
    p.join()

What happens if you run the command (the one executed by your task) on your terminal? If it returns immediatelly it's possible that it's daemonizing itself, let's say via fork(). Another way of testing the cause is using the Django shell: python manage.py shell, then execute the necessary imports and call your functions manually. — jweyrich
– jweyrich, Commented Jul 10, 2014 at 23:15
Executing from the command line does what I expect. It begins running a script, which typically takes several hours to complete. I also considered my logic for html may be wrong and I'm returning the page I want after the task has finished by mistake. — Jason Hawkins
– Jason Hawkins, Commented Jul 10, 2014 at 23:28
My issue was actually that I didn't understand the outputs from celery. I was looking at the 'worker' tab of flower, rather than 'tasks'. I also interpreted the logging in celery beginning at the start of my log file as a sign it was starting the process again. It turns out it was creating a second task (as expected and desired). It was good to verify in the command prompt everything was working. — Jason Hawkins
– Jason Hawkins, Commented Jul 10, 2014 at 23:36

jweyrich · Accepted Answer · 2014-07-11 00:43:13Z

1

What happens if you run the command (the one executed by your task) on your terminal? If it returns immediatelly it's possible that it's daemonizing itself, let's say via fork().

Another way of testing the cause is using the Django shell: python manage.py shell, and then execute the necessary imports to call your functions manually.

answered Jul 11, 2014 at 0:43

jweyrich

32.5k5 gold badges70 silver badges101 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Executing Python Script as Celery Task

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related