Using Celery on processes and gevent in tasks at the same time

You can run celery with multiple threads containing multiple greenlets like this:

$ celery multi start 4 -P gevent -l info -c:1-4 1000

As far as I was able to learn, this is not possible. If someone finds out a better answer, I will accept it instead of this mine.

The only option is to use gevent also as a backend for Celery workers. What one have to do in order to accomplish such a thing is to add following in to the config file:

CELERYD_POOL = 'gevent'

More details about this options can be found here. More information about the gevent pool is on this page. Mind the fact that the gevent pool is still marked as experimental. I found no benchmarks available to compare processes and async gevent pool on different tasks (IO-oriented tasks, CPU-oriented tasks), but finally I realized even my CPU-bound tasks will be in fact more IO than CPU, because I use database to save results and the database connection will be a bottleneck, not the computing part. I will have no scientific tasks which would really hit the CPU.

From my weird experience, Celery Beat can't work properly with workers with gevent pool (scheduled tasks are blocked and wait forever), unless you activate gevent monkey patching for Beat process.

However, celery beat doesn't support --pool=gevent or -P gevent option. The proper way to inject gevent monkey patching is to use a curstomized celery binary, such as:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

from gevent import monkey
monkey.patch_all()

import re
import sys

from celery.__main__ import main

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
    sys.exit(main())

Save it as celery-gevent, and run Beat service as follows:

celery-gevent beat --app=proj.celery:app --loader=djcelery.loaders.DjangoLoader -f /var/log/celery/beat.log -l INFO --workdir=/my/proj --pidfile=/var/run/celery/beat.pid

In proj.celery you should also patch Django connection to avoid DatabaseError:

from __future__ import absolute_import

import os
# Set the Django settings module for the 'celery' program
os.environ.setdefault('DJANGO_SETTINGS_MODULE', 'proj.settings')

import django
# Load Django model definitions, etc
django.setup()

from django.db import connection
# Allow thread sharing to ensure that Django database connection
# works properly with gevent.
connection.allow_thread_sharing = True

from django.conf import settings
from celery import Celery

app = Celery('proj')

# Using a string here means the worker will not have to
# pickle the object when using Windows.
app.config_from_object('django.conf:settings')
app.autodiscover_tasks(lambda: settings.INSTALLED_APPS)

(The above example works for Python 2.7.10, Celery 3.1.18, Django 1.8.2 and gevent 1.0.2)