Configure Celery with JSON Serializer (Python + node.js)

Question

I'm running a Celery worker in Python with the celery module v3.1.25, and a Celery client in node.js with the node-celery npm package v0.2.7 (not the latest).

The Python Celery worker works fine when sending a job using a Python Celery client.

Problem: When using a node-celery client to send a task to the Celery backend, we get an error in the JS console:

(STDERR) Celery should be configured with json serializer

Python Celery worker is configured with:

app = Celery('tasks', 
    broker='amqp://test:[email protected]:5672//',
    backend='amqp://',
    task_serializer='json',
    include=['proj.tasks'])

node-celery client is configured with:

var celery = require('node-celery')
var client = celery.createClient({
    CELERY_BROKER_URL: 'amqp://test:[email protected]:5672//',
    CELERY_RESULT_BACKEND: 'amqp',
    CELERY_TASK_SERIALIZER: "json"
});

client.on('connect', function() {
    console.log('connected');

    client.call('proj.tasks.getPriceEstimates', [start_latitude, start_longitude],
        function(result) {
            console.log('result: ', result);
            client.end();
        })
});

Is this a problem with the configuration on the Python Celery worker? Did we miss out on a configuration parameter which can change the return serialization format to json?

Update

Updated with result_serializers and accept_content parameters as suggested by ChillarAnand

from __future__ import absolute_import, unicode_literals
from celery import Celery

app = Celery('tasks', 
    broker='amqp://test:[email protected]:5672//',
    backend='amqp://',
    task_serializer='json',
    result_serializer='json',
    accept_content=['application/json'],
    include=['proj.tasks'])

But node.js Celery client still thinks that its not in json, throwing the same error message.

It gives that error because the results were in the form of 'application/x-python-serialize'.

Checked this to be the case, as RabbitMQ management console shows the results to be content_type: application/x-python-serialize

This forum post says that it is because the tasks were created before the configs were loaded.

Here are how my files are like:

proj/celery.py

from __future__ import absolute_import, unicode_literals
from celery import Celery

app = Celery('tasks', 
    broker='amqp://test:[email protected]:5672//',
    backend='amqp://',
    task_serializer='json',
    result_serializer='json',
    accept_content=['application/json'],
    include=['proj.tasks'])

proj/tasks.py

from __future__ import absolute_import, unicode_literals
from .celery import app

@app.task
def myTask():
    ...
    return ...

Is there a better way to structure the code to ensure that the configs are loaded before the tasks?

Did you manage to solve your problem?

leoschet
– leoschet

2018-06-28 00:21:43 +00:00
Commented Jun 28, 2018 at 0:21 — leoschet
– leoschet, Commented Jun 28, 2018 at 0:21

Chillar Anand · Accepted Answer · 2017-05-02 05:18:18Z

1

When configuring a serializer, you should specify content type, task serializer and result serializer as well.

app = Celery(
    broker='amqp://guest@localhost//',
    backend='amqp://',
    include=['proj.tasks'],

    task_serializer='json',
    result_serializer='json',
    accept_content = ['application/json'],
)

answered May 2, 2017 at 5:18

Chillar Anand

29.8k10 gold badges126 silver badges144 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Nyxynyx Over a year ago

I made the suggested changes for my Python Celery worker. but the node client is still giving the same error when receiving the results. I believe it gives that error because the results was in the form of 'application/x-python-serialize'. Any other setting to make to switch it to json?

Chillar Anand Over a year ago

For celery_node app, did you set CELERY_ACCEPT_CONTENT=['json'],CELERY_TASK_SERIALIZER='json', CELERY_RESULT_SERIALIZER='json'?

Collectives™ on Stack Overflow

Configure Celery with JSON Serializer (Python + node.js)

Update

1 Answer 1

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

Update

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related