celery result_backend with postgresql - null for some fields in the task table

Question

I am using Celery for a POC. My objective is to create a workflow across a distributed system.

For the purpose of a POC I have created a couple of docker containers each with a worker started with the queue option. I initiated a chain from yet another container. The chain executes successfully across the containers. Now I decided to enable the result_backend to store the results of each task getting executed.

I set the result_backend to postgresql with the schema option. After executing the chain now, I do see the tables created in postgres. However the task_meta table has some columns as null (e.g. the worker, queue.) Where as the task_id and status are correctly populated.

Has anyone faced a similar issue? Any help is appreciated.

Could it be that there were no workers to receive the task, and it still got the default values? — DejanLekic
– DejanLekic, Commented Aug 27, 2020 at 14:52
Actually both the tasks in the chain got executed - with result of one passed to the other. I was unable to check what are the values for the args in the DB it being a bytea type. — LNB
– LNB, Commented Aug 27, 2020 at 15:02

Coweos · Accepted Answer · 2021-04-08 08:35:13Z

7

I know it has been a long time since you asked the question but I hope it will help others.

By default, Celery does not write all task result attributes to the backend. Your have to configure it by setting result_extended to True as stated here: https://docs.celeryproject.org/en/stable/userguide/configuration.html#result-extended

So if you configure your app from python you will have to do something like:

import celery

app = Celery(
    broker=broker_url,
    backend=postgresq_url,
)

app.conf.update(result_extended=True)

answered Apr 8, 2021 at 8:35

Coweos

711 silver badge4 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

celery result_backend with postgresql - null for some fields in the task table

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related