django/celery data import fails to check for existing data

Question

I have running celery with django. I import a stream of objects into my database by using tasks. each task imports one object. concurrency is 2. within the stream objects can be duplicated, but should not be inside my database.

code i'm running:

if qs.exists() and qs.count() == 1:
    return qs.get()
elif qs.exists():
    logger.exception('Multiple venues for same place')
    raise ValueError('Multiple venues for same place')
else:
    obj = self.create(**defaults)

problem is that if objects inside the stream are duplicate and very close to each other, the app still imports the same objects twice.

I assume that the db checks are not working properly with this concurrency setup. what architecture du you recommend to resolve this issue?

ahmed · Accepted Answer · 2016-04-06 20:57:15Z

1

You have to use locking architecture, so will block the the two tasks from executing the object fetching part at the same time, you can use python-redis-lock to do that.

answered Apr 6, 2016 at 20:57

ahmed

5,6001 gold badge23 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

patroqueeet Over a year ago

wow, great advice. will go for it! fyi my interim solution was to put a unique constraint on two cols to have at least ensured that db stays consistent. the raised error kills the celery task. it's for now more a brute force-like attempt.

Collectives™ on Stack Overflow

django/celery data import fails to check for existing data

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related