Django cache.set() causing duplicate key error

Question

My Django site recently started throwing errors from my caching code and I can't figure out why...

I call:

from django.core.cache import cache
cache.set('blogentry', some_value)

And the error thrown by Django is:

TransactionManagementError: This code isn't under transaction management

But looking at the PostgreSQL database logs, it seems to stem from this error:

STATEMENT:  INSERT INTO cache_table (cache_key, value, expires) VALUES (E'blogentry', E'pickled_version_of_some_value', E'2009-07-27 11:10:26')
ERROR:  duplicate key value violates unique constraint "cache_table_pkey"

For the life of me I can't figure out why Django is trying to do an INSERT instead of an UPDATE. Any thoughts?

Isn't caching to the database kind of defeating the purpose of caching? — thedz
– thedz, Commented Jul 27, 2009 at 21:52
I edited it to "blogentry" there, but it's actually caching a whole slew of related data for a blog sidebar widget. — Gabriel Hurley
– Gabriel Hurley, Commented Jul 27, 2009 at 23:54

Glenn Maynard · Accepted Answer · 2009-07-27 21:38:55Z

4

That's a typical race. It checks if the key you inserted exists; if it doesn't, it does an insert, but someone else can insert the key between the count and the insert. Transactions don't prevent this.

The code appears to expect this and to try to deal with it, but when I looked at the code to handle this case I could see immediately that it was broken. Reported here: http://code.djangoproject.com/ticket/11569

I'd strongly recommend sticking to the memcache backend.

answered Jul 27, 2009 at 21:38

Glenn Maynard

57.9k11 gold badges123 silver badges133 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Gabriel Hurley Over a year ago

I'd read through the source for the db cache backend and was thinking the same thing about the ticket you opened. Glad it wasn't just me feeling like I was missing something here. We're gonna set up memcached instead. Reading the source for the mecached backend is so much more reassuring. Thanks!

hughdbrown · Accepted Answer · 2009-07-27 21:25:55Z

0

The code in core/cache/backend/db.py reads in part:

cursor.execute("SELECT cache_key, expires FROM %s WHERE cache_key = %%s" % self._table, [key])
try:
    result = cursor.fetchone()
    if result and (mode == 'set' or
            (mode == 'add' and result[1] < now)):
        cursor.execute("UPDATE %s SET value = %%s, expires = %%s WHERE cache_key = %%s" % self._table, [encoded, str(exp), key])
    else:
        cursor.execute("INSERT INTO %s (cache_key, value, expires) VALUES (%%s, %%s, %%s)" % self._table, [key, encoded, str(exp)])

So I'd say that you are doing the INSERT INTO instead of the UPDATE because result evaluates to false. For some reason, cursor.fetchone() returns 0 rows when there is actually one there.

if you can't break in a debugger here, I'd put trace statements into the source to confirm that this is actually happening.

answered Jul 27, 2009 at 21:25

hughdbrown

49.3k20 gold badges89 silver badges111 bronze badges

1 Comment

Gabriel Hurley Over a year ago

yeah, I'd read the source over and over... as Glenn pointed out, the real culprit here is that the db backend has potential concurrency problems where a second INSERT statement can occur after the select returns 0, causing the original cache.set() to fail which then triggers a non-functional transaction.rollback() call.

gerdemb · Accepted Answer · 2011-10-21 19:57:42Z

0

I solved this problem by creating a custom cache backend, overriding the _base_set() function and changing the INSERT INTO statement like this. This SQL trick prevents the INSERT from happening in the case the cache_key already exists.

cursor.execute("INSERT INTO %s (cache_key, value, expires) SELECT %%s, %%s, %%s WHERE NOT EXISTS (SELECT 1 FROM %s WHERE cache_key = %%s)" % (table, table),
               [key, encoded, connections[db].ops.value_to_db_datetime(exp), key])

answered Oct 21, 2011 at 19:57

gerdemb

11.5k17 gold badges68 silver badges74 bronze badges

Collectives™ on Stack Overflow

Django cache.set() causing duplicate key error

3 Answers 3

1 Comment

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related