Enumerating keys in Django Database Cache

Question

I'm caching some calculation results in a Django application using Django Database Cache (https://docs.djangoproject.com/en/1.8/topics/cache/#database-caching).

What is the way to enumerate all keys stored in the cache at a given moment. Occasionally I need to invalidate (delete) parts of the cache before expiration (because I'm debugging). The cache keys are generated by elaborate calculation and I don't want to repeat that computation. I know the prefix of the cache keys I would like to delete but I don't know the full key strings.

I don't see right away on the cache API how could I do that. I can get entries, create keys, delete entries and clear the whole cache: https://docs.djangoproject.com/en/1.8/topics/cache/#the-low-level-cache-api

Right now I have to extract the keys with SQL statements which is a PITA. I'd like to write a management command I can use to invalidate sections.

Example:

settings.py:

CACHES = {
    'default': {
         'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
         'LOCATION': 'default-cache',
    },
    'staticfiles': {
         'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
         'LOCATION': 'static-files',
    },
    'bla_stats': {
        'BACKEND': 'django.core.cache.backends.db.DatabaseCache',
        'LOCATION': 'django_bla_stats_cache',
    }
}

I created the cache as explained in the Django documentation I linked. Placing some data into the cache (no expiration: I control the entries).

from django.core.cache import caches

cache = caches['bla_stats']
cache.set("a_d3e6a1e1-0565-4d20-8887-4fda47186299", "foo", None)
cache.set("a_e79a1e0d-bfe1-4a04-8db3-42495c09e780", "bar", None)
cache.set("b_390d42ec-2b70-436d-8600-404034b07fe9", "fiz", None)
cache.set("b_a2d3cb52-8941-4812-8186-676ee3de0ec3", "baz", None)

And here is the question: how can I find all the keys in the cache which has the key prefix "b_" at any given moment?

This should be possible. If you add your existing code to the question it will be easier to understand exactly what you want to achieve and suggest alternative ways. — Håken Lid
– Håken Lid, Commented Jun 3, 2016 at 19:14

Bryant C · Accepted Answer · 2016-06-04 06:00:17Z

Assuming you are using MySQL as your DB backend this sub class of the DatabaseCache should work to return a dictionary of all the results from a LIKE type query on cache keys.

class DatabaseCacheExtended(DatabaseCache):
    def get_where(self, query, default=None, version=None):
        db = router.db_for_read(self.cache_model_class)
        table = connections[db].ops.quote_name(self._table)

        with connections[db].cursor() as cursor:
            cursor.execute("SELECT cache_key, value, expires FROM %s "
                           "WHERE cache_key LIKE %%s" % table, [query])
            rows = cursor.fetchall()
        if len(rows) < 1:
            return {}
        return_d ={}
        for row in rows:
            value = connections[db].ops.process_clob(row[1])
            return_d[row[0]] = pickle.loads(base64.b64decode(force_bytes(value)))
        return return_d

Then you just need to change the registered backend in your settings.py

    'bla_stats': {
        'BACKEND': 'path.to.DatabaseCacheExtended',
        'LOCATION': 'django_bla_stats_cache',
     }

Example:

>>> from django.core.cache import caches
>>> cache = caches['bla_stats']
>>> cache.get_where("b_%")
... {"b_key1":"val1", "b_key2":"val2"}

JSTL · Accepted Answer · 2016-06-03 22:13:42Z

1

Django's cache API doesn't offer what you are looking for, so none of the implementations will either. One solution would be to create your own backend.

You're still going to have to muck around some SQL queries, but you can subclass DatabaseCache to create your own custom backend. Add a method that allows you to query for keys by prefix, or to delete by prefix, and then wrap that into a management command for easy access.

answered Jun 3, 2016 at 22:13

JSTL

8681 gold badge14 silver badges26 bronze badges

Collectives™ on Stack Overflow

Enumerating keys in Django Database Cache

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related