Django way to "Pre-cache" an existing database table?

Question

I have a 200MB sized csv file containing rows where a key term is matched against a list of strings inside the second column.

term_x | ["term_1","term_2"]
term_y | ["term_1","term_2"]
term_z | ["term_1","term_2"]

My Django app is not configured to use any complex memory caching (Redis, Memcached) and in practice, I want to pass a term into the database table to retrieve the corresponding list value. Due to its size however, retrieving the list from the correct row takes around half a second to do, on top of other actions being performed while loading the page.

Is it possible in Django to "pre-cache" this table upon server startup? i.e. add all of those values to the cache with the first column being the key? I have attempted something similar by overriding the "ready" method in my app.py to load the database table into the cache on startup, but I get null values when I try to use a term I know is in the table:

class MyAppConfig(AppConfig):

    name = 'test_display'

    def ready(self):
        print("Loading RS Lookup cache..."),

        #setup database connection....

        cache_df = pd.read_sql_query("Select * from table_to_cache", engine)

        print("Table loaded")

        for index, row in cache_df.iterrows():

             cache.set(row['term'], row['list_of_terms'], None)

        print("RS Table loaded")

My init.py in the same Django app has only one line:

default_app_config = 'test_display.apps.MyAppConfig'

Please provide the presumably not working code from apps.py (btw did you make sure you set the correct appconfig in the settings or in the __init__.py?) — Art
– Art, Commented Dec 6, 2017 at 15:57

blong · Accepted Answer · 2024-12-06 18:22:05Z

2

Check whether the following is correct:

In the project settings you did not configure caching or used the local memory caching as described in the documentation.
You only use the default cache (from django.core.cache import cache) or correctly handle cache names.

Make sure your code in .ready() actually stores the values you are trying to read later. You can use one of the following:

 assert "my_term" in cache, "The term is not cached!"

or

 from django.core.cache.backends import locmem
 print(locmem._caches)
 # now check what you have inside using your very own eyes and patience

As for the following:

Is it possible in Django to "pre-cache" ... ?

Your solution utilizes AppConfig.ready() which is generally a very good place for activities that your server should only perform once per instance. At least I am not aware of a better solution.

edited Dec 6, 2024 at 18:22

blong

2,7138 gold badges47 silver badges114 bronze badges

answered Dec 6, 2017 at 16:46

Art

2,3651 gold badge23 silver badges36 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Django way to "Pre-cache" an existing database table?

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related