Python open json file keep in memory

Question

Question, I have an application that accesses data in a json file. Right now, every time the application needs the data, I will open and close the file as such.

def access_file():
    try:
        with open(my_file, 'r') as json_data:
            json_data = json.load(json_data)
            return json_data
    except FileNotFoundError:
        logging.error("my_file not found.")

I am assuming it is not smart to continually be opening and closing this file so many times. How do I only open it once if it's not open and keep it open, then I can just access in memory if I need it.

If the file is not very huge you can simply preserve it in an object which is accessible in required namespaces. And there are plenty of ways to do such thing which is strongly related to the rest of your code and how you using this function in your code. — Kasravnd
– Kasravnd, Commented May 17, 2018 at 15:22
Let's say the file is huge, and will become even bigger. Is there a way to do it? — WhitneyChia
– WhitneyChia, Commented May 17, 2018 at 15:23
Assuming your json file ever really gets too big to hold it in memory, then the problem would first occur inside the json module trying to parse it, not your code trying to hold the result in variable. So as long as json_data can be returned, everything is fine. — Jeronimo
– Jeronimo, Commented May 17, 2018 at 15:39

Samuel Dion-Girardeau · Accepted Answer · 2018-05-17 16:43:25Z

5

One way to do it without too much overhead is to use the standard lib's lru_cache. You can decorate a function with it, and it will remember the result of that function (depending on the arguments, in this case none). Next time the function is called, the result will be returned from the cache in memory, rather than re-executing the function.

As you can see in this example, this works with a very minimal addition to your code.

import json
from functools import lru_cache

my_file = 'foo.json'

@lru_cache(maxsize=1)
def access_file():
    try:
        with open(my_file, 'r') as json_data:
            json_data = json.load(json_data)
            return json_data
    except FileNotFoundError:
        logging.error("my_file not found.")


print(access_file())

import os
os.remove(my_file)

print(access_file())

Here I even remove the file to prove it really works, but I advise you not to :) If you run this code, you'll see the content of the JSON file printed twice.

answered May 17, 2018 at 16:43

Samuel Dion-Girardeau

3,2401 gold badge35 silver badges41 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

WhitneyChia Over a year ago

Thanks Samuel! This looks like what I need, will test this out. Quick question though, let's say my application is running and I make a json change, would I have to restart the application to clear the cache? Or should I build in a way to clear the cache?

Samuel Dion-Girardeau Over a year ago

@WhitneyChia If the JSON changes while you are running the application, you will use the old version, so you have to be careful.

Samuel Dion-Girardeau Over a year ago

@WhitneyChia Some approaches to mitigate that: 1) Keep an md5 (or some other hash) of the file content, and only look in the cache if the hash hasn't changed, 2) store the last modification date os.path.getmtime and read the file if the last modified time doesn't match the one you stored.

Collectives™ on Stack Overflow

Python open json file keep in memory

1 Answer 1

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related