Could not reserve memory block, import json error in python

Question

import pandas as pd
with open(r'data.json') as f:
   df = pd.read_json(f, encoding='utf-8')

I'm getting a "Could not reserve memory block" error. The JSON file is 300MB in size. Is there any limit for reserving memory for a running program in Python? I have 8GB RAM on PC, using Windows 10.

loading of json file into df
Traceback (most recent call last):
  File "C:\Program Files\JetBrains\PyCharm 2018.1.4\helpers\pydev\pydev_run_in_console.py", line 52, in run_file
    pydev_imports.execfile(file, globals, locals)  # execute the script
  File "C:\Program Files\JetBrains\PyCharm 2018.1.4\helpers\pydev\_pydev_imps\_pydev_execfile.py", line 18, in execfile
    exec(compile(contents+"\n", file, 'exec'), glob, loc)
  File "C:/Users/Beorn/PycharmProjects/project_0/projekt/test.py", line 7, in <module>
    df = pd.read_json(f, encoding='utf-8')
  File "C:\Users\Beorn\AppData\Local\Programs\Python\Python36-32\lib\site-packages\pandas\io\json\json.py", line 422, in read_json
    result = json_reader.read()
  File "C:\Users\Beorn\AppData\Local\Programs\Python\Python36-32\lib\site-packages\pandas\io\json\json.py", line 529, in read
    obj = self._get_object_parser(self.data)
  File "C:\Users\Beorn\AppData\Local\Programs\Python\Python36-32\lib\site-packages\pandas\io\json\json.py", line 546, in _get_object_parser
    obj = FrameParser(json, **kwargs).parse()
  File "C:\Users\Beorn\AppData\Local\Programs\Python\Python36-32\lib\site-packages\pandas\io\json\json.py", line 638, in parse
    self._parse_no_numpy()
  File "C:\Users\Beorn\AppData\Local\Programs\Python\Python36-32\lib\site-packages\pandas\io\json\json.py", line 853, in _parse_no_numpy
    loads(json, precise_float=self.precise_float), dtype=None)
ValueError: Could not reserve memory block
PyDev console: starting.
Python 3.6.6 (v3.6.6:4cf1f54eb7, Jun 27 2018, 02:47:15) [MSC v.1900 32 bit (Intel)] on win32

I have never seen that error, please show the full traceback. — roganjosh
– roganjosh, Commented Jul 14, 2018 at 9:05
mmm, my searches seem to indicate there's more at play here. Are you using Apache? — roganjosh
– roganjosh, Commented Jul 14, 2018 at 9:10
When Pandas reads a JSON file, it loads it into memory using Python's json module (the relevant line from your stacktrace is github.com/pandas-dev/pandas/blob/v0.23.3/pandas/io/json/…, which calls through to json.loads). If loading your data into a plain dictionary with json.load() gives the same error, then you can rule out Pandas the problem. There are a few questions on SO about problems with loading large JSON files (e.g. stackoverflow.com/questions/10382253/…) - so this is the first thing to try. — jelford
– jelford, Commented Jul 14, 2018 at 9:44

Michael Mior · Accepted Answer · 2022-05-17 13:28:39Z

2

So after reading plenty of posts and solutions, I decided to just reduce my file size by getting rid of useless data. Maybe you will find this useful. By the way, I read somewhere that you need at least 25x more memory than your JSON file has, so in my case, I needed more than 8GB.

with open('data.json', 'r') as data_file:
    data = json.load(data_file)

print(data.keys())
del data['author']

with open('datav2.json', 'w') as data_file:
    data = json.dump(data, data_file)

edited May 17, 2022 at 13:28

Michael Mior

28.9k10 gold badges95 silver badges118 bronze badges

answered Jul 14, 2018 at 11:05

Beorn

4372 gold badges7 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Could not reserve memory block, import json error in python

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related