Memory Error while loading a large JSON file python 3.4

Question

I'm trying to load a large Json file (size: 8,5 GB) using the following command pandas.read_json(myfile)

which throws a memory error

MemoryError                               Traceback (most recent call last)
<ipython-input-6-88d3fb70f2f4> in <module>()
----> 1 df1=pd.read_json("part1")

/root/anaconda3/lib/python3.4/site-packages/pandas/io/json.py in read_json(path_or_buf, orient, typ, dtype, convert_axes, convert_dates, keep_default_dates, numpy, precise_float, date_unit, encoding, lines)
    279         obj = FrameParser(json, orient, dtype, convert_axes, convert_dates,
    280                           keep_default_dates, numpy, precise_float,
--> 281                           date_unit).parse()
    282 
    283     if typ == 'series' or obj is None:

/root/anaconda3/lib/python3.4/site-packages/pandas/io/json.py in parse(self)
    347 
    348         else:
--> 349             self._parse_no_numpy()
    350 
    351         if self.obj is None:

/root/anaconda3/lib/python3.4/site-packages/pandas/io/json.py in _parse_no_numpy(self)
    564         if orient == "columns":
    565             self.obj = DataFrame(
--> 566                 loads(json, precise_float=self.precise_float), dtype=None)
    567         elif orient == "split":
    568             decoded = dict((str(k), v)

MemoryError:

I've got a phisical RAM of 32 GB, i'm working on a CentOs virtual machine in which I have 28 GB of RAM. Is there a way to overcome this problem ? Thanks in advance

You would need 8.5 GB of memory to store the bytes read from the JSON file; you need far more to store a dict resulting from decoding it. — chepner
– chepner, Commented May 16, 2017 at 15:24

Exprator · Accepted Answer · 2017-05-16 15:26:46Z

1

break the json according to your available physical memory into files less than your ram, then load each and save it before loading another json file

answered May 16, 2017 at 15:26

Exprator

27.8k6 gold badges54 silver badges64 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

ABK Over a year ago

do you have any idea how to break the json file ?

Exprator Over a year ago

which os are you using>

Exprator Over a year ago

unix.stackexchange.com/questions/1588/… go through this link

MattSom Over a year ago

@Exprator Nice information but that leads not to a Python based solution.

Collectives™ on Stack Overflow

Memory Error while loading a large JSON file python 3.4

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related