MemoryError while pickling data in python

Question

I am trying to dump a dictionary into pickle format, using 'dump' command provided in python. The file size of the dictionary is around 150 mb, but an exception occurs when only 115 mb of the file is dumped. The exception is:

Traceback (most recent call last): 
  File "C:\Python27\generate_traffic_pattern.py", line 32, in <module> 
    b.dump_data(way_id_data,'way_id_data.pickle') 
  File "C:\Python27\class_dump_load_data.py", line 8, in dump_data 
    pickle.dump(data,saved_file) 
  File "C:\Python27\lib\pickle.py", line 1370, in dump 
    Pickler(file, protocol).dump(obj) 
  File "C:\Python27\lib\pickle.py", line 224, in dump 
    self.save(obj) 
  File "C:\Python27\lib\pickle.py", line 286, in save 
    f(self, obj) # Call unbound method with explicit self 
  File "C:\Python27\lib\pickle.py", line 649, in save_dict 
    self._batch_setitems(obj.iteritems()) 
  File "C:\Python27\lib\pickle.py", line 663, in _batch_setitems 
    save(v) 
  File "C:\Python27\lib\pickle.py", line 286, in save 
    f(self, obj) # Call unbound method with explicit self 
  File "C:\Python27\lib\pickle.py", line 600, in save_list 
    self._batch_appends(iter(obj)) 
  File "C:\Python27\lib\pickle.py", line 615, in _batch_appends 
    save(x) 
  File "C:\Python27\lib\pickle.py", line 286, in save 
    f(self, obj) # Call unbound method with explicit self 
  File "C:\Python27\lib\pickle.py", line 599, in save_list 
    self.memoize(obj) 
  File "C:\Python27\lib\pickle.py", line 247, in memoize 
    self.memo[id(obj)] = memo_len, obj 
MemoryError

I am really confused, since my same code was working fine earlier.

This is not specific to Pickle. Python is requesting more memory from the OS to store some more objects and the OS told Python that there is no more memory available for the process. This error could have happened anywhere in your code. — Martijn Pieters
– Martijn Pieters, Commented May 6, 2013 at 17:09
To test the code, I even tried to load the same pickle file (which I have dumped earlier) and then tried to dump it again, and strangely I am getting the same exception. — tanzil
– tanzil, Commented May 6, 2013 at 17:10
When pickling, the same loop keeps creating objects as needed, so it could be that the same location triggers the same exception, yes. Apparently, there was either more memory available earlier on your machine, or your Python session used less memory in other locations, and did not hit the limits set by the OS. — Martijn Pieters
– Martijn Pieters, Commented May 6, 2013 at 17:11
If this is a systemic problem for your script, your options are to get more memory, lift the OS per-process restrictions (using ulimit perhaps) or profiling your application memory use (see Which Python memory profiler is recommended?). — Martijn Pieters
– Martijn Pieters, Commented May 6, 2013 at 17:14

Vajk Hermecz · Accepted Answer · 2013-08-27 09:01:32Z

3

Are you dumping just that one object, and that's all?

If you are calling dump many times, then calling Pickler.clear_memo() between dumps will flush the internally stored backreferences (causing the 'leak'). And your code should just work fine...

answered Aug 27, 2013 at 9:01

Vajk Hermecz

5,8122 gold badges36 silver badges27 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

richie · Accepted Answer · 2013-12-11 09:30:28Z

3

Have you tried this?

import cPickle as pickle
p = pickle.Pickler(open("temp.p","wb")) 
p.fast = True 
p.dump(d) # d is your dictionary

answered Dec 11, 2013 at 9:30

richie

18.7k19 gold badges53 silver badges71 bronze badges

3 Comments

Luca Citi Over a year ago

This did the trick for me. If you are sure each object is only referenced once, it can significantly reduce the memory footprint.

Amarpreet Singh Over a year ago

Hey @richie How to load it back so that it doesn't cause memoryerror while loading the pickled file back?

Stefan Over a year ago

Why does this work?

Collectives™ on Stack Overflow

MemoryError while pickling data in python

2 Answers 2

Comments

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related