python pickler - recursion depth exceeded

Question

I'm trying to pickle instance of my cellular automata class, but I get this error:

RuntimeError: maximum recursion depth exceeded while calling a Python object

My cellular automata consist from list of cells (and bunch of other things) where each cell has pointer to it's neighbours. In this particular CA, there is 256 cells. Now, I know that pickler should be able to recognise already pickled objects.

From docs:
*The pickle module keeps track of the objects it has already serialized, so that later references to the same object won’t be serialized again.

So I don't really know, why I exceeding max recursion depth.

I think that maybe pickler does depth-first pickling, so that it first follow pointers, exceed recursion stack and then raise exception. I know I can extend maximum recursion depth with sys.setrecursionlimit(), but I don't consider that good nor scalable solution.

First question: Does pickler depth-first pickling?
Second question: Any idea how to prevent this exception?

Yes, pickle goes depth-first. Unfortunately I don't think there is a around this. Try cPickle but it'll probably give the same error. — Elmar Peise
– Elmar Peise, Commented Apr 13, 2013 at 20:08
@AswinMurugesh actually I have too much code to post. However here are some key files if you are interested: neighbourhood cell pickling Exp: cPickle is the same, I tried it before. I was afraid of this answer — sjudǝʊ
– sjudǝʊ, Commented Apr 13, 2013 at 20:45
@ExP do you have some more specific info / resources about how pickle works internally? Maybe creating subclass of pickler? There's always way. I just need to take a closer look :) — sjudǝʊ
– sjudǝʊ, Commented Apr 15, 2013 at 21:33
@sjudǝʊ If you look at the output of pickle given dummy data, you will see that it gos depth first. Since I guess that the format does not allow forward references, I'm afraid you won't be able to modify or create subclass of it in order to solve your problem... — Elmar Peise
– Elmar Peise, Commented Apr 16, 2013 at 8:32

sjudǝʊ · Accepted Answer · 2013-04-16 18:07:19Z

6

So, as @ExP said, pickler does depth-first pickling which causes recursion exceeded exception. Anyway, I found solution to this issue here bugs.python.org. That means for python 3.1 pickler works even on recursive data such as graphs for example.

There is also little less elegant solution which takes a lot more time to pickle some recursive data, but it's simple (just a matter of few lines of code). Link here.

As it seems, it's probably time to start slowly moving towards the python3. Hope that someone find this answer usefull.

answered Apr 16, 2013 at 18:07

sjudǝʊ

1,5651 gold badge23 silver badges34 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

kevin948 Over a year ago

If you end up using the fix from bugs.python.org/issue2480, make sure you use the python file nonrecursivepickler-fixed.py over the other one. Also of note, this pickle solution compresses much less than vanilla pickle, so your pickle files will be larger. Thanks for the solution!

strubbly Over a year ago

Although this issue2480 version works, it is very inefficient for large structures because of its use of list.pop(0) and list.extend. On my large data structure it was taking 10 hours to pickle. A rewrite to use deques reduced that to 4 minutes.

Collectives™ on Stack Overflow

python pickler - recursion depth exceeded

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related