Python Overflow Error: iter index too large

Question

I have a problem with big iterator in for loop in the code below. It generates floats by reading a string list containing numbers.

def float_generator(tekstowe):
        x = ''
        for c in tekstowe:
            if c != ' ':
                x += c
            else:
                out = float(x)
                x = ''
                yield(out)

I'm getting a "OverflowError: iter index too large". I try to use really big iter numbers (like billions of values in a searched file). Is iter range somehow limited in for loops?

Using Python 2.7 64 bit. Thanks.

They are 0-255 valued and look a bit random like this: 25.75000000 0.340 22.33333397 0.667. I mean these are 4 example floats. — Konrad Gje
– Konrad Gje, Commented Sep 13, 2016 at 12:28
The answer is here github.com/python/cpython/blob/… and here docs.python.org/2/c-api/long.html#c.PyLong_AsSsize_t — Padraic Cunningham
– Padraic Cunningham, Commented Sep 13, 2016 at 12:33

ShadowRanger · Accepted Answer · 2016-09-13 13:04:00Z

Looks like tekstowe is a sequence type that only implements __getitem__, not __iter__, so it's using the Python iterator wrapper that calls __getitem__ with 0, then 1, 2, 3, etc., until __getitem__ raises IndexError.

As an implementation detail, Python 2.7.11 and higher limits the value of the index passed by the iterator wrapper to LONG_MAX (before 2.7.11, it wasn't bounds checked but it still used a long for index storage, so it would wrap and start indexing with negative values). This doesn't matter on most non-Windows 64 bit builds, where LONG_MAX is 2**63 - 1 (larger than you'd likely encounter), but on Windows, C longs remain 32 bit quantities even on 64 bit builds, so LONG_MAX remains 2**31 - 1, which is low enough to be reached in human timescales.

Your options are:

Change the implementation of whatever class tekstowe is to give it a true __iter__ method, so it doesn't get wrapped by the sequence iterator wrapper when you use it
Upgrade to Python 3.4+, ideally 3.5 (2.7.10/3.4.3 and below lacks the check for overflow entirely, but this could mean wraparound causes infinite looping; 3.4.4/3.5.0 added the check, and they use a signed size_t, testing against PY_SSIZE_T_MAX, which means it will not error until the index reaches 2**63 - 1 on any 64 bit build, Windows or otherwise)

The changes to add the overflow checks were made to resolve Python bug #22939; the type change (from long to Py_ssize_t) for the sequence iterator's index storage occurred in 3.4.0's release, resolving Python bug #17932.

Collectives™ on Stack Overflow

Python Overflow Error: iter index too large

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related