Why python has limit for count of file handles?

Question

I wrote a simple code to test how many files may be open in python script:

for i in xrange(2000):
    fp = open('files/file_%d' % i, 'w')
    fp.write(str(i))
    fp.close()

fps = []
for x in xrange(2000):
    h = open('files/file_%d' % x, 'r')
    print h.read()
    fps.append(h)

and I get the exception:

IOError: [Errno 24] Too many open files: 'files/file_509'

@wiso, +stdin, stdout, stderr makes 1024 - where have I seen that number before? — John La Rooy
– John La Rooy, Commented Jul 21, 2011 at 10:55
You should use try..finally or with to safely close a file. To your problem: maybe you want to tell us what you are going to do because want your code does makes no sense at all for me. — schlamar
– schlamar, Commented Jul 21, 2011 at 11:33
@gnibber: ulimit -n gives me 1024. I think you need to count also /usr/lib64/python2.7/atexit.py and /home/xyz/.pystartup as opened files. — Ruggero Turra
– Ruggero Turra, Commented Jul 21, 2011 at 17:45
Were you to try this in another language on the same operating system, you'll quicky discover that this is not a Python limitation. — johnsyweb
– johnsyweb, Commented May 7, 2015 at 8:49

John La Rooy · Accepted Answer · 2011-07-21 11:03:00Z

45

The number of open files is limited by the operating system. On linux you can type

ulimit -n

to see what the limit is. If you are root, you can type

ulimit -n 2048

now your program will run ok (as root) since you have lifted the limit to 2048 open files

answered Jul 21, 2011 at 11:03

John La Rooy

306k54 gold badges378 silver badges514 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

mouad Over a year ago

+1, i will add if you want to check the limit using a python code use import resource; print resource.getrlimit(resource.RLIMIT_NOFILE).

chuse Over a year ago

Apparently, you don't need to be root to change it (I just tried!)

Dhara · Accepted Answer · 2015-01-29 11:46:12Z

15

I see same behavior on Windows when running your code. The limit exists from C runtime. You can use win32file to change the limit value:

import win32file

print win32file._getmaxstdio()

The above shall give you 512, which explains the failure at #509 (+stdin, stderr, stdout as others have already stated)

Execute the following and your code shall run fine:

win32file._setmaxstdio(2048)

Note that 2048 is the hard limit, though (hard limit of the underlying C Stdio). As a result, executing the _setmaxstdio with a value greater than 2048 fails for me.

edited Jan 29, 2015 at 11:46

Dhara

6,8052 gold badges34 silver badges48 bronze badges

answered Jan 29, 2015 at 10:35

Punit S

3,2571 gold badge24 silver badges26 bronze badges

2 Comments

B K Over a year ago

to make this work, I first had to do this install: pip install pywin32

Nick Muise Over a year ago

Also note that as of 2022, the C runtime libraries now support an upper limit of 8192 open files, not 2048. See Microsoft's documentation on the _setmaxstdio function.

Franck Dernoncourt · Accepted Answer · 2015-08-19 04:18:27Z

12

To check change the limit of open file handles on Linux, you can use the Python module resource:

import resource

# the soft limit imposed by the current configuration
# the hard limit imposed by the operating system.
soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
print 'Soft limit is ', soft 

# For the following line to run, you need to execute the Python script as root.
resource.setrlimit(resource.RLIMIT_NOFILE, (3000, hard))

On Windows, I do as Punit S suggested:

import platform

if platform.system() == 'Windows':
    import win32file
    win32file._setmaxstdio(2048)

answered Aug 19, 2015 at 4:18

Franck Dernoncourt

84.7k81 gold badges374 silver badges556 bronze badges

1 Comment

林果皞 Over a year ago

I noticed I don't have to be root to run setrlimit in Ubuntu.

Guffa · Accepted Answer · 2011-07-21 10:40:42Z

7

Most likely because the operating system has a limit for the number of files that an application can have open.

answered Jul 21, 2011 at 10:40

Guffa

703k111 gold badges760 silver badges1k bronze badges

Comments

ababak · Accepted Answer · 2022-01-18 08:33:09Z

2

On Windows one can get or set the limit with the built-in ctypes library:

import ctypes
print("Before: {}".format(ctypes.windll.msvcrt._getmaxstdio()))
ctypes.windll.msvcrt._setmaxstdio(2048)
print("After: {}".format(ctypes.windll.msvcrt._getmaxstdio()))

edited Jan 18, 2022 at 8:33

answered Sep 16, 2021 at 13:57

ababak

1,8031 gold badge13 silver badges25 bronze badges

Comments

S.Lott · Accepted Answer · 2011-07-21 12:53:55Z

0

Since this is not a Python problem, do this:

for x in xrange(2000):
    with open('files/file_%d' % x, 'r') as h:
        print h.read()

The following is a very bad idea.

fps.append(h)

answered Jul 21, 2011 at 12:53

S.Lott

393k83 gold badges521 silver badges791 bronze badges

2 Comments

EML Over a year ago

Whether this is a bad idea or not depends on what you're doing. Even Guido himself uses this strategy of holding onto open files in a particular sorting algorithm: neopythonic.blogspot.com/2008/10/…

iCanLearn Over a year ago

@S.Lott: Can you explain why you think fps.append(h) would be a bad idea (and which alternative would you suggest)?

andrew pate · Accepted Answer · 2014-07-08 10:14:14Z

-1

The append is needed so the garbage collector does not clean up and close the files

answered Jul 8, 2014 at 10:14

andrew pate

4,3771 gold badge43 silver badges32 bronze badges

Collectives™ on Stack Overflow

Why python has limit for count of file handles?

7 Answers 7

2 Comments

2 Comments

1 Comment

Comments

Comments

2 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

7 Answers 7

2 Comments

2 Comments

1 Comment

Comments

Comments

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related