80

I am using subprocess to call another program and save its return values to a variable. This process is repeated in a loop, and after a few thousands times the program crashed with the following error:

Traceback (most recent call last):
  File "./extract_pcgls.py", line 96, in <module>
    SelfE.append( CalSelfEnergy(i) )
  File "./extract_pcgls.py", line 59, in CalSelfEnergy
    p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
  File "/usr/lib/python3.2/subprocess.py", line 745, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.2/subprocess.py", line 1166, in _execute_child
    errpipe_read, errpipe_write = _create_pipe()
OSError: [Errno 24] Too many open files

Code:

cmd = "enerCHARMM.pl -parram=x,xtop=topology_modified.rtf,xpar=lipid27_modified.par,nobuildall -out vdwaals {0}".format(cmtup[1])
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)
out, err = p.communicate()
Improve this question
2
  • 1
    Communicate() closes the pipe, so that's not your problem. In the end, Popen() is just the command that happens to run when you run out of pipes... the problem could be elsewhere in your code with other files being left open. I noticed "SelfE.append" ... are you opening other files and keeping them in a list? Commented May 13, 2013 at 17:38
  • did you try doing ulimit -Sn unlimited before running your python script? Commented Feb 15, 2021 at 21:32

11 Answers 11

Reset to default
70

In Mac OSX (El Capitan) See current configuration:

#ulimit -a
core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 256
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 709
virtual memory          (kbytes, -v) unlimited

Set open files value to 10K :

#ulimit -Sn 10000

Verify results:

#ulimit -a

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
file size               (blocks, -f) unlimited
max locked memory       (kbytes, -l) unlimited
max memory size         (kbytes, -m) unlimited
open files                      (-n) 10000
pipe size            (512 bytes, -p) 1
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 709
virtual memory          (kbytes, -v) unlimited
Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

the output of ulimit -a looks slightly different as of Oktober 2019 (El Capitan 10.11.6), e.g. -n is now "file descriptors" not "open files" -n: file descriptors. But ulimit -Sn 50000 solved my issue. Thanks.
why not ulimit -Sn unlimited?
Seconding the above comment. ulimit -Sn unlimited makes sense but is it too dangerous for blocking other system processes or something similar?
30

I guess the problem was due to the fact that I was processing an open file with subprocess:

cmd = "enerCHARMM.pl -par param=x,xtop=topology_modified.rtf,xpar=lipid27_modified.par,nobuildall -out vdwaals {0}".format(cmtup[1])
p = subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True)

Here the cmd variable contain the name of a file that has just been created but not closed. Then the subprocess.Popen calls a system command on that file. After doing this for many times, the program crashed with that error message.

So the message I learned from this is

Close the file you have created, then process it

Improve this answer

Comments

21

You can try raising the open file limit of the OS:

ulimit -n 2048
Improve this answer

2 Comments

Actually that command will not raise the limit above what has been set in /etc/security/limits.conf. To raise it, you need to place lines like these * soft nofile 4096 / * hard nofile 4096 in that file (replace 4096 with your own value).
Ran into this problem yesterday, and I had to edit both /etc/security/limits.conf AND raise the limit via ulimit -n in ubuntu to overcome this error.
12

As others have noted, raise the limit in /etc/security/limits.conf and also file descriptors was an issue for me personally, so I did

sudo sysctl -w fs.file-max=100000

And added to /etc/sysctl.conf:

fs.file-max = 100000

Reload with:

sudo sysctl -p

Also if you want to make sure that your process is not affected by anything else (which mine was), use

cat /proc/{process id}/limits

to find out what the actual limits of your process are, as for me the software running the python scripts also had its limits applied which have overridden the system wide settings.

Posting this answer here after resolving my particular issue with this error and hopefully it helps someone.

Improve this answer

Comments

6

A child process created by Popen() may inherit open file descriptors (a finite resource) from the parent. Use close_fds=True on POSIX (default since Python 3.2), to avoid it. Also, "PEP 0446 -- Make newly created file descriptors non-inheritable" deals with some remaining issues (since Python 3.4).

Improve this answer

12 Comments

I don't think this works, at least in all cases. I generated 1200 sleep spawned processes on a system with a 1024 open file limit (default on Ubuntu) and it blew up even with close_fds=True. So I think there's more to it than that. Since you still have over the limit in open processes anyways and this only works if you assume that the problem lies in finished processes that left open file descriptors.
@Sensei it does work: open files in the parent (make sure the fds are inheritable) then spawn subprocesses with close_fds=False (both are default on old Python versions, follow the links). See how much sooner you'll get the error. Obviously close_fds can't prevent the error in the general case: you don't even need to spawn a new process, to get it.
Except that it doesn't. I ran a simple for loop and generated enough subprocesses to hit the OS limit. I did this with close_fds=True. It had no impact. I may be wrong about why but my guess is simply that this solution only works if you're spawning a few subprocesses and never cleaning up the descriptors. In such a case this argument makes sense but I don't see it working if you actually intend to spawn and run that many processes at once.
@Sensei: I know that it works because there are tests in the stdlib that exercise this option (i.e., I know that it works not only for me). Now, your code may not work as you expect it. In that case, create a minimal but complete code example, describe the exact behavior that you expect and what happens instead step by step and publish it as a separate SO question (mention OS, Python version).
I think there is a misunderstanding between the two of you, I guess what Sensei says is that if you spawn (and don't terminate) many processes it will still crash. Whereas I think what you say is if you spawn many subprocess (that will at some point terminate) then this solution works. I have a case where I just ran many asyncio.create_subprocess_exec (most of them sequentially, something like at most 10 opened simultaneously) and I still had a "bug", when I looked at how many descriptors where opened by my script the number was way higher than 10, way way higher. I'm trying with your idea.
|
6

Maybe you are invoking the command multiple times. If so, each time you're doing stdout=subprocess.PIPE. Between each call try doing p.stdout.close().

Improve this answer

Comments

6

Use context managers instead:

cmd = "enerCHARMM.pl -param=x,xtop=topology_modified.rtf,xpar=lipid27_modified.par,nobuildall -out vdwaals {0}".format(cmtup[1])
with subprocess.Popen(cmd, stdout=subprocess.PIPE, shell=True) as p:
    out, err = p.communicate()

This will close p.stdout and p.stderr after the last line.

Related codes in Python: https://github.com/python/cpython/blob/208a7e957b812ad3b3733791845447677a704f3e/Lib/subprocess.py#L1031-L1038

Related document: https://docs.python.org/3/library/subprocess.html#subprocess.Popen

Improve this answer

Comments

3

If you are working on Linux you can easily debug this problem

1 - Start the command that eventually will fail due Too Many Open Files in a terminal.

python -m module.script

2 - Let it run for a while (so it can start opening the actual files) and whenever you believe it has done so just press CTRL+Z so the process will be suspended. You will have an output with the process id.

^Z
[2]  + 35245 suspended  python -m module.script

35245 is your PID.

3 - Now you can check what files are actually opened and not closed.

ls -alht /proc/35245/fd/

In my case I was doing something very similar to the original post, but I was creating a temporarily file with tempfile.mkstemp() before adding some data and actually running the subprocess.Popen.

In this case you need to close the file twice, once for adding the information and the second one due mkstemp

fd, path = tempfile.mkstemp()
with open(path, "wb") as f:
    f.write(bytes('my data', encoding='utf8'))
    f.close()   # this is one time
process = subprocess.Popen("my command that requires the previous file" ,[...])
os.close(fd)   # this is second time and the one I missed
Improve this answer

2 Comments

Thanks for the command to look into open files using the PID, very useful. However, I have one question about your code snippet: Isn't the whole point of using with with open that it will automatically close the file at the end of the block? Why are you closing it explicitly inside the with block?
@Iotif you are right. with with open should close the file at the end of the block. I would have to test again to check, but answers as this one stackoverflow.com/a/50113736/998649 would definitely create the problem.
2

Raise the limit to e.g. 32768 by adding the following lines to /etc/security/limits.conf:

* soft nofile 32768
* hard nofile 32768

Then, run ulimit -n 32768 as well.

Source: Dan D.'s comment.

Improve this answer

Comments

0

I raised the limit in /etc/security/limits.conf but my python process still only had a limit of max 1024 open files.

My python process is running as a service, turns out systemd injects 1024 as a default and thus overrides the setting in limits.conf. I needed to add this into /etc/systemd/system/my.service:

[Service]
LimitNOFILE=327680
Improve this answer

Comments

-1

opens file in subprocess. It is blocking call.

ss=subprocess.Popen(tempFileName,shell=True)
 ss.communicate()
Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.