Error closing wget subprocess in Python

Question

I am downloading a file using wget in Python using the code below:

p1 = subprocess.Popen(['wget',
                       '-P', 
                       'path/to/folder','http://www.ncbi.nlm.nih.gov/Traces/wgs/?download=ACHI01.1.fsa_nt.gz'],
                       stdout=subprocess.PIPE)

p1.studout.close()

The file gets downloaded and saved correctly in the given folder but the process keeps running. I tried p1.kills() but that doesn't work either. Please advise.

Thanks!

Because you use stdout=subprocess.PIPE, you have to read stdout by p1.communicate(). — Nizam Mohamed
– Nizam Mohamed, Commented Jun 29, 2015 at 0:01
As an aside there's a third party wget module pypi.python.org/pypi/wget or use urllib2 (see here stackoverflow.com/questions/24346872/…). Philosophically, I think, wrapping linux cmds is not really the pythonic way of doing things (usually the "best" way is to call the underlying c libs directly using a third party python module someone else wrote for you). (no idea if this is what the wget python module does though). — demented hedgehog
– demented hedgehog, Commented Jun 29, 2015 at 0:13
Also this urlgrabber.baseurl.org suggested in this post stackoverflow.com/questions/2467609/using-wget-via-python — demented hedgehog
– demented hedgehog, Commented Jun 29, 2015 at 0:15

Stefan Gruenwald · Accepted Answer · 2015-06-28 23:59:02Z

1

Use subprocess.call

import subprocess 

subprocess.call (['wget', '-P', '/', 'http://www.ncbi.nlm.nih.gov/Traces/wgs/?download=ACHI01.1.fsa_nt.gz'])

answered Jun 28, 2015 at 23:59

Stefan Gruenwald

2,64828 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Prasad Over a year ago

Thanks everyone for the quick replies. using subprocess.call worked great !!

jfs Over a year ago

@user1777629: don't use stdout=PIPE with subprocess.call() function; it may hang the child process if it generates enough output to fill the OS pipe buffer.

jcoppens · Accepted Answer · 2015-06-28 23:53:48Z

0

A call to wait() is missing after the Popen, to make the main process wait till the child process is terminated.. The following code seems to work fine:

import subprocess as sp

def main(args):
p1 = sp.Popen(['wget', '-P path',
               'http://www.ncbi.nlm.nih.gov/Traces/wgs/?download=ACHI01.1.fsa_nt.gz'],
               stdout = sp.PIPE)
p1.wait()
return 0

if __name__ == '__main__':
    import sys
    sys.exit(main(sys.argv))

(You can also group commandline parameters with their values, if they have any).

answered Jun 28, 2015 at 23:53

jcoppens

5,4726 gold badges32 silver badges53 bronze badges

2 Comments

jfs Over a year ago

don't use stdout=PIPE unless you read from the pipe; it may block the subprocess. For example, try to run python -c "print(chr(97)*10**2)" as a subprocess. It should work as is with 10**2 characters. Change it to 10**5 or 10**6, to see that it blocks forever. Replace p1.wait() with output = p1.communicate()[0], to see that it does not block forever if the pipe is consumed. To discard output, use DEVNULL instead

Prasad Over a year ago

Hi @J.F.Sebastian, thanks for your suggestion. I changed the code to not use stdout and I am just using subprocess.call.

Collectives™ on Stack Overflow

Error closing wget subprocess in Python

2 Answers 2

2 Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related