12

I am downloading a compressed file from the internet:

with lzma.open(urllib.request.urlopen(url)) as file:
    for line in file:
        ...

After having downloaded and processed a a large part of the file, I eventually get the error:

File "/usr/lib/python3.4/lzma.py", line 225, in _fill_buffer raise EOFError("Compressed file ended before the " EOFError: Compressed file ended before the end-of-stream marker was reached

I am thinking that it might be caused by an internet connection that drops or the server not responding for some time. If that is the case, is there anyway to make it keep trying, until connection is reestablished, instead of throwing an exception. I don't think it is a problem with the file, as I have manually downloaded many files like it from the same website manually and decompressed it. I have also been able to download and decompress some smaller files with Python. The file I am trying to download has a compressed size of about 20 GB.

Improve this question
5
  • How long does it take to download before you get the error? Some firewalls/proxies seem to terminate connections after a fixed timeout (e.g. 10 minutes). If it always fails after the same time interval, that may be a clue... Commented Apr 1, 2015 at 8:48
  • Possible duplicate of Python LZMA : Compressed data ended before the end-of-stream marker was reached Commented May 23, 2016 at 22:51
  • 2
    I'm having the same problem while trying to work with a very large file online using urllib.request.urlopen() and gzip. About 12 hours in I get a similar traceback. Commented Jun 29, 2016 at 20:21
  • Can't parse file if don't read headers(packet). Need check packet index and size(so urllib not resolved your problem). EOF and answer end header how to separate each one ? My opinion : urllib detect file EOFas answer END. Commented Jul 1, 2016 at 9:05
  • did you happen to solve this? Commented Apr 6, 2018 at 23:15

4 Answers 4

Reset to default
3

from the urllib.urlopen docs:

One caveat: the read() method, if the size argument is omitted or negative, may not read until the end of the data stream; there is no good way to determine that the entire stream from a socket has been read in the general case.

Maybe the lzma.open trips on huge size/connection errors/timeout because of the above.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

2

It's probably liblzma bug. As a workaround try adding:

lzma._BUFFER_SIZE = 1023

before calling lzma.open().

Improve this answer

Comments

2

Have you tried using the requests library? I believe it provides an abstraction over urllib.

The following solution should work for you, but it uses the requests library instead of urllib (but requests > urllib anyway!). Let me know if you prefer to continue using urllib.

import os
import requests
def download(url, chunk_s=1024, fname=None):
    if not fname:
        fname = url.split('/')[-1]
    req = requests.get(url, stream=True)
    with open(fname, 'wb') as fh:
        for chunk in req.iter_content(chunk_size=chunk_s):
            if chunk:
                fh.write(chunk)
    return os.path.join(os.getcwd(), fname)
Improve this answer

Comments

0

Assuming you need to download a big file, it is better to use the "write and binary" mode when writing content to a file in python.

You may also try to use the python requests module more than the urllib module:

Please see below a working code:

import requests
url="http://www.google.com"
with open("myoutputfile.ext","wb") as f:
    f.write( requests.get(url).content )

Could you test that piece of code and answer back if it doesn't solve your issue.

Best regards

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.