GZip Error Reading from File in Python Script

Question

I have a .gz file that I'm trying to open and parse to be put into a database. Running the code below...

def process_file(filename):
    with gzip.GzipFile(filename, 'rU', 9) as uncompressed_file:
        uncompressed_file.next()  # Skip the headers

        for line in uncompressed_file:
            line = line.replace('\n', '').split('\t')
            # Do some more stuff with the line

Generates this error...

    File "path/to/script", line 169, in process_file
        uncompressed_file.next()  # Skip the headers
    File "/usr/lib/python2.7/gzip.py", line 450, in readline
        c = self.read(readsize)
    File "/usr/lib/python2.7/gzip.py", line 256, in read
        self._read(readsize)
    File "/usr/lib/python2.7/gzip.py", line 307, in _read
        uncompress = self.decompress.decompress(buf)
error: Error -3 while decompressing: invalid distance too far back

What's particularly strange is that the code works perfectly on my local machine (Mac OSX 10.9.4), but not on my server (Ubuntu 12.04.4 LTS).

Any insight is appreciated as I'm currently out of ideas.

Are you sure both archive files are same and one is not corrupted? — konsolebox
– konsolebox, Commented Aug 12, 2014 at 21:46
@konsolebox it appears the files are ok. I downloaded manually outside the program and unzipped them. — Patrick Boocock
– Patrick Boocock, Commented Aug 12, 2014 at 22:13
i have suddenly ecnoutered this issue. my scripts was previously working until suddenly i get these same errors from gzip.py. — Sydalmighty
– Sydalmighty, Commented May 12, 2020 at 6:24

Patrick Boocock · Accepted Answer · 2014-08-12 22:36:51Z

1

Resolved this issue. It seems that using the 'with' syntax on gzip.open or gzip.GzipFile isn't fully supported on all platforms.

It's still not clear why it wasn't working consistently, but moving to this code resolved the issue:

def process_file(filename):
    f = gzip.open(filename, 'rb')
    f.next() # Skip the headers

    for line in f:
        line = line.replace('\n', '').split('\t')

    f.close()

answered Aug 12, 2014 at 22:36

Patrick Boocock

471 silver badge7 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

GZip Error Reading from File in Python Script

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related