zlib.error: Error -5 while decompressing data: incomplete or truncated stream in Python

Question

I have been pulling my hair out trying to get a proxy working. I need to decrypt the packets from a server and client ((this may be out of order..)), then decompress everything but the packet header.

The first 2 packets ((10101 and 20104)) are not compressed, and decrypt, destruct, and decompile properly.

Alas, but to no avail; FAIL!; zlib.error: Error -5 while decompressing data: incomplete or truncated stream

Same error while I am attempting to decompress the encrypted version of the packet.

When I include the packet header, I get a randomly chosen -3 error.

I have also tried changing -zlib.MAX_WBITS to zlib.MAX_WBITS, as well as a few others, but still get the same error.

Here's the code;

import socket, sys, os, struct, zlib
from Crypto.Cipher import ARC4 as rc4

cwd = os.getcwd()

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
ss = socket.socket(socket.AF_INET, socket.SOCK_STREAM)

s.bind(('192.168.2.12',9339))
s.listen(1)
client, addr = s.accept()

key = "fhsd6f86f67rt8fw78fw789we78r9789wer6renonce"
cts = rc4.new(key)
stc = rc4.new(key)

skip = 'a'*len(key)
cts.encrypt(skip)
stc.encrypt(skip)

ss.connect(('game.boombeachgame.com',9339))

ss.settimeout(0.25)
s.settimeout(0.25)

def io():
    while True:
        try:
            pack = client.recv(65536)
            decpack = cts.decrypt(pack[7:])
            msgid, paylen = dechead(pack)
            if msgid != 10101:
                decopack = zlib.decompress(decpack, -zlib.MAX_WBITS)
            print "ID:",msgid
            print "Payload Length",paylen
            print "Payload:\n",decpack
            ss.send(pack)
            dump(msgid, decpack)
        except socket.timeout:
            pass
        try:
            pack = ss.recv(65536)
            msgid, paylen = dechead(pack)
            decpack = stc.decrypt(pack[7:])
            if msgid != 20104:
                decopack = zlib.decompress(decpack, -zlib.MAX_WBITS)
            print "ID:",msgid
            print "Payload Length",paylen
            print "Payload:\n",decpack
            client.send(pack)
            dump(msgid, decpack)
        except socket.timeout:
            pass

def dump(msgid, decpack):
    global cwd
    pdf = open(cwd+"/"+str(msgid)+".bin",'wb')
    pdf.write(decpack)
    pdf.close()

def dechead(pack):
    msgid = struct.unpack('>H', pack[0:2])[0]
    print int(struct.unpack('>H', pack[5:7])[0])
    payload_bytes = struct.unpack('BBB', pack[2:5])
    payload_len = ((payload_bytes[0] & 255) << 16) | ((payload_bytes[1] & 255) << 8) | (payload_bytes[2] & 255)
    return msgid, payload_len

io()

I realize it's messy, disorganized and very bad, but it all works as intended minus the decompression.

Yes, I am sure the packets are zlib compressed.

What is going wrong here and why?

Full Traceback:

Traceback (most recent call last):
  File "bbproxy.py", line 68, in <module>
    io()
  File "bbproxy.py", line 33, in io
    decopack = zlib.decompress(decpack, zlib.MAX_WBITS)
zlib.error: Error -5 while decompressing data: incomplete or truncated stream

Kirko · Accepted Answer · 2019-12-02 23:01:10Z

I ran into the same problem while trying to decompress a file using zlib with Python 2.7. The issue had to do with the size of the stream (or file input) exceeding the size that could be stored in memory. (My PC has 16 GB of memory, so it was not exceeding the physical memory size, but the buffer default size is 16384.)

The easiest fix was to change the code from:

import zlib
f_in = open('my_data.zz', 'rb')
comp_data = f_in.read()
data = zlib.decompress(comp_data)

To:

import zlib
f_in = open('my_data.zz', 'rb')
comp_data = f_in.read()
zobj = zlib.decompressobj()  # obj for decompressing data streams that won’t fit into memory at once.
data = zobj.decompress(comp_data)

It handles the stream by buffering it and feeding in into the decompressor in manageable chunks.

I hope this helps to save you time trying to figure out the problem. I had help from my friend Jordan! I was trying all kinds of different window sizes (wbits).

DavidC · Accepted Answer · 2018-11-09 11:16:58Z

Edit: Even with the below working on partial gz files for some files when I decompressed I got empty byte array and everything I tried would always return empty though the function was successful. Eventually I resorted to running gunzip process which always works:

def gunzip_string(the_string):
    proc = subprocess.Popen('gunzip',stdout=subprocess.PIPE, 
    stdin=subprocess.PIPE, stderr=subprocess.DEVNULL)
    proc.stdin.write(the_body)
    proc.stdin.close()
    body = proc.stdout.read()
    proc.wait()
    return body

Note that the above can return a non-zero error code indicating that the input string is incomplete but it still performs the decompression and hence the stderr being swallowed. You may wish to check errors to allow for this case.

/edit

I think the zlib decompression library is throwing an exception because you are not passing in a complete file just a 65536 chunk ss.recv(65536). If you change from this:

 decopack = zlib.decompress(decpack, -zlib.MAX_WBITS)

to

 decompressor = zlib.decompressobj(-zlib.MAX_WBITS)
 decopack = decompressor(decpack)

it should work as that way can handle streaming.

A the docs say

zlib.decompressobj - Returns a decompression object, to be used for decompressing data streams that won’t fit into memory at once.

or even if it does fit into memory you might just want to do the beginning of the file

Kirko · Accepted Answer · 2019-12-02 22:09:55Z

0

Try this:

decopack = zlib.decompressobj().decompress(decpack, zlib.MAX_WBITS)

edited Dec 2, 2019 at 22:09

Kirko

644 bronze badges

answered Jan 13, 2016 at 13:53

Alexandre Pollet

11

Collectives™ on Stack Overflow

zlib.error: Error -5 while decompressing data: incomplete or truncated stream in Python

3 Answers 3

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related