Decoding gzipped http response that is inside json object

Question

I am attemping to get data from an HTTP post using the following snippet:

import request

# Using @ to signify personal info
url = "@@@@@"

querystring = {"user":"@@@@@yahoo.com"}

headers = {
    'token': 1234,
    'content-type': "application/x-www-form-urlencoded",
    'host': "@@@",
    'connection': "Keep-Alive",
    'accept-encoding': "gzip",
    'content-length': "0",
    'cache-control': "no-cache"
}

response = requests.request("POST", url, headers=headers,     params=querystring)

print(response.text)

I get the following result (the ... is the rest of the 'data' string):

{'data': 'H4sIAAAAAAAAAO...T7R358EQT/u3/8D+8Rtv5/DwIA', 'error_code': 0, 'response_desc': 'SUCCESS'}

Unlike all other questions I have found on stackoverflow, the gzipped data of interest lives within a json response. The json object itself is not compressed. I have tried

zlib.decompress(response.json().get("data"),15 + 32)

But get the error

    TypeError: a bytes-like object is required, not'str'

Am I going about handling the decoding correctly? I see plenty of documentation on decoding a gz file but this is just a string that is gz compressed.

Additional attempt: trying

zlib.decompress(response.json().get("data").encode(), 16 + zlib.MAX_WBITS)

Gives me the error:

incorrect header checkk

Try zlib.decompress(response.json().get("data").encode(), 15 + 32) — cs95
– cs95, Commented Jun 8, 2017 at 20:46
Hmm... I get an error: "incorrect header check". I feel confident that the data is valid gz since I can use an online decoder such as txtwizard.net/compression to get the correct decoded string. — kevbud007
– kevbud007, Commented Jun 8, 2017 at 20:48
Yes. Try zlib.decompress(response.json().get("data").encode(), 16 + zlib.MAX_WBITS) The previous header was incorrect it seems. — cs95
– cs95, Commented Jun 8, 2017 at 20:51
Is there a way you can share your compressed data? It would help with debugging. — cs95
– cs95, Commented Jun 8, 2017 at 20:58

kevbud007 · Accepted Answer · 2017-06-09 18:07:11Z

4

This answer's credit is to the authors of http://www.txtwizard.net/compression:

It turns out the string needed to be base64 decoded first before being decompressed with the format suggested by @coldspeed in the comments.

zlib.decompress(base64.b64decode(response.json().get("data")), 16 + zlib.MAX_WBITS)

answered Jun 9, 2017 at 18:07

kevbud007

991 silver badge11 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Soviut · Accepted Answer · 2017-06-08 20:47:53Z

0

You need to convert the string into a byte array first. This can be done two ways

bytes = bytes(your_string)

or

bytes = your_string.encode()

answered Jun 8, 2017 at 20:47

Soviut

92.2k53 gold badges210 silver badges285 bronze badges

Collectives™ on Stack Overflow

Decoding gzipped http response that is inside json object

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related