How to force http.client to send chunked-encoding HTTP body in python?

Question

I want to send chunked HTTP body to test my own HTTP server. So I wrote this python code:

import http.client

body = 'Hello World!' * 80

conn = http.client.HTTPConnection("some.domain.com")
url = "/some_path?arg=true_arg"

conn.request("POST", url, body, {"Transfer-Encoding":"chunked"})

resp = conn.getresponse()
print(resp.status, resp.reason)

I expect the HTTP request's body is transferrd chunked, but I capture the network package with Wireshark, the HTTP request's body is not transferred chunked.

How to transfer chunked body by http.client lib in python?

Celebi · Accepted Answer · 2012-02-11 07:23:20Z

8

OK, I get it.

First, write my own chunked encode function.

Then use putrequest(), putheader(), endheaders() and send() instead of request()

import http.client

def chunk_data(data, chunk_size):
    dl = len(data)
    ret = ""
    for i in range(dl // chunk_size):
        ret += "%s\r\n" % (hex(chunk_size)[2:])
        ret += "%s\r\n\r\n" % (data[i * chunk_size : (i + 1) * chunk_size])

    if len(data) % chunk_size != 0:
        ret += "%s\r\n" % (hex(len(data) % chunk_size)[2:])
        ret += "%s\r\n" % (data[-(len(data) % chunk_size):])

    ret += "0\r\n\r\n"
    return ret


conn = http.client.HTTPConnection(host)
url = "/some_path"
conn.putrequest('POST', url)
conn.putheader('Transfer-Encoding', 'chunked')
conn.endheaders()
conn.send(chunk_data(body, size_per_chunk).encode('utf-8'))

resp = conn.getresponse()
print(resp.status, resp.reason)
conn.close()

answered Feb 11, 2012 at 7:23

Celebi

1,3104 gold badges16 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

PW. Over a year ago

works better for me whith only one separator in the second line inside the for of chunk_data (ie: ret += "%s\r\n" % (data[i * chunk_size : (i + 1) * chunk_size]))

Community · Accepted Answer · 2017-05-23 10:26:34Z

I'd suggest that if you already know the size of your data like in the answer given you could just set the Content-Length and send it all back in one hit, which is kind of what you're doing with the single call to conn.send anyway.

Chunked transfer encoding is most useful when you don't know how big the data is e.g. dynamically generated content. I've modified your code to illustrate:

import httplib

def write_chunk(conn, data):
    conn.send("%s\r\n" % hex(len(data))[2:])
    conn.send("%s\r\n" % data)

def dynamically_generate_data():
    for i in range(80):
        yield "hello world"

conn = httplib.HTTPConnection("localhost")
url = "/some_path"
conn.putrequest('POST', url)
conn.putheader('Transfer-Encoding', 'chunked')
conn.endheaders()

for new_chunk in dynamically_generate_data():
    write_chunk(conn, new_chunk)
conn.send('0\r\n')

resp = conn.getresponse()
print(resp.status, resp.reason)
conn.close()

Collectives™ on Stack Overflow

How to force http.client to send chunked-encoding HTTP body in python?

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related