Post multipart/form-data using pythons httpx library only form data

Question

How do I send a POST with python HTTPX that will minic a CURL POST that works? This is to an opengear rest API. I believe it has to do something with the data field of the post.

This is the working curl hitting the rest API properly.

curl -s -k \
-X POST \
-H "Authorization: Token ${TOKEN}" \
-H "Content-Type: multipart/form-data" \
-F firmware_url="https://example.com/path/to/file/file.name" \
-F firmware_options="-R" \
https://${SERVER}/api/v2/system/firmware_upgrade

Based on httpx documentation this should be a simple call with a dictionary of the fields passed in the post method. It also says that posts with files defaults to streaming. It does not clearly state posts with out files use streaming. Based on the results from the logs of the nginx server it appears that it's not a streaming process.

I'm modifying the header content-type as the server is expecting a multipart/form-data and the httpx.post defaults to the application/x-www-form-urlencoded.

The python and different types of posts that I've attempted not all at once.

# Get a session token and creates the httpx.Client object api_client.
# Multiple GETs are called from the client before trying to post.
...
# POST section
headers = {"Content-Type": "multipart/form-data"}
data = {
    "firmware_url": f"https://example.com/path/to/file/file.name",
    "firmware_options": "-R"
}

# Simple post
res = api_client.post(f"https://{SERVER}/api/v2/system/firmware_upgrade", data=data, headers=headers)

# Data converted to json string
res = api_client.post(f"https://{SERVER}/api/v2/system/firmware_upgrade", data=json.dumps(data), headers=headers)

# Data converted to byte encoded json string
byte_data = json.dumps(data).encode(encoding="utf-8")
res = api_client.post(f"https://{SERVER}/api/v2/system/firmware_upgrade", data=data, headers=headers)

The nginx logs show different lines depending on which post is used.

CURL with -F form fields (working, appears to be a stream)
::ffff:10.0.0.1 - - [24/Jan/2025:22:39:47 +0000] "POST /api/v2/system/firmware_upgrade HTTP/1.1" 200 54 "-" "curl/8.5.0" rt=0.054 uct="0.001" uht="0.054" urt="0.054"

HTTPX with raw data dict
::ffff:10.0.0.1 - - [24/Jan/2025:22:38:51 +0000] "POST /api/v2/system/firmware_upgrade HTTP/1.1" 000 0 "-" "python-httpx/0.28.1" rt=0.000 uct="-" uht="-" urt="-"
::ffff:10.0.0.1 - - [24/Jan/2025:22:38:51 +0000] "firmware_url=https%3A%2F%2Fexample.com%2Fpath%2Fto%2Ffile%2Ffile.name&firmware_options=-R" 400 157 "-" "-" rt=0.027 uct="-" uht="-" urt="-"

HTTPX with JSON dumps
::ffff:10.0.0.1 - - [24/Jan/2025:22:38:08 +0000] "POST /api/v2/system/firmware_upgrade HTTP/1.1" 000 0 "-" "python-httpx/0.28.1" rt=0.000 uct="-" uht="-" urt="-"
::ffff:10.0.0.1 - - [24/Jan/2025:22:38:08 +0000] "{\x22firmware_url\x22: \x22https://example.com/path/to/file/file.name\x22, \x22firmware_options\x22: \x22-R\x22}" 400 157 "-" "-" rt=0.028 uct="-" uht="-" urt="-"

HTTPX with JSON dumps to byte encoded
::ffff:10.0.0.1 - - [27/Jan/2025:19:01:45 +0000] "POST /api/v2/system/firmware_upgrade HTTP/1.1" 000 0 "-" "python-httpx/0.28.1" rt=0.000 uct="-" uht="-" urt="-"
::ffff:10.0.0.1 - - [27/Jan/2025:19:01:45 +0000] "{\x22firmware_url\x22: \x22https://example.com/path/to/file/file.name\x22, \x22firmware_options\x22: \x22-R\x22}" 400 157 "-" "-" rt=0.006 uct="-" uht="-" urt="-"

I found this link, have you tried doing something like this ⇾ r = httpx.post(data=data, headers=headers, files={'file': ''}). — Serhii Fomenko
– Serhii Fomenko, Commented Jan 28 at 13:02
Thanks, I did try adding the files empty field I received a 200 but the API responded the status of "error". Probably due to the file being empty when it's expecting a byte stream. It says at least 'file' or 'firmware_url' are required. I'm using URL from a HTTP server that is hosting the file rather than streaming it form my machine. POST methods that use json=data work. It's just the multipart/form-data and how it sends it it would appear. URLENCODED/JSON/BYTE strings. — krizzo
– krizzo, Commented Jan 28 at 19:48
I was just about to write some more ideas that you could still try as a response, if you're interested I can and post it. — Serhii Fomenko
– Serhii Fomenko, Commented Jan 28 at 19:57
In brief, the idea is to read a remote file using stream and redirect it immediately, this should work for you as well. — Serhii Fomenko
– Serhii Fomenko, Commented Jan 28 at 20:00

Serhii Fomenko · Accepted Answer · 2025-01-29 09:46:34Z

The official documentation states that form fields should be passed using the data keyword, using the dict object.

It does not clearly state posts with out files use streaming

Just passing form fields using POST will not use streaming. To make sure of this I did a bit of searching of the source code, this can be seen here, in the Request.encode_request method, if only data is passed, we will eventually get the headers and ByteStream object as the result of the function - encode_urlencoded_data, after which it will be read immediately into memory. For example, if we send something like this: httpx.post(url=url, data={'name': 'Foo'}), on the server I will see request.body=b'name=Foo'.

Since your endpoint accepts to send either file or firmware_url, my idea is as follows::

import httpx  
  
  
class GeneratorReader:  
    def __init__(self, gen_func):  
        self.gen_func = gen_func  
        self.active_gen = None  
  
    def read(self, chunk_size=1024):  
        if self.active_gen is None:  
            self.active_gen = self.gen_func(chunk_size=chunk_size)  
        return next(self.active_gen, None)  
  
  
file_url = 'https://example.com/path/to/file/file.name'  
client = httpx.Client()  
  
with client.stream(method='GET', url=file_url) as response:  
    client.post(  
        url='https://${SERVER}/api/v2/system/firmware_upgrade',  
        files={'file': GeneratorReader(gen_func=response.iter_bytes)}, 
        headers={  
            'Authorization': 'Token ${TOKEN}',  
            'Content-Type': 'multipart/form-data',  
        },  
    )

Essentially - all this code does is read your file using the stream method so as not to load everything into memory and redirects the data to the target endpoint. The GeneratorReader, we need because files={'file': ...}, must be either a string-like or file-like object with a read method.

Try doing it this way, should work. Alternatively, you could download your remote file (for example, through httpx) and save it as a local file and use something like this: files={'file': open('file.name', 'rb')}. It might also be interesting if you suddenly need to pass some metadata.

Collectives™ on Stack Overflow

Post multipart/form-data using pythons httpx library only form data

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related