Curl POST request into pycurl code

Question

I'm trying to convert following curl request into pycurl:

curl -v
-H Accept:application/json \
-H Content-Type:application/json \
-d "{
    name: 'abc',
    path: 'def',
    target: [ 'ghi' ]
}" \
-X POST http://some-url

I have following python code:

import pycurl, json

c = pycurl.Curl()
c.setopt(pycurl.URL, 'http://some-url')
c.setopt(pycurl.HTTPHEADER, ['Accept: application/json'])
data = json.dumps({"name": "abc", "path": "def", "target": "ghi"})
c.setopt(pycurl.POST, 1)
c.setopt(pycurl.POSTFIELDS, data)
c.setopt(pycurl.VERBOSE, 1)
c.perform()
print curl_agent.getinfo(pycurl.RESPONSE_CODE)
c.close()

Executing this I had an error 415: Unsupported media type, so I have changed:

c.setopt(pycurl.HTTPHEADER, ['Accept: application/json'])

into:

c.setopt(pycurl.HTTPHEADER, [ 'Content-Type: application/json' , 'Accept: application/json'])

This time I have 400: Bad request. But bash code with curl works. Do you have any idea what should I fix in python code?

ayush3504 · Accepted Answer · 2019-12-11 00:20:52Z

17

PycURL is a wrapper on the libcurl library written in C language so its Python API can be bit puzzling. As some people are advocating use of python requests instead I just want to point out that it isn't a perfect replacement. For me, its lack of DNS resolution timeout was a deal breaker. I also find it much slower on my Raspberry Pi. This comparison may be relevant: Python Requests vs PyCurl Performance

So here's something that doesn't evade OP's question:

import pycurl
import json
from cStringIO import StringIO

curl = pycurl.Curl()
curl.setopt(pycurl.URL, 'http://some-url')
curl.setopt(pycurl.HTTPHEADER, ['Accept: application/json',
                                'Content-Type: application/json'])
curl.setopt(pycurl.POST, 1)

# If you want to set a total timeout, say, 3 seconds
curl.setopt(pycurl.TIMEOUT_MS, 3000)

## depending on whether you want to print details on stdout, uncomment either
# curl.setopt(pycurl.VERBOSE, 1) # to print entire request flow
## or
# curl.setopt(pycurl.WRITEFUNCTION, lambda x: None) # to keep stdout clean

# preparing body the way pycurl.READDATA wants it
# NOTE: you may reuse curl object setup at this point
#  if sending POST repeatedly to the url. It will reuse
#  the connection.
body_as_dict = {"name": "abc", "path": "def", "target": "ghi"}
body_as_json_string = json.dumps(body_as_dict) # dict to json
body_as_file_object = StringIO(body_as_json_string)

# prepare and send. See also: pycurl.READFUNCTION to pass function instead
curl.setopt(pycurl.READDATA, body_as_file_object) 
curl.setopt(pycurl.POSTFIELDSIZE, len(body_as_json_string))
curl.perform()

# you may want to check HTTP response code, e.g.
status_code = curl.getinfo(pycurl.RESPONSE_CODE)
if status_code != 200:
    print "Aww Snap :( Server returned HTTP status code {}".format(status_code)

# don't forget to release connection when finished
curl.close()

There are some more interesting features worth checking out in the libcurl curleasy setopts documentation

edited Dec 11, 2019 at 0:20

answered Dec 6, 2019 at 4:18

ayush3504

7271 gold badge7 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Max Base Over a year ago

The StringIO and cStringIO modules are gone. Instead, import the io module and use io.StringIO or io.BytesIO for text and data respectively.

Max Base Over a year ago

Maybe useful:from io import StringIO

ayush3504 Over a year ago

@MaxBase Interesting but cStringIO has its own place as it is written in C unlike io.StringIO or StringIO.StringIO and it can be seen how well it performs in this benchmark A common use of PycURL which is written in C is to write higher performing code so I think readers would find cStringIO to be a better compliment.

martin10 Over a year ago

Thanks a lot, that helped me a lot after days of investigation

Daniel Hepper · Accepted Answer · 2015-08-05 09:08:37Z

10

In your bash example, the property target is an array, in your Python example it is a string.

Try this:

data = json.dumps({"name": "abc", "path": "def", "target": ["ghi"]})

I also strongly advise you to check out the requests library which has a much nicer API:

import requests
data = {"name": "abc", "path": "def", "target": ["ghi"]}
response = requests.post('http://some-url', json=data)
print response.status_code

edited Aug 5, 2015 at 9:08

answered Aug 5, 2015 at 9:02

Daniel Hepper

30.1k10 gold badges78 silver badges79 bronze badges

3 Comments

Nimrod Over a year ago

The requests library is broken in many way as to be useless for any server that is strict or slightly non-conforming. There are requests that work perfectly fine with curl but that get rejected when made with the requests library due to its various unnecessary sanitizing.

Daniel Hepper Over a year ago

@Nimrod thanks for your comment. I wasn't aware of that and never had any issues with requests myself, but would be interested in learning more about the problems you describe. Can you point me to any further information?

Chidananda Nayak Over a year ago

No,i strongly disagree, python requests is not always a champion. If you want to upload a file in readable stream then pycurl's FORM_FILE comes to rescue.

Brian · Accepted Answer · 2017-01-05 17:42:24Z

4

I know this is over a year old now, but please try removing the whitespace in your header value.

c.setopt(pycurl.HTTPHEADER, ['Accept:application/json'])

I also prefer using the requests module as well because the APIs/methods are clean and easy to use.

answered Jan 5, 2017 at 17:42

Brian

492 bronze badges

Comments

Nick Larsen · Accepted Answer · 2019-01-04 13:39:56Z

1

I had similar problem, and I used your code example but updated the httpheader section as follows:

c.setopt(pycurl.HTTPHEADER, ['Content-Type:application/json'])

edited Jan 4, 2019 at 13:39

Nick Larsen

18.9k6 gold badges70 silver badges97 bronze badges

answered Jan 4, 2019 at 12:31

Ashwin Venkat

113 bronze badges

Comments

IJL · Accepted Answer · 2017-01-06 00:22:09Z

-7

It's better simple to use requests library. (http://docs.python-requests.org/en/latest)

I append python code for your original curl custom headers.

import json
import requests

url = 'http://some-url'
headers = {'Content-Type': "application/json; charset=xxxe", 'Accept': "application/json"}
data = {"name": "abc", "path": "def", "target":  ["ghi"]}
res = requests.post(url, json=data, headers=headers)
print (res.status_code)
print (res.raise_for_status())

edited Jan 6, 2017 at 0:22

answered Jan 5, 2017 at 10:10

IJL

11 bronze badge

1 Comment

Ovidiu Dolha Over a year ago

Please explain why it's better

Collectives™ on Stack Overflow

Curl POST request into pycurl code

5 Answers 5

4 Comments

3 Comments

Comments

Comments

1 Comment

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

4 Comments

3 Comments

Comments

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Linked

Related