Curl vs requests python 3

Question

I am doing a curl post request to a service:

curl -v --data "cp4=2765&cp3=350&method%3AsearchPC2=Procurar" https://www.ctt.pt/feapl_2/app/open/postalCodeSearch/postalCodeSearch.jspx

I can see that it successful because we have a div with the results in the response body:

...
<div class="highlighted-result text-left">

    <h4 class="subheader">Rua Sacadura Cabral</h4>


    <h4 class="subheader">Ímpares de 11 a 233</h4>


    <h3 class="subheader">Galiza</h3>


    <h2>2765-350 ESTORIL</h2>

</div>
...

The wired thing is that if I do it with python + requests, it doesn't give me the expected result as the curl above does, I even tried to set the user agent to the same as curl:

import requests as r

headers_p = {
    'User-Agent': 'curl/7.47.0',
    'Host': 'www.ctt.pt'
}

payload = {'cp4': 2765, 'cp3': 350, 'method':'',  'searchPC2': 'Procurar'}
req_p = r.post('https://www.ctt.pt/feapl_2/app/open/postalCodeSearch/postalCodeSearch.jspx', data=payload)
print(req_p.text) # doesn't have the the same content as the curl, I need the html block above

But it fails, the server doesn't send me the results html block

headers = {'Content-Type': 'application/xml'} use this snippet and pass this as part of r.post as headers=headers . See if you are getting output. — Hara
– Hara, Commented May 22, 2017 at 11:13
Thanks @Haranadh. Nop, didn't work. The curl, the one that works, sends Content-Type: application/x-www-form-urlencoded on the request header — Hula Hula
– Hula Hula, Commented May 22, 2017 at 11:15
you can print the response object and see the data. I am able to run in the python3 also. I can see some data. But not seeing specific html content as part of result. My python version is Python 3.4.3, installed in venv. — Hara
– Hara, Commented May 22, 2017 at 11:44
I know that @KeerthanaPrabhakaran, is there something unclear about the question? I just saying it's diferent from the curl request, I want the same response body, with the block html posted above — Hula Hula
– Hula Hula, Commented May 22, 2017 at 15:29

IsaacE · Accepted Answer · 2017-11-08 13:07:49Z

2

In case you have a proxy configured at your environment, define it at your session/request as well.

For example with session:

    my_proxies = {  
        'http': 'http://myproxy:8080',  
        'https': 'https://myproxy:8080'  
    }

    session = requests.Session()  
    request = requests.Request('POST', 'http://my.domain.com', data=params_template, headers=req_headers, proxies=my_proxies)  
    prepped = session.prepare_request(request)  
    response = session.send(prepped)

see documentation:
request http://docs.python-requests.org/en/master/user/quickstart/
session http://docs.python-requests.org/en/master/user/advanced/

Another option can be the security, in case you use have a problem with the ssl, add verify=False For example:

    response = requests.get('http://my.domain.com', verify=False)

answered Nov 8, 2017 at 13:07

IsaacE

3052 silver badges10 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Hara · Accepted Answer · 2017-05-22 11:28:47Z

0

When I try the following i am getting output.

import requests as r
from requests import Response

headers = {'Content-Type': 'application/xml'}

payload = {'cp4': 2765, 'cp3': 350, 'method':'',  'searchPC2': 'Procurar'}
given_url = 'https://www.ctt.pt/feapl_2/app/open/postalCodeSearch/postalCodeSearch.jspx'
req_p = r.post(given_url, data=payload, headers=headers)

print req_p, req_p.text

Try to parse the content as you need. I am getting the output. I tried to change the headers = {'Content-Type': 'application/x-www-form-urlencoded'} still i am getting some output.

Note: I am using python2.7

answered May 22, 2017 at 11:28

Hara

1,5024 gold badges19 silver badges35 bronze badges

1 Comment

Hula Hula Over a year ago

Thanks. The problem remains, I'm getting the out as well, just it doesn't give the html I want, unlike curl that It's outping the expected, I just need to why/how to make it work, and why it works with curl but doesn't with requests.

user3930728 · Accepted Answer · 2017-11-08 10:26:27Z

0

Please use the below code as the server is searching for json format.

import requests as r
import json
from requests import Response
headers = {'Content-Type': 'application/json'}
payload = {'cp4': 2765, 'cp3': 350, 'method':'',  'searchPC2': 'Procurar'}
data_json = json.dump(payload)
given_url = 'https://www.ctt.pt/feapl_2/app/open/postalCodeSearch/postalCodeSearch.jspx'
req_p = r.post(given_url, data=data_json, headers=headers)
print req_p, req_p.text

edited Nov 8, 2017 at 10:26

answered Nov 8, 2017 at 9:57

user3930728

113 bronze badges

Collectives™ on Stack Overflow

Curl vs requests python 3

3 Answers 3

Comments

1 Comment

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Related