2

I am trying to post form data to a URL. I am not getting the expected response and curious about some information I am getting from requests module (2.6.2). The following is the post method:

>>> response = requests.post(url, data={'uname':user, 'pwd':password,'phrase':'','submit':True})

As you can see I am using the post() method, so I expected the method to be POST. The keys for the data object match the names of the form elements. The URL is the form action.

>>> vars(response.request)
{'method': 'GET', 'body': None, '_cookies': <<class 'requests.cookies.RequestsCookieJar'>[]>, 'headers': {'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Content-Type': 'application/x-www-form-urlencoded', 'User-Agent': 'python-requests/2.6.2 CPython/3.3.6 Windows/8', 'Connection': 'keep-alive'}, 'hooks': {'response': []}, 'url': url}

The response.request property should contain information about the request that was sent for this response. It's method property is GET, I expected POST. The URL looks correct. The page is expected to return something else if the form posts though, strange, I'll check the request headers.

>>> response.request.headers
{'Accept-Encoding': 'gzip, deflate', 'Accept': '*/*', 'Content-Type': 'application/x-www-form-urlencoded', 'User-Agent': 'python-requests/2.6.2 CPython/3.3.6 Windows/8', 'Connection': 'keep-alive'}
>>>

Those look OK except, wait a minute! Where is my form data? Why is it not being sent with the request? I check the history and find I am being redirected. This time I add allow_redirects=False to my post() call. Then check the response.request object and its headers.

>>> vars(response.request)
{'method': 'POST', 'body': 'phrase=&pwd=****&uname=****&submit=True', '_cookies': <<class 'requests.cookies.RequestsCookieJar'>[]>, 'headers': {'Accept-Encoding': 'gzip, deflate', 'Content-Type': 'application/x-www-form-urlencoded', 'User-Agent': 'python-requests/2.6.2 CPython/3.3.6 Windows/8', 'Content-Length': '56', 'Accept': '*/*', 'Connection': 'keep-alive'}, 'hooks': {'response': []}, 'url': 'http://myurl.com/path/to/script.php'}

This time it's a POST so that seems correct. I feel like I'm on the right track. That's strange, the body property appears like a query string, not like I'd expect for a form post.

>>> response.request.headers
{'Accept-Encoding': 'gzip, deflate', 'Content-Type': 'application/x-www-form-urlencoded', 'User-Agent': 'python-requests/2.6.2 CPython/3.3.6 Windows/8', 'Content-Length': '56', 'Accept': '*/*', 'Connection': 'keep-alive'}

Those headers, again no form data in the header. What's this? 'Content-Type': 'application/x-www-form-urlencoded'? Is that why my form data looks like a query string? What would a normal content-type be? It's the same type as reported by Chrome hitting the same URL so I doubt that's the issue.

If none of this looks off, they may be smart and rejecting a post from a non-local origin right? My main concern is that the form data is a string in the body property, that seems wrong. If not, can I get clever and set the HTTP header origin?

Improve this question
1
  • I tried spoofing the headers with .post(headers={'origin':...,'host':..., ...}) and it made no difference. Commented Apr 24, 2015 at 5:46

1 Answer 1

Reset to default
2 
'application/x-www-form-urlencoded'

This is the default content-type when posting your data.

And yes it's ok if your POST data appears to look like a query string, this is how it's sent with x-www-form-urlencoded. From the previous link:

The control names/values are listed in the order they appear in the document. The name is separated from the value by = and name/value pairs are separated from each other by &.

Next:

Those headers, again no form data in the header

With a POST, the form data is not sent in the header. It is sent in the body of the request. Try

>>> response.request.body
'uname=x&pwd=y&submit=True&phrase='

to view this data

Improve this answer
Sign up to request clarification or add additional context in comments.

8 Comments

So seems like you're saying everything looks correct. Perhaps my issue is that the result is saving what I am looking for in session cookies. I have tried to preface the post() with session = requests.session() in order to capture these. Would you advise updating the question with to determine how to retrieve these session cookies?
The server will respond with HTTP Header Set-Cookie when it wants the client to save a cookie. You can capture this cookie data by using response.headers['set-cookie']
Well, there must be something going on elsewhere because the response I get from this is a 302 redirect. If the credentials were accepted it would be a different page, but it's returning me to the form URL. I expect the set-cookie header and am not getting it. I'm also not getting the expected message about incorrect credentials, so it must not be accepting my post data. I thank you for your answer and consider it to have addressed my question, even though I am still stuck with an unknown issue.
If you are being 302 redirected to the improper page, I would suggest visiting that page with Google Chrome, and open up the network tools page (ctrl+shift+j and then click the "Network" tab). Here you will see the exact POST parameters sent by the browser, and you should be able to mimic this request with Python no problem.
You're correct, I am being redirected with a GET request. My issue is that in that redirect on Chrome there is a set-cookie header to store the uname and a mutation of the pwd values in the session cookies. As far as I can tell, I am mimicking the request as described. I was having trouble getting those session cookies, but I'm also not being redirected where I would expect if credentials were validated. Obviously, it's me doing something wrong.
|

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.