How to click submit button in a form?

Question

I am trying to scrap this website for fund price history, by providing the start date, end date and click the 'Get Prices' button via POST method.

However the page requests.post() return does not contain the results, as if the "Get Price" button was never pressed. This is the URL I put together through the code:

https://www.nysaves.org/nytpl/fundperform/fundHistory.do?submit=Get+Prices&endDate=02%2F20%2F2016&fundid=1003022&startDate=01%2F01%2F2016

I read other posts on Stackoverflow about submitting form via POST in Python and I couldn't make it work. Could you please help? Many thanks!

import requests
import datetime

startDate = datetime.datetime(2016,1,1).strftime('%m/%d/%Y')
endDate = datetime.datetime(2016,2,20).strftime('%m/%d/%Y')

serviceurl = 'https://www.nysaves.org/nytpl/fundperform/fundHistory.do?'
payload = {'fundid':1003022, 'startDate':startDate, 'endDate': endDate, 'submit':'Get Prices'}
r = requests.post(serviceurl, params=payload)

#from IPython.core.display import HTML
#HTML(r.content.decode('utf-8'))

Quentin Pradet · Accepted Answer · 2018-10-10 04:07:21Z

4

You need to use "data", not "params". "params" is used for GET parameters encoded into the URL, but "data" is sent in the body.
The correct url is https://www.nysaves.org/nytpl/fundperform/fundHistorySearch.do, not https://www.nysaves.org/nytpl/fundperform/fundHistory.do?.
You don't need the submit keyword. But adding it won't hurt.

This code will work:

startDate = datetime.datetime(2016,1,1).strftime('%m/%d/%Y')
endDate = datetime.datetime(2016,2,20).strftime('%m/%d/%Y')

serviceurl = 'https://www.nysaves.org/nytpl/fundperform/fundHistorySearch.do'
payload = {'fundid': 1003022, 'startDate': startDate, 'endDate': endDate}
r = requests.post(serviceurl, data=payload)
print(r.text)

edited Oct 10, 2018 at 4:07

answered Feb 23, 2016 at 5:34

Quentin Pradet

4,7712 gold badges32 silver badges42 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

How to click submit button in a form?

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related