I am learning about how to login to websites with python and requests and went through a number of different posts on stackoverflow and videos on youtube on what is required and how to do it.
I found that the below information is being sent through the form when I clicked submit on the browser. I went to the developer tools under network and took a look at the headers
What I can tell
What I can tell is that on the login page itself, they only ask for username and password of which the below code is the html extract.
<div style="margin:0;padding:0;display:inline"><input name="utf8" type="hidden" value="✓"><input name="authenticity_token" type="hidden" value="T8NxfsxCHqUPzdvmM++VIpipimDyjsLHkg4Oz3Yuouk="></div>
<ul class="sic_loginFailed">
<li>
<label for="sic_login_header_username">Username</label>
<input id="sic_login_header_username" name="name" type="text" class="sic_formText">
</li>
<li>
<label for="sic_login_header_password">Password</label>
<input id="sic_login_header_password" name="password" type="password" class="sic_formText">
</li>
<li class="sic_remember">
<input id="sic_login_header_remember" name="remember" type="checkbox">
<label for="sic_login_header_remember">Remember my login.</label>
</li>
<li>
<input type="hidden" name="redirect"
value="https://www.shareinvestor.com/sg">
<input id="sic_login_submit" type="submit" value="Sign In" class="sic_greenInputButton">
</li>
</ul>
So that means that authenticity token and password_m is auto generated by the website? Note: I have a feeling password_m is auto assigned to me when I created my account. But the token is auto generated at every login.
My question
I wrote the below code based on what I know and what I have researched, but I am still unable to log into the website.
url = "https://www.shareinvestor.com/user/login.html" # This is the main URL login page
login_data = {'name': 'test_user',
'password': 'test_password',
'password_m': '5d93ceb70e2bf5daa84ec3d0cd2c731a',
'remember': True,
'redirect': 'https://www.shareinvestor.com/sg'}
with requests.Session() as s:
a = s.get(url).text
b = bs4.BeautifulSoup(a, 'lxml')
c = b.findAll('input', type='hidden') # This is to draw out the token. I tried searching for it in the cookies previously, but failed badly....
for i in c:
login_data[i['name']] = i['value']
# I use the this url for the response because as per the `Headers` in the picture above, it says that this is the request URL that the form is submitting to.
response = requests.post('https://www.shareinvestor.com/user/do_login.html?use_https=1', data=login_data)
response = requests.get('https://www.shareinvestor.com/user/edit_profile.html', cookies=response.cookies)
print(response.text)
If you have read this far, I would really appreciate if you could shed some light on what I am doing right or wrong in trying to login to the website, and persist the login.

s.postands.getinstead of last two requestss.post / s.getinstead of requests. This solves my issue. I was so close...