1

I am learning about how to login to websites with python and requests and went through a number of different posts on stackoverflow and videos on youtube on what is required and how to do it.

I found that the below information is being sent through the form when I clicked submit on the browser. I went to the developer tools under network and took a look at the headers

form response when submitting on website

What I can tell

What I can tell is that on the login page itself, they only ask for username and password of which the below code is the html extract.

<div style="margin:0;padding:0;display:inline"><input name="utf8" type="hidden" value="✓"><input name="authenticity_token" type="hidden" value="T8NxfsxCHqUPzdvmM++VIpipimDyjsLHkg4Oz3Yuouk="></div>
<ul class="sic_loginFailed">
    <li>
      <label for="sic_login_header_username">Username</label>
      <input id="sic_login_header_username" name="name" type="text" class="sic_formText">
    </li>
    <li>
      <label for="sic_login_header_password">Password</label>
      <input id="sic_login_header_password" name="password" type="password" class="sic_formText">
    </li>
    <li class="sic_remember">
      <input id="sic_login_header_remember" name="remember" type="checkbox">
      <label for="sic_login_header_remember">Remember my login.</label>
    </li>
    <li>
        <input type="hidden" name="redirect" 
value="https://www.shareinvestor.com/sg">



      <input id="sic_login_submit" type="submit" value="Sign In" class="sic_greenInputButton">
    </li>
  </ul>

So that means that authenticity token and password_m is auto generated by the website? Note: I have a feeling password_m is auto assigned to me when I created my account. But the token is auto generated at every login.

My question

I wrote the below code based on what I know and what I have researched, but I am still unable to log into the website.

url = "https://www.shareinvestor.com/user/login.html"  # This is the main URL login page

login_data = {'name': 'test_user',
              'password': 'test_password',
              'password_m': '5d93ceb70e2bf5daa84ec3d0cd2c731a',
              'remember': True,
              'redirect': 'https://www.shareinvestor.com/sg'}

with requests.Session() as s:
    a = s.get(url).text
    b = bs4.BeautifulSoup(a, 'lxml')
    c = b.findAll('input', type='hidden')  # This is to draw out the token. I tried searching for it in the cookies previously, but failed badly.... 
    for i in c:
        login_data[i['name']] = i['value']

    # I use the this url for the response because as per the `Headers` in the picture above, it says that this is the request URL that the form is submitting to. 
    response = requests.post('https://www.shareinvestor.com/user/do_login.html?use_https=1', data=login_data)
    response = requests.get('https://www.shareinvestor.com/user/edit_profile.html', cookies=response.cookies)

    print(response.text)

If you have read this far, I would really appreciate if you could shed some light on what I am doing right or wrong in trying to login to the website, and persist the login.

2
  • 1
    Use the session while making requests. s.post and s.get instead of last two requests Commented Jun 22, 2017 at 6:17
  • Thanks @AndrewCherevatkin. You are right. I should be using s.post / s.get instead of requests. This solves my issue. I was so close... Commented Jun 22, 2017 at 14:58

1 Answer 1

1

As per what Andrew Cherevatkin has mentioned, I should be using s.post and s.get instead of requests. By using requests.post and requests.get I lose everything that I collected through thesession()

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.