0

From this question, the last responder seems to think that it is possible to use python to open a webpage, let me sign in manually, go through a bunch of menus then let the python parse the page when I get where I want. The website has a weird sign in procedure so using requests and passing a user name and password will not be sufficient.

However it seems from this question that it's not a possibility.

SO the question is, is it possible? if so, do you know of some example code out there?

6
  • Who said you need selenium? github.com/jmcarp/robobrowser Commented Mar 12, 2017 at 4:22
  • What's so weird about the login procedure that requests can't handle? I've seen some doozies and, so long as you trap the conversation between the client and server, requests can do anything. Commented Mar 12, 2017 at 4:30
  • @cricket_007 Thanks, I'll try this. Commented Mar 12, 2017 at 4:38
  • @cricket_007 No one really told me, just from looking around it seemed like I kept running into questions and articles using selenium. I just assumed that was what most people used. Commented Mar 12, 2017 at 4:52
  • @Carlos, I've not had any luck due to due the use of a external token to get a code to log in with. Maybe I can figure out a way for it to stop, and let me enter the code then let it do its thing afterwards. You're probably right, I can probably explore the requests option a bit more. Commented Mar 12, 2017 at 4:58

1 Answer 1

1

The way to approach this problem is when you login normally have the developer tools next to you and see what the request is sending.

When logging in to bandcamp the XHR request that's being sent is the following:

Bandcamp

From that response you can see that an identity cookie is being sent. That's probably how they identify that you are logged in. So when you've got that cookie set you would be authorized to view logged in pages.

So in your program you could login normally using requests, save the cookie in a variable and then apply the cookie to further requests using requests.

Of course login procedures and how this authorization mechanism works may differ, but that's the general gist of it.

So when do you actually need selenium? You need it if a lot of the things are being rendered by javascript. requests is only able to get the html. So if the menus and such is rendered with javascript you won't ever be able to see that information using requests.

Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.