How to authenticate a site with Python using urllib2?

Question

After much reading here on Stackoverflow as well as the web I'm still struggling with getting things to work.

My challenge: to get access to a restricted part of a website for which I'm a member using Python and urllib2.

From what I've read the code should be like this:

mgr = urllib2.HTTPPasswordMgrWithDefaultRealm()

url = 'http://www.domain.com'

mgr.add_password(None, url, 'username', 'password')
handler = urllib2.HTTPBasicAuthHandler(mgr)
opener = urllib2.build_opener(handler)

urllib2.install_opener(opener)

try:
    response = urllib2.urlopen('http://www.domain.com/restrictedpage')
    page = response.read()
    print page.geturl()
except IOError, e:
    print e

The print doesn't print "http://www.domain.com/restrictedpage", but shows "http://www.domain.com/login" so my credentials aren't stored/processed and I'm being redirected.

How can I get this to work? I've been trying for days and keep hitting the same dead ends. I've tried all the examples I could find to no avail.

My main question is: what's needed to authenticate to a website using Python and urllib2? Quick question: what am I doing wrong?

does the site you basic Http Authentication?? many sites do not, you will be required to find out what variables are posted to what URL, and do everything manually. — dm03514
– dm03514, Commented Mar 3, 2012 at 19:55
How can I find out? I know about the 401 headers, but I can't seem to capture them. Do you have an example on how to do it manually? — Roland
– Roland, Commented Mar 3, 2012 at 20:03

samuel5 · Accepted Answer · 2013-12-26 18:01:15Z

Check first manually what is really happening when you are successfully authenticated (instructions with Chrome):

Open develper tools in Chrome (Ctrl + Shift + I)
Click Network tab
Go and do the authentication manually (go the the page, type user + passwd + submit)
check the POST method in the Network tab of the developer tools
check the Request Headers, Query String Parameters and Form Data. There you find all the information needed what you need to have in your own POST.

Then install "Advanced Rest Client (ARC)" Chrome extension

Use the ARC to construct a valid POST for authentication.

Now you know what to have in your headers and form data. Here's a sample code using Requests that worked for me for one particular site:

import requests

USERNAME = 'user' # put correct usename here
PASSWORD = 'password' # put correct password here

LOGINURL = 'https://login.example.com/'
DATAURL = 'https://data.example.com/secure_data.html'

session = requests.session()

req_headers = {
    'Content-Type': 'application/x-www-form-urlencoded'
}

formdata = {
    'UserName': USERNAME,
    'Password': PASSWORD,
    'LoginButton' : 'Login'
}

# Authenticate
r = session.post(LOGINURL, data=formdata, headers=req_headers, allow_redirects=False)
print r.headers
print r.status_code
print r.text

# Read data
r2 = session.get(DATAURL)
print "___________DATA____________"
print r2.headers
print r2.status_code
print r2.text

Yugal Jindle · Accepted Answer · 2012-03-04 06:46:13Z

0

For HTTP Basic Auth you can refer this : http://www.voidspace.org.uk/python/articles/authentication.shtml

answered Mar 4, 2012 at 6:46

Yugal Jindle

46k43 gold badges136 silver badges201 bronze badges

1 Comment

Roland Over a year ago

Thanks for the link, but unfortunately that didn't help. I'm now looking at Requests as it's a lib aimed at making these things more easy.

Collectives™ on Stack Overflow

How to authenticate a site with Python using urllib2?

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related