URLLib2.URL Error: Reading Server Response Codes (Python)

Question

I have a list of urls. I'd like to see the server response code of each and find out if any are broken. I can read server errors (500) and broken links (404) okay, but the code breaks once a non-website is read (e.g. "notawebsite_broken.com"). I've searched around and not found the answer... I hope you can help.

Here's the code:

import urllib2

#List of URLs. The third URL is not a website
urls = ["http://www.google.com","http://www.ebay.com/broken-link",
"http://notawebsite_broken"]

#Empty list to store the output
response_codes = []

# Run "for" loop: get server response code and save results to response_codes
for url in urls:
    try:
        connection = urllib2.urlopen(url)
        response_codes.append(connection.getcode())
        connection.close()
        print url, ' - ', connection.getcode()
    except urllib2.HTTPError, e:
        response_codes.append(e.getcode())
        print url, ' - ', e.getcode()

print response_codes

This gives the output of...

http://www.google.com  -  200
http://www.ebay.com/broken-link  -  404
Traceback (most recent call last):
  File "test.py", line 12, in <module>
    connection = urllib2.urlopen(url)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 127, in urlopen
    return _opener.open(url, data, timeout)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 404, in open
    response = self._open(req, data)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 422, in _open
    '_open', req)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 382, in _call_chain
    result = func(*args)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1214, in http_open
    return self.do_open(httplib.HTTPConnection, req)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/urllib2.py", line 1184, in do_open
    raise URLError(err)
urllib2.URLError: <urlopen error [Errno 8] nodename nor servname provided, or not known>

Does anyone know a fix for this or can anyone point me in the right direction?

Padraic Cunningham · Accepted Answer · 2014-06-25 23:13:30Z

3

You could use requests:

import requests

urls = ["http://www.google.com","http://www.ebay.com/broken-link",
"http://notawebsite_broken"]

for u in urls:
    try:
        r = requests.get(u)
        print "{} {}".format(u,r.status_code)
    except Exception,e:
        print "{} {}".format(u,e)

http://www.google.com 200
http://www.ebay.com/broken-link 404
http://notawebsite_broken HTTPConnectionPool(host='notawebsite_broken', port=80): Max retries exceeded with url: /

answered Jun 25, 2014 at 23:13

Padraic Cunningham

181k30 gold badges264 silver badges327 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Levon Over a year ago

+1 Very nice. Changing the code to use except Exception as e will make it work in Python 3.x.x too.

Sohcahtoa82 · Accepted Answer · 2014-06-25 23:03:12Z

1

When urllib2.urlopen() fails to connect to the server, or fails to resolve the IP of the host, it raises a URLError instead of HTTPError. You'll need to catch urllib2.URLError in addition to urllib2.HTTPError to deal with those cases.

answered Jun 25, 2014 at 23:03

Sohcahtoa82

6393 silver badges13 bronze badges

Comments

Jonathan Vanasco · Accepted Answer · 2014-06-25 23:18:02Z

1

The API for the urllib2 library is a nightmare.

Many people, myself included, strongly recommend using the requests package:

http://docs.python-requests.org/en/latest/

One of the nicer things about requests is that any request issues inherit from a base Exception class. When you use urllib2 "raw", a number of Exceptions can be raised from urllib2, in addition to the socket module and possibly some others ( i can't remember, but its messy )

tldr -- just use the requests library.

answered Jun 25, 2014 at 23:18

Jonathan Vanasco

15.8k11 gold badges53 silver badges74 bronze badges

1 Comment

Sam Perry Over a year ago

Nice. Requests is so much easier! Thanks.

Collectives™ on Stack Overflow

URLLib2.URL Error: Reading Server Response Codes (Python)

3 Answers 3

1 Comment

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related