How to scrape text from websites using Python

Question

I wrote a code in python using 'requests' and 'beautifulSoup' api to scrape text data from first 100 sites return by google. Well it works good on most of sites but it is giving errors on those which are responding later or not responding at all I am getting this error

raise MaxRetryError(_pool, url, error or ResponseError(cause)) requests.packages.urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='www.lfpress.com', port=80): Max retries exceeded with url: /2015/11/06/fair-with-a-flare-samosas-made-easy (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 11001] getaddrinfo failed',))

Am I supposed to change code written inside requests API? Or I need to use some proxy? How can I leave that site and move on to next one? As error is stopping my execution.

try:.. except: pass ?

Sid Shukla
– Sid Shukla

2016-01-02 22:13:13 +00:00
Commented Jan 2, 2016 at 22:13 — Sid Shukla
– Sid Shukla, Commented Jan 2, 2016 at 22:13

jayme · Accepted Answer · 2016-01-02 22:26:20Z

2

Add a "try except" block around your call to catch that exception and continue if you don't care about the error like:

import requests
try:
    requests.get('http://stackoverflow.com/')
except requests.packages.urllib3.exceptions.MaxRetryError as e:
    print repr(e)

edited Jan 2, 2016 at 22:26

answered Jan 2, 2016 at 22:17

jayme

1,32612 silver badges26 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Muhammad Zeeshan Over a year ago

Well thanks, How can I avoid all exceptions present in requests.packages.urllib3.exceptions? Not just MaxRetryError?

Malik Brahimi Over a year ago

@MuhammadZeeshan That's called passive error handling. Use except alone without specifying.

Untitled123 Over a year ago

To expand ^, you can write except Exception as e: smth smth e

Collectives™ on Stack Overflow

How to scrape text from websites using Python

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related