Python: urllib.error.HTTPError: HTTP Error 525: Origin SSL Handshake Error

Question

I am using Python 3 to crawl many web pages on one website with urllib.request.build_opener. Each web_page_url is opened like below:

_masterOpener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(CookieJar()))
_masterOpener.addheaders = [('Cookie', some_cookie)]
request = _masterOpener.open(web_page_url)
content = request.read()

It always works smoothly when crawling the first hundreds of pages for about 10 minutes (I tried a few times), and then an error like below occurs:

File "D:\ProgramData\Anaconda3\lib\urllib\request.py", line 532, in open
response = meth(req, response)
File "D:\ProgramData\Anaconda3\lib\urllib\request.py", line 642, in http_response
'http', request, response, code, msg, hdrs)
File "D:\ProgramData\Anaconda3\lib\urllib\request.py", line 570, in error
return self._call_chain(*args)
File "D:\ProgramData\Anaconda3\lib\urllib\request.py", line 504, in _call_chain
result = func(*args)
File "D:\ProgramData\Anaconda3\lib\urllib\request.py", line 650, in http_error_default
raise HTTPError(req.full_url, code, msg, hdrs, fp)
urllib.error.HTTPError: HTTP Error 525: Origin SSL Handshake Error

I searched on the Web and failed to find a solution. How to solve the problem 'urllib.error.HTTPError: HTTP Error 525: Origin SSL Handshake Error' as described?

plaes · Accepted Answer · 2018-02-11 11:25:52Z

2

HTTP Status 5xx errors indicate an error in the server, and it is your responsibility to handle them gracefully (for example, not to crash your crawler).

In this case, the error 525 issue seems to be CloudFlare-specific, where connection to original site via CloudFlare has timed out.

So just add the try...except clause to handle this error gracefully:

try:
    _masterOpener = urllib.request.build_opener(urllib.request.HTTPCookieProcessor(CookieJar()))
    _masterOpener.addheaders = [('Cookie', some_cookie)]
    request = _masterOpener.open(web_page_url)
    content = request.read()
except urllib.error.HTTPError as e:
    # Possible issue with CloudFlare, just fall through
    if e.code == 525:
        # TODO: Log warning about broken url
        pass
    # TODO: ... handle all the other 5xx errors
    # Raise the original exception
    raise

edited Feb 11, 2018 at 11:25

answered Feb 11, 2018 at 11:20

plaes

32.9k11 gold badges95 silver badges89 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Python: urllib.error.HTTPError: HTTP Error 525: Origin SSL Handshake Error

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related