Python requests: how to determine response code on MaxRetries exception

Question

I would like to know the cause of the issue when my get or post have error responses until maximum retries are exceeded.

I have an example test below. This throws a MaxRetriesError as expected. However, the original HTTPError seems to be lost.

from requests import Session, exceptions
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
from unittest import TestCase
import logging


class TestRequests(TestCase):

    def test_retries(self):
        session = Session()
        retries = Retry(total=5,
                        backoff_factor=0.1,
                        status_forcelist=[500, 502, 503, 504],
                        method_whitelist=frozenset(['GET', 'POST']))

        session.mount('https://', HTTPAdapter(max_retries=retries))
        session.mount('http://', HTTPAdapter(max_retries=retries))

        try:
            result = session.get('https://httpbin.org/status/500',
                                 stream=True,
                                 timeout=2)
            print(result)
        except exceptions.HTTPError as e:
            logging.error('http', exc_info=e)
        except exceptions.RetryError as e:
            logging.error('retry', exc_info=e)

The Traceback shows the following information, it only goes back as far as the last RetryError and not to the HTTPError. The exception's string does mention the response code but parsing error message strings is not a robust approach.

ERROR:root:retry
Traceback (most recent call last):
  File "/home/giles/venv/gphotos-sync3/lib/python3.6/site-packages/requests/adapters.py", line 449, in send
    timeout=timeout
  File "/home/giles/venv/gphotos-sync3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 731, in urlopen
    body_pos=body_pos, **response_kw)
  File "/home/giles/venv/gphotos-sync3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 731, in urlopen
    body_pos=body_pos, **response_kw)
  File "/home/giles/venv/gphotos-sync3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 731, in urlopen
    body_pos=body_pos, **response_kw)
  [Previous line repeated 2 more times]
  File "/home/giles/venv/gphotos-sync3/lib/python3.6/site-packages/urllib3/connectionpool.py", line 711, in urlopen
    retries = retries.increment(method, url, response=response, _pool=self)
  File "/home/giles/venv/gphotos-sync3/lib/python3.6/site-packages/urllib3/util/retry.py", line 398, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='httpbin.org', port=443): Max retries exceeded with url: /status/500 (Caused by ResponseError('too many 500 error responses',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/giles/github/gphotos-sync/test/test_requests.py", line 36, in test_retries
    timeout=2)
  File "/home/giles/venv/gphotos-sync3/lib/python3.6/site-packages/requests/sessions.py", line 546, in get
    return self.request('GET', url, **kwargs)
  File "/home/giles/venv/gphotos-sync3/lib/python3.6/site-packages/requests/sessions.py", line 533, in request
    resp = self.send(prep, **send_kwargs)
  File "/home/giles/venv/gphotos-sync3/lib/python3.6/site-packages/requests/sessions.py", line 646, in send
    r = adapter.send(request, **kwargs)
  File "/home/giles/venv/gphotos-sync3/lib/python3.6/site-packages/requests/adapters.py", line 507, in send
    raise RetryError(e, request=request)
requests.exceptions.RetryError: HTTPSConnectionPool(host='httpbin.org', port=443): Max retries exceeded with url: /status/500 (Caused by ResponseError('too many 500 error responses',))


Ran 1 test in 4.092s

OK

Hi, using http://httpbin.org/status/500 gets the same exceptions. (also I need to use https!) — Giles Knap
– Giles Knap, Commented Jan 25, 2019 at 12:18
Hey can you make it a bit clearer on what your question is? Are u asking the reason why you're not getting the HTTPError? — 0xInfection
– 0xInfection, Commented Jan 25, 2019 at 13:12
Yes, I want to see the HttpError. I have a possible answer, see below. — Giles Knap
– Giles Knap, Commented Jan 25, 2019 at 13:17
Try setting the retries to 0. It should throw out the exception. — 0xInfection
– 0xInfection, Commented Jan 25, 2019 at 13:18
I just tried that and it still throws MaxRetries! The solution using raise_on_status=False works for me. Thanks for the suggestions. — Giles Knap
– Giles Knap, Commented Jan 25, 2019 at 13:29

Giles Knap · Accepted Answer · 2019-01-25 13:20:59Z

I have found that I can see the original error if I tell Retries not to raise its own error (raise_on_status=False), but then inspect the last response instead. See modified code below.

from requests import Session, exceptions
from requests.adapters import HTTPAdapter
from urllib3.util.retry import Retry
from urllib3.exceptions import MaxRetryError
from unittest import TestCase
import logging


class TestRequests(TestCase):

    def test_retries(self):
        session = Session()
        retries = Retry(total=5,
                        backoff_factor=0.1,
                        status_forcelist=[500, 502, 503, 504],
                        method_whitelist=frozenset(['GET', 'POST']),
                        raise_on_status=False)

        session.mount('https://', HTTPAdapter(max_retries=retries))
        session.mount('http://', HTTPAdapter(max_retries=retries))

        try:
            result = session.get('https://httpbin.org/status/500',
                                 stream=True,
                                 timeout=2)
            print(result)
            result.raise_for_status()

        except exceptions.HTTPError as e:
            logging.error('http', exc_info=e)
        except MaxRetryError as e:
            logging.error('retry', exc_info=e)

This results in:

------------------------------------------------------------- Captured stdout call --------------------------------------------------------------
<Response [500]>
--------------------------------------------------------------- Captured log call ---------------------------------------------------------------
test-requests.py            30 ERROR    http
Traceback (most recent call last):
  File "/home/hgv27681/github/tests/test-requests.py", line 27, in test_retries
    result.raise_for_status()
  File "/home/hgv27681/github/venv3/lib/python3.6/site-packages/requests/models.py", line 940, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 500 Server Error: INTERNAL SERVER ERROR for url: https://httpbin.org/status/500

One thing to note with this approach is it disguises the retry errors so you don't know if your gets are retrying, only if they succeed or reach max-retries.

Collectives™ on Stack Overflow

Python requests: how to determine response code on MaxRetries exception

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related