What HTTP response codes are retried by python Requests

Question

What are the list of HTTP response status codes retried by default and how many times, by the Python Requests. How can I change the number of retries? I couldn't find any documentation for it.

I tried below code and there were two retries on 401 status code.

import requests
from http.client import HTTPConnection
from requests.auth import HTTPDigestAuth

HTTPConnection.debuglevel = 1
requests.adapters.DEFAULT_RETRIES = 5

def test():
    data = 'testdata'
    username = 'testuser'
    password = 'test'
    url='https://example.com:443/captionen_0001.vtt'
    try:
        response = requests.put(url, auth=HTTPDigestAuth(username,password), data=data, verify=False)
    except Exception as e:
        print('error'+str(e))

test()

  warnings.warn(
send: b'PUT /channel_captionen_0001.vtt HTTP/1.1\r\nHost: example.com\r\nUser-Agent: python-requests/2.24.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Length: 8\r\n\r\n'
send: b'testdata'
reply: 'HTTP/1.1 401 Authorization Required\r\n'
header: Date: Sat, 05 Feb 2022 07:50:25 GMT
header: WWW-Authenticate: Digest realm="WebDAV", nonce="tE/JnkDX845db3", algorithm=MD5, qop="auth"
  
  warnings.warn(
send: b'PUT /channel_captionen_0001.vtt HTTP/1.1\r\nHost: example.com\r\nUser-Agent: python-requests/2.24.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nContent-Length: 8\r\nAuthorization: Digest username="testuser", realm="WebDAV", nonce="tE/JnkDX845db3", uri="/channel_captionen_0001.vtt", response="1c3299c716797e8f36528f6e6dbaeb50", algorithm="MD5", qop="auth", nc=00000001, cnonce="dd0835ef485c6b71"\r\n\r\n'
send: b'testdata'
reply: 'HTTP/1.1 401 Authorization Required\r\n'
header: Date: Sat, 05 Feb 2022 07:50:25 GMT
header: WWW-Authenticate: Digest realm="WebDAV", nonce="/vXKnkDXBQc098a4", algorithm=MD5, qop="auth

I'm very surprised that requests appears to be retrying a 401. This is more likely to be some oddity in the logging. There would never be any point in retrying a 4xx code expect (possibly) 408. Some of the 5xx codes are worth retrying — jackal
– jackal, Commented Feb 5, 2022 at 8:15
This is not a retry. It is the typical flow of the digest authentication. At first the server answers with a 401 and additional challenge parameters in the headers, then the client sends a response to the challenge. You can see that the second request contains more data. — Klaus D.
– Klaus D., Commented Feb 5, 2022 at 9:49

Corralien · Accepted Answer · 2022-02-05 08:40:04Z

2

It's not obviously to find. You have to know requests is not the package that manage the connection, urllib3 does.

In the source code of HTTPAdapter (use it when you want more control on requests), the docstring on max_retries parameter said:

If you need granular control over the conditions under which we retry a request, import urllib3's Retry class and pass that instead

Now you can refer to the documentation of urllib3 for Retry class.

Read especially status_forcelist parameter and RETRY_AFTER_STATUS_CODES (default: frozenset({413, 429, 503}))

Update

import requests
import urllib3

my_code_list = [401, 403, ...]

s = requests.Session()
r = urllib3.util.Retry(status_forcelist=my_code_list)
a = requests.adapters.HTTPAdapter(max_retries=r)
s.mount('http://', a)

edited Feb 5, 2022 at 8:40

answered Feb 5, 2022 at 8:19

Corralien

121k8 gold badges44 silver badges69 bronze badges

Sign up to request clarification or add additional context in comments.

12 Comments

jackal Over a year ago

Maybe I missed something but the documentation for the Retry class doesn't indicate the circumstances under which it comes into play - i.e., which HTTP status codes might be retried.

Corralien Over a year ago

@OlvinRoght. Can you check the status_forcelist please and RETRY_AFTER_STATUS_CODES

Joe Over a year ago

Worth noting that "By default, this is disabled with None"? RETRY_AFTER_STATUS_CODES looks like it refers to statuses where Retry-After will be respected.

Corralien Over a year ago

Try to pass Retry(status_forcelist=[401]) to HTTPAdapter

Tomalak Over a year ago

@OlvinRoght "why would one ever want to retry a 401" - Retrying 401 is part of the challenge/response authentication scheme in Windows authentication. But that would require that the server sends the proper WWW-Authenticate header.

|

Collectives™ on Stack Overflow

What HTTP response codes are retried by python Requests

1 Answer 1

12 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

12 Comments

Your Answer

Sign up or log in

Post as a guest

Related