cURL queries work, but python-request fails

Question

Summary

We are using a custom CA in our enviroment. I have imported the CA certificates in the trust store by downloading the Base64 certs and updating the trust store using update-ca-certificates.

I am able to run cURL queries to my REST API, however the request library fails with an SSL error when running the same.

What I have tried

I have tried specifying the root ca certificate file path to the library, but got the same error. How do I troubleshoot this issue? Setting verify to false is not an option.

Code Run

Works

curl -X GET https://api.me.com/admin/ -H 'Authorization: Token 4ae5'

Doesn't work

requests.get('https://api.me.com/admin/', headers={'Authorization': 'Token 4ae5'}, verify='/etc/ssl/certs/root.pem')

Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 600, in urlopen
    chunked=chunked)
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 345, in _make_request
    self._validate_conn(conn)
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 844, in _validate_conn
    conn.connect()
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connection.py", line 326, in connect
    ssl_context=context)
  File "/usr/local/lib/python3.5/dist-packages/urllib3/util/ssl_.py", line 325, in ssl_wrap_socket
    return context.wrap_socket(sock, server_hostname=server_hostname)
  File "/usr/lib/python3.5/ssl.py", line 377, in wrap_socket
    _context=self)
  File "/usr/lib/python3.5/ssl.py", line 752, in __init__
    self.do_handshake()
  File "/usr/lib/python3.5/ssl.py", line 988, in do_handshake
    self._sslobj.do_handshake()
  File "/usr/lib/python3.5/ssl.py", line 633, in do_handshake
    self._sslobj.do_handshake()
ssl.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 376, in send
    timeout=timeout
  File "/usr/local/lib/python3.5/dist-packages/urllib3/connectionpool.py", line 630, in urlopen
    raise SSLError(e)
requests.packages.urllib3.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/lib/python3/dist-packages/requests/api.py", line 67, in get
    return request('get', url, params=params, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/api.py", line 53, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 480, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/lib/python3/dist-packages/requests/sessions.py", line 588, in send
    r = adapter.send(request, **kwargs)
  File "/usr/lib/python3/dist-packages/requests/adapters.py", line 447, in send
    raise SSLError(e, request=request)
requests.exceptions.SSLError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:645)

requests.get(..., verify='/etc/ssl/certs/root.pem') - are you sure you are using the correct path? Given your description of using update-ca-certificates I guess you are using Debian or similar (i.e. Ubuntu). In this case the relevant path should be /etc/ssl/certs/ca-certificates.crt. — Steffen Ullrich
– Steffen Ullrich, Commented Feb 5, 2019 at 11:57
God, this is embarrassing. You are absolutely right, I had to use the bundled certificate. I kept on trying the absolute paths i specified in the /etc/ca-certificates.conf and kept getting errors! :D Please post it as an answer, I will accept it. — NerdSec
– NerdSec, Commented Feb 5, 2019 at 12:24

Steffen Ullrich · Accepted Answer · 2019-02-05 12:35:02Z

3

... requests.get(..., verify='/etc/ssl/certs/root.pem')

Given your description of using update-ca-certificates it looks like you are using Debian or similar (i.e. Ubuntu). In this case the relevant path should be /etc/ssl/certs/ca-certificates.crt.

answered Feb 5, 2019 at 12:35

Steffen Ullrich

125k11 gold badges155 silver badges194 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Martin Hepp · Accepted Answer · 2021-02-09 01:04:29Z

1

The reason is that Python Requests uses certificates from the python-certifi package., not those of the underlying operating system.

certifi includes all CA certificates from Mozilla, and it is also relatively straightforward to add missing certificates to it.

See this post for details on adding certificates to certifi: https://stackoverflow.com/a/66111417/516699

The good thing is also that you do not tweak the SSL of the underlying system, just that of the Python environment your are working in.

answered Feb 9, 2021 at 1:04

Martin Hepp

1,59515 silver badges21 bronze badges

Comments

Alez · Accepted Answer · 2024-05-29 14:33:16Z

0

To add on, for the python requests to work, you may need to create the certificate chain file:

cat your_cert.crt root_cert.crt > chain.pem

You can then directly pass this in your requests call:

chain_cert_path = 'chain.pem'
response = requests.get(url, verify=chain_cert_path)

edited May 29, 2024 at 14:33

Alez

2,87710 gold badges29 silver badges38 bronze badges

answered May 29, 2024 at 14:12

StillAyo

416 bronze badges

Collectives™ on Stack Overflow

cURL queries work, but python-request fails

Summary

What I have tried

Code Run

Works

Doesn't work

3 Answers 3

Comments

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

Summary

What I have tried

Code Run

Works

Doesn't work

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related