2

I am trying to web-scrape data from https://www.mcmaster.com. They have provided me with a .pfx file and a passphrase. When making a GET request on Postman using their .json file, I input my website login/password and upload the .pfx certificate with its passphrase and everything works fine. Now I am trying to do this same thing but in Python, but am a bit unsure.

Here is my current Python code, I am unsure where I would put the website email/password login and how to successfully do a GET request.

import requests_pkcs12
from requests_pkcs12 import get

r = get('https://api.mcmaster.com/v1/login', pkcs12_filename='Schallert.pfx', pkcs12_password='mcmasterAPI@1901')

response = requests_pkcs12.get(r)

print(response.text)

Here is how I have it setup in Postman (Website email/pw login) enter image description here

.PFX Certificate page enter image description here

3 Answers 3

4

Postman has a built in feature where it will convert requests into code. You can do it like so:

  1. On the far right click the Code Snippet Button (</>)
  2. Once you are on that page, there is two available python options
  3. Then all you need to do is copy the code into a Python file and add all your customizations (Should be already optimized)

One thing I’ll warn you about though is the URL. Postman doesn’t add http:// or https:// to the URL, which means Python will throw a No Scheme Supplied error.

Available Packages for Auto Conversion:

  1. Requests
  2. http.client

Meaning you will have to use a different package instead of requests_pkcs12

Sign up to request clarification or add additional context in comments.

2 Comments

I did exactly as you said and copied and pasted the code from the "Code Snippet Button", but it gives me an error in Python. Is it because I am missing the .pfx file? import requests url = "https://api.mcmaster.com/v1/login" payload = "{\n \"UserName\":\"[email protected]\",\n \"Password\":\"PASSWORD123\"\n}" headers = { 'Authorization': 'Bearer RVVZbEs284796820187waFZrVGFBZ3E0a3VwQXJESTlnOWI0P=', 'Content-Type': 'text/plain' } response = requests.request("POST", url, headers=headers, data=payload) print(response.text)
With both requests and http.client I am getting a SSL certificate issue/error
1

The package requests_pkcs12 is a wrapper written above the requests package. So, all the parameters that accept requests will accept by the requests_pkcs12.

Here is the source code proof for that. https://github.com/m-click/requests_pkcs12/blob/master/requests_pkcs12.py#L156

Also, from your screenshots, I understood that you are using POST not GET.

import json
from requests_pkcs12 import post

url = "https://api.mcmaster.com/v1/login"
payload = {'Username': 'yourusername',
           'Password': 'yourpassword'}

resp = post(url, pkcs12_filename='Schallert.pfx', pkcs12_password='mcmasterAPI@1901', data=json.dumps(payload))
print(resp)

Footnote: I hope the password mcmasterAPI@1901 it's a fake one. if not please don't share any credentials in the platform.

1 Comment

I now get a successful login code <200> but on Postman, it gives me a Bearer token which I re-input to then retrieve actual data. I don't see the bearer token upon successful login with Python like I do in Pycharm. Postman: gyazo.com/f0f7baec22146d03615849b413f410bb Python: gyazo.com/69f513da4d2af3d427afb2ee18335948 Could you please help?
0

After a quick web search, it looks like you need to create a temporary certificate as a .pem file, which is then passed to the request.

from contextlib import contextmanager
from pathlib import Path
from tempfile import NamedTemporaryFile

import requests
from cryptography.hazmat.primitives.serialization import Encoding, PrivateFormat, NoEncryption
from cryptography.hazmat.primitives.serialization.pkcs12 import load_key_and_certificates


@contextmanager
def pfx_to_pem(pfx_path, pfx_password):
    pfx = Path(pfx_path).read_bytes()
    private_key, main_cert, add_certs = load_key_and_certificates(pfx, pfx_password.encode('utf-8'), None)

    with NamedTemporaryFile(suffix='.pem') as t_pem:
        with open(t_pem.name, 'wb') as pem_file:
            pem_file.write(private_key.private_bytes(Encoding.PEM, PrivateFormat.PKCS8, NoEncryption()))
            pem_file.write(main_cert.public_bytes(Encoding.PEM))
            for ca in add_certs:
                pem_file.write(ca.public_bytes(Encoding.PEM))
        yield t_pem.name

with pfx_to_pem('your pfx file path', 'your passphrase') as cert:
    requests.get(url, cert=cert, data=payload)

5 Comments

Do I just include this code with the python code from Postman?
Where do I use my website login as well?
@Kaevonz pass the auth details in the header
I am a bit confused can you show me?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.