Selenium Python - Get Network response body

Question

I use Selenium to react to the reception of data following a GET request from a website. The API called by the website is not public, so if I use the URL of the request to retrieve the data, I get {"message":"Unauthenticated."}.

All I've managed to do so far is to retrieve the header of the response.

I found here that using driver.execute_cdp_cmd('Network.getResponseBody', {...}) might be a solution to my problem.

Here is a sample of my code:

import json
from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import DesiredCapabilities

capabilities = DesiredCapabilities.CHROME
capabilities["goog:loggingPrefs"] = {"performance": "ALL"}
driver = webdriver.Chrome(
    r"./chromedriver",
    desired_capabilities=capabilities,
)

def processLog(log):
    log = json.loads(log["message"])["message"]
    if ("Network.response" in log["method"] and "params" in log.keys()):
        headers = log["params"]["response"]
        body = driver.execute_cdp_cmd('Network.getResponseBody', {'requestId': log["params"]["requestId"]})
        print(json.dumps(body, indent=4, sort_keys=True))
        return log["params"]
        

logs = driver.get_log('performance')
responses = [processLog(log) for log in logs]

Unfortunately, the driver.execute_cdp_cmd('Network.getResponseBody', {...}) returns:

unknown error: unhandled inspector error: {"code":-32000,"message":"No resource with given identifier found"}

Do you know what I am missing?

Do you have any idea on how to retrieve the response body?

Thank you for your help!

Seglinglin · Accepted Answer · 2020-12-23 12:47:20Z

12

In order to retrieve response body, you have to listen specifically to Network.responseReceived:

def processLog(log):
    log = json.loads(log["message"])["message"]
    if ("Network.responseReceived" in log["method"] and "params" in log.keys()):
        body = driver.execute_cdp_cmd('Network.getResponseBody', {'requestId': log["params"]["requestId"]})

However, I ended using a different approach relying on requests. I just retrieved the authorization token from the browser console (Network > Headers > Request Headers > Authorization) and used it to get the data I wanted:

import requests

def get_data():
    url = "<your_url>"
    headers = {
        "Authorization": "Bearer <your_access_token>",
        "Content-type": "application/json"
    }
    params = {
        key: value,
        ...
    }

    r = requests.get(url, headers = headers, params = params)

    if r.status_code == 200:
        return r.json()

answered Dec 23, 2020 at 12:47

Seglinglin

5011 gold badge4 silver badges19 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

fpghost Over a year ago

Hmm, I have verified the method is responseReceived but upon getResponseBody I still get exactly your original error

Boris Over a year ago

you are my savior!!!!!!

Recently_Created_User · Accepted Answer · 2022-05-09 06:30:14Z

10

Probably some responses don't have a body, thus selenium throws an error that "no resource" for given identifier was found. Error message is a bit ambiguous here.

Try doing like this:

from selenium.common import exceptions

try:
    body = chromedriver.execute_cdp_cmd('Network.getResponseBody', {'requestId': log["params"]["requestId"]})
    log['body'] = body
except exceptions.WebDriverException:
    print('response.body is null')

This way responses without body will not crash your script.

edited May 9, 2022 at 6:30

answered Apr 19, 2022 at 9:10

Recently_Created_User

1713 silver badges14 bronze badges

3 Comments

Luke Over a year ago

Don't forget to import the exception: from selenium.common import exceptions or from selenium.common.exceptions import WebDriverException

Recently_Created_User Over a year ago

@Luke, thanks, forgot to add import statement for the exception, edited the post.

Piotr Kula Jan 8 at 21:12

Even though EncodedDataLength is > 0 .. i still sometimes get exceptions and the whole thing falls over :(

Collectives™ on Stack Overflow

Selenium Python - Get Network response body

2 Answers 2

2 Comments

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related