BrowserMob Proxy + selenium capturing https requests

Question

I am trying to use BrowserMob Proxy with browsermob-proxy-py on Python. I need to capture URLs of all requests from the page. But I can't find https requests in the HAR file. Selenium and BrowserMob Proxy are running on remote machine.

Example code:

from selenium import webdriver
import browsermobproxy

SELENIUM_EXECUTOR = 'http://<remote_ip>:4444/wd/hub'
SELENIUM_DESIRED_CAPABILITIES = {
        'browserName': 'firefox',
        'version': '20.0.0',
        'javascriptEnabled': True,
}

prox = browsermobproxy.Client('<remote_ip>:8080')
driver = webdriver.Remote(
    command_executor=SELENIUM_EXECUTOR,
    desired_capabilities=SELENIUM_DESIRED_CAPABILITIES,
    proxy=prox)

url_to_get = 'http://google.ru'
prox.new_har()
driver.get(url_to_get)
for ent in prox.har['log']['entries']:
    print ent['request']['url']

driver.quit()
prox.close()

This example returns 5 http requests.

But if I change url_to_get to 'https://...', I will see only 3 http requests, and none https requests

Does anyone know how to catch https headers too?

Aaron B. · Accepted Answer · 2014-01-06 17:10:54Z

2

I can't speak to the python piece, but in our java/junit selenium tests, I had to specify both an http and and https proxy for selenium.

I believe the newest (2.39) version of selenium, they've deprecated specifying an https proxy, instead the parameter specifies an ssl proxy.

Our (java) code looks something like this when we spin up the browser (you'll have to poke around to find the equivalent python - sorry!):

DesiredCapabilities capabilities = DesiredCapabilities.chrome();
//Proxy is from the namespace org.openqa.selenium
Proxy proxy = new Proxy();
//10.0.53.132 == our browsermob proxy server
proxy.setHttpProxy("10.0.53.132:8888");
proxy.setSslProxy("10.0.53.132:8888");
capabilities.setCapability(CapabilityType.PROXY,proxy);
this.driverCapabilities = capabilities;

answered Jan 6, 2014 at 17:10

Aaron B.

711 silver badge8 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

sergzach Over a year ago

Is browsermob server on the same host as the tested website?

Aaron B. Over a year ago

That was an awfully long time ago...! :) But no, the browserMob proxy was not running from the app server. And Selenium has had a few revision since then. So your mileage may vary.

sergzach Over a year ago

I just try to understand if proxy server could turn round https by installing on the same server and by attaching it on https (with help of nginx using information about certification keys).

Collectives™ on Stack Overflow

BrowserMob Proxy + selenium capturing https requests

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related