How do I set a proxy for phantomjs/ghostdriver in python webdriver?

Question

I'm trying to figure out how to route my requests through an HTTP proxy.

I'm initializing webdriver like this:

user_agent = 'my user agent 1.0'
DesiredCapabilities.PHANTOMJS['phantomjs.page.settings.userAgent'] = user_agent
driver = webdriver.PhantomJS()

I've gone through the docs and the source and can't seem to find a way to use a proxy server with phantomjs for through webdriver.

Any suggestions?

Alex Nik · Accepted Answer · 2013-05-03 10:52:47Z

74

Below is the example of how to set proxy for PhantomJs in Python. You may change proxy type: socks5/http.

service_args = [
    '--proxy=127.0.0.1:9999',
    '--proxy-type=socks5',
    ]
browser = webdriver.PhantomJS('../path_to/phantomjs',service_args=service_args)

edited May 3, 2013 at 10:52

answered May 3, 2013 at 7:13

Alex Nik

7605 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Brendan Wood Over a year ago

In case you are using a proxy requiring authentication, you'll need to add another field to the service_args list '--proxy-auth=username:password'.

Chiedo Over a year ago

Any idea, how I would do the same thing in Ruby? I got the above working with Python but the following won't work for me:

service_args = [ '--proxy=127.0.0.1:9999', '--proxy-type=socks5'] driver = Selenium::WebDriver.for(:phantomjs, :service_args => service_args)

Chiedo Over a year ago

I actually figured it out, phantomjs_args = [ '--proxy=127.0.0.1:9999', '--proxy-type=socks5'] phantomjs_caps = { "phantomjs.cli.args" => phantomjs_args } driver = Selenium::WebDriver.for(:phantomjs, :desired_capabilities => phantomjs_caps)

ballade4op52 Over a year ago

Is there a way to change the proxy without having to open a new driver?

Pykler · Accepted Answer · 2013-05-28 17:23:19Z

I dug a little and I found that the functionality is there, but it is not exposed. So it requires a handy monkey wrench to patch it up. Here is the solution that works for me until this functionality is fully exposed in the webdriver call.

EDIT: it seems the service_args are now exposed, you no longer need to monkey patch selenium to use the proxy ... see @alex-czech answer for how to use.

from selenium import webdriver
from selenium.webdriver.phantomjs.service import Service as PhantomJSService

phantomjs_path = '/usr/lib/node_modules/phantomjs/lib/phantom/bin/phantomjs'
# monkey patch Service temporarily to include desired args
class NewService(PhantomJSService):
    def __init__(self, *args, **kwargs):
        service_args = kwargs.setdefault('service_args', [])
        service_args += [
            '--proxy=localhost:8080',
            '--proxy-type=http',
        ]
        super(NewService, self).__init__(*args, **kwargs)
webdriver.phantomjs.webdriver.Service = NewService
# init the webdriver
self.driver = webdriver.PhantomJS(phantomjs_path)
# undo monkey patch
webdriver.phantomjs.webdriver.Service = PhantomJSService

Also useful are the following settings, especially when using a proxy that may take a very long time to load.

max_wait = 60
self.driver.set_window_size(1024, 768)
self.driver.set_page_load_timeout(max_wait)
self.driver.set_script_timeout(max_wait)

Sam R. · Accepted Answer · 2015-08-15 02:11:46Z

5

The following is how to do the same with the Webdriver in Ruby. I couldn't find this anywhere online until I dug into the source code:

phantomjs_args = [ '--proxy=127.0.0.1:9999', '--proxy-type=socks5']
phantomjs_caps = { "phantomjs.cli.args" => phantomjs_args }
driver = Selenium::WebDriver.for(:phantomjs, :desired_capabilities => phantomjs_caps)

edited Aug 15, 2015 at 2:11

Sam R.

16.4k14 gold badges76 silver badges126 bronze badges

answered Mar 21, 2015 at 13:40

Chiedo

7,5843 gold badges28 silver badges23 bronze badges

1 Comment

Maxim Mazurok Over a year ago

Thanks! You helped me to finally do it in PHP:

$capabilities = array("phantomjs.cli.args" => ['--proxy=127.0.0.1:9999', '--proxy-type=socks5']); $driver = RemoteWebDriver::create($host, $capabilities, 5000);

Tom · Accepted Answer · 2017-10-19 16:48:49Z

I ended up needing to pass the credentials in both the service_args & as a proxy-auth header. I don't believe phantomjs passes the proxy auth onwards correctly.

service_args = [
    "--ignore-ssl-errors=true",
    "--ssl-protocol=any",
    "--proxy={}".format(proxy),
    "--proxy-type=http",
]

caps = DesiredCapabilities.PHANTOMJS

authentication_token = "Basic " + base64.b64encode(b'{}:{}'.format(username, password))

caps['phantomjs.page.customHeaders.Proxy-Authorization'] = authentication_token

self.driver = webdriver.PhantomJS(
        service_args=service_args,
        desired_capabilities=caps,
        executable_path="./phantomjs-2.1.1-linux-x86_64/bin/phantomjs")

Where proxy's structure is defined as http://username:password@domain:port

I'd hazard a guess that the first auth-parameters aren't passed as a header to the proxy, so you need to do both manually.

jofrepp · Accepted Answer · 2021-01-19 20:07:10Z

0

PhantomJS updated the CLI arguments without updating the documentation. The proxy type has to be included in the proxy address as follows:

service_args = ['--proxy=http://0.0.0.0:0']
driver = webdriver.PhantomJS(service_args=service_args)

answered Jan 19, 2021 at 20:07

jofrepp

1291 silver badge4 bronze badges

Collectives™ on Stack Overflow

How do I set a proxy for phantomjs/ghostdriver in python webdriver?

5 Answers 5

4 Comments

Comments

1 Comment

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

4 Comments

Comments

1 Comment

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related