Error running Selenium Webdriver with a proxy in Python

Question

I tested a script with Selenium Python to retrieve information from a web page. It works, at least until my IP is found. I would like to try using a proxy. I have tried the following two alternatives.

First way:

PROXY = "176.31.68.252:20213"

webdriver.DesiredCapabilities.CHROME['proxy'] = {
    "httpProxy": PROXY,
    "ftpProxy": PROXY,
    "sslProxy": PROXY,
    "noProxy": None,
    "proxyType": "MANUAL",
    "class": "org.openqa.selenium.Proxy",
    "autodetect": False
}

url = 'https://www.ufficiocamerale.it/'

driver = webdriver.Chrome(desired_capabilities=webdriver.DesiredCapabilities.CHROME)
wait = WebDriverWait(driver, 20)
driver.get(url)

# rest of the code

Second way:

PROXY = "176.31.68.252:20213"

chrome_options = webdriver.ChromeOptions()
chrome_options.add_argument('--proxy-server=https://%s' % PROXY)

url = 'https://www.ufficiocamerale.it/'

driver = webdriver.Chrome(chrome_options=chrome_options)
wait = WebDriverWait(driver, 20)
driver.get(url)

# rest of the code

But in the first case the webpage returns the error ERR_EMPTY_RESPONSE and in the second case it returns the error ERR_TIMED_OUT.

I found the proxy in an online free proxy list.

Is this a problem with my chosen proxy? Or is it a code problem? What is the standard way to proceed in these cases?

xtekky · Accepted Answer · 2022-03-17 01:10:20Z

1

Google Restriction

You are getting an error message because of the proxy,

Google blocks all free proxy's

Proxy Spider

I also coded a proxy spider that tests all proxy's on: https://free-proxy-list.net/

Code: https://github.com/xtekky/proxy-spider

Setup Own Proxy

Here is an article that tells you how to create your own proxy server.

Setup Selenium with proxy

Try this script:

Github Code: https://github.com/xtekky/selenium-tutorials/tree/main/selenium%20proxy

from selenium import webdriver
from selenium.webdriver.common.proxy import ProxyType, Proxy #proxy module
import time

proxy_ip = 'ip:port' #get a free proxy from the websites in the description

#setting up proxy
proxy =Proxy()
proxy.proxy_type = ProxyType.MANUAL
proxy.http_proxy = proxy_ip
proxy.ssl_proxy = proxy_ip

#linking proxy and setting up driver
capabilities = webdriver.DesiredCapabilities.CHROME
proxy.add_to_capabilities(capabilities)
driver = webdriver.Chrome('CHROMEDRIVER PATH', desired_capabilities=capabilities) # replace the chromedriver path

#loading test page
driver.get('https://httpbin.org/ip')
time.sleep(8)
driver.quit()

edited Mar 17, 2022 at 1:10

answered Mar 16, 2022 at 17:14

xtekky

6401 gold badge7 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

LJG Over a year ago

Thank you very much for your reply. If Google blocks all free proxies, how can I use the code in your answer? I got a list of valid proxies from free-proxy-list.net.

xtekky Over a year ago

these proxies still dont work, or partly, my proxy spider checks which ones work but its generally around 4-5 of the 300 listed on the site. The code on my answer is simply a code that allow's you to use a proxy with selenium, your scripts didn't work for me

LJG Over a year ago

So the code in your answer should work for example with non-free proxies?

xtekky Over a year ago

yes, but to get private proxies you either have to pay or set up your own proxy server

Collectives™ on Stack Overflow

Error running Selenium Webdriver with a proxy in Python

1 Answer 1

Google Restriction

Proxy Spider

Setup Own Proxy

Setup Selenium with proxy

Try this script:

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Google Restriction

Proxy Spider

Setup Own Proxy

Setup Selenium with proxy

Try this script:

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related