Multi threading using seleniumbase driver

Question

hello i have this code that use seleniumbase driver with rotated proxy to get over 50000 links i need multithreading with it here is my code :

from seleniumbase import Driver
def initialize_driver():
    driver = Driver(uc=True, incognito=True,proxy='username:password@ip:port',headless=True)
    return driver
def sen_email(link):
    driver = initialize_driver()
    (rest of the code )
    return dic
def contact_agent():
    leads_list = get_leads()
    data_list = []
    for link in leads_list :
        dic=sen_email(link)
        data_list.append(dic)
    return data_list

i did use some concurrent.futures and threading soultions but it wont work it just open mutliple instance of the driver one of them geting the link other not , some time one insten get 2 links with two windows and that not. what i want is each link to get open with it one instance of driver

Michael Mintz · Accepted Answer · 2023-09-12 20:34:34Z

0

If you're trying to run multi-threaded SeleniumBase tests in UC Mode with a proxy, here's how:

Multi-threading in UC Mode is possible if you use pytest multi-threading provided by pytest-xdist.

You'll need to use command-line options for this format, eg --uc to activate UC Mode, -n4 (for 4 parallel processes, etc), and --proxy=user:pass@host:port to set proxy settings.

Below is a sample run command:

pytest --uc -n4

(Add --proxy=user:pass@host:port to include proxy settings.)

Here's a sample file that uses @pytest.mark.parametrize() to turn one test into four tests when run with pytest:

import pytest

@pytest.mark.parametrize("", [[]] * 4)
def test_multi_threaded(sb):
    sb.driver.get("https://nowsecure.nl/#relax")
    try:
        sb.assert_text("OH YEAH, you passed!", "h1", timeout=5.25)
        sb.post_message("Selenium wasn't detected!", duration=2.8)
        sb._print("\n Success! Website did not detect Selenium! ")
    except Exception:
        sb.fail('Selenium was detected! Try using: "pytest --uc"')

Here's the output when running that file with pytest --uc -n4:

pytest test_multi_uc.py --uc -n4
============================ test session starts =============================
platform darwin -- Python 3.11.4, pytest-7.4.2, pluggy-1.3.0
rootdir: ~/github/SeleniumBase/examples
configfile: pytest.ini
plugins: html-2.0.1, rerunfailures-12.0, cov-4.1.0, metadata-3.0.0, ordering-0.6, xdist-3.3.1, seleniumbase-4.18.5
4 workers [4 items]     

 Success! Website did not detect Selenium! 

 Success! Website did not detect Selenium! 
..
 Success! Website did not detect Selenium! 
.
 Success! Website did not detect Selenium! 
.
============================= 4 passed in 9.38s ==============================

Some websites may block you if they detect multiple simultaneous connections like that. Be careful where you go.

Note that SeleniumBase has different syntax formats. See: https://github.com/seleniumbase/SeleniumBase/blob/master/help_docs/syntax_formats.md

Also note that pytest-xdist (the multi-processing library for pytest) is more advanced than https://docs.python.org/3/library/concurrent.futures.html for complex multi-threading of this nature.

answered Sep 12, 2023 at 20:34

Michael Mintz

16.1k9 gold badges51 silver badges109 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Ismail Amouma Over a year ago

thank u michael the methode u provided worked yestrday but i really need the multi threading on the Derver ,is there a way to do the threading on initialize_driver fonction becouse i switch my script to web app that get the progresse of sen email fonction in there when i need teh threading to be , so my qoustion is there any way to the theading Driver and get multiple instances of it at one time

Prakash Dahal Over a year ago

Hi Michael, can you update your answer by running 4 different links with 4 different proxies (each proxy for each link)? That would help a lot. Thanks

Collectives™ on Stack Overflow

Multi threading using seleniumbase driver

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related