Multiprocessing and Selenium Python

Question

I have 3 drivers (Firefox browsers) and I want them to do something in a list of websites.

I have a worker defined as:

def worker(browser, queue):
    while True:
        id_ = queue.get(True)
        obj = ReviewID(id_)
        obj.search(browser)
        if obj.exists(browser):
            print(obj.get_url(browser))
        else:
            print("Nothing")

So the worker will just acces to a queue that contains the ids and use the browser to do something.

I want to have a pool of workers so that as soon as a worker has finished using the browser to do something on the website defined by id_, then it immediately starts to work using the same browser to do something on the next id_ found in queue. I have then this:

pool = Pool(processes=3)  # I want to have 3 drivers
manager = Manager()
queue = manager.Queue()
# Define here my workers in the pool
for id_ in ids:
    queue.put(id_)
for i in range(3):
    queue.put(None)

Here I have a problem, I don't know how to define my workers so that they are in the pool. To each driver I need to assign a worker, and all the workers share the same queue of ids. Is this possible? How can I do it?

Another idea that I have is to create a queue of browsers so that if a driver is doing nothing, it is taken by a worker, along with an id_ from the queue in order to perform a new process. But I'm completely new to multiprocessing and actually don't know how to write this.

I appreciate your help.

Why not instantiate the browser in the worker, before the while loop begins? — Levi Noecker
– Levi Noecker, Commented Oct 3, 2016 at 14:47
@LeviNoecker can you post an answer in order to accept is as the answer? — Vladimir Vargas
– Vladimir Vargas, Commented Oct 4, 2016 at 0:50

Levi Noecker · Accepted Answer · 2016-10-04 12:36:46Z

6

You could try instantiating the browser in the worker:

def worker(queue):
    browser = webdriver.Chrome()
    try:
        while True:
            id_ = queue.get(True)
            obj = ReviewID(id_)
            obj.search(browser)
            if obj.exists(browser):
                print(obj.get_url(browser))
            else:
                print("Nothing")
    finally:
        brower.quit()

edited Oct 4, 2016 at 12:36

answered Oct 4, 2016 at 3:02

Levi Noecker

3,3191 gold badge19 silver badges31 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Multiprocessing and Selenium Python

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related