Scrolling to an element in Selenium

Question

I am using selenium to scrape a dynamic website. My problem is that I can't scroll down to the items I am looking for. First, I made a list of those items I want to get information from (python_button1). I have to click on each item in this list to see a new window, in which I want to scrape them. Then I made a loop to click on those items. After that, I close the window I opened. Unfortunately, I can't scroll down to each item to click them. I used the following code but it scrolls down to the last item and skips other items. I would really appreciate if you can help me. Thanks!

python_button1 = driver.find_elements_by_class_name('simboloEvento')

for x in python_button1:
    x.click()
    time.sleep(2)
    driver.find_element_by_class_name('cerrarBoton').click() 
    driver.execute_script("coordinates=arguments[0]. 
    getBoundingClientRect();scrollTo( coordinates.x,coordinates.y);", x)
    time.sleep(2)


innerHTML = driver.execute_script("return document.body.innerHTML")
print(innerHTML)

the scroll down list is a JavaScript Object. I have to click on each of those objects to activate the javascript function ( a new window). I can activate those objects via my code, but the problem is when at least one of those items are not in the current screen. That's why I should scroll down to that object ( here x) first to be able to click on it. But I don't know how to scroll down to that object. I have used the code I used in this post as well as this one : driver.execute_script("arguments[0].scrollIntoView();", x) but none of them worked for me!

Please show HTML(don't use screenshot) and a screenshot of the items — yong
– yong, Commented May 21, 2018 at 23:43
Yes, the scroll down list is a JavaScript Object. I have to click on each of those objects to activate the javascript function ( a new window). I can activate those objects via my code, but the problem is when at least one of those items are not in the current screen. That's why I should scroll down to that object ( here x) first to be able to click on it. But I don't know how to scroll down to that object. I have used the code I used in this post as well as this one : driver.execute_script("arguments[0].scrollIntoView();", x) but none of them worked for me! — Mori shah
– Mori shah, Commented May 22, 2018 at 14:17

dmb · Accepted Answer · 2018-05-24 04:29:45Z

1

For this you need two things. In first place know where the element is, and you can do it with python-js like this:

script = 'return arguments[0].getBoundingClientRect().top + document.documentElement.scrollTop;'
posY = driver.execute_script(script, element).split('.')[0]

The above combo will return a string that represent the relative position of a element to the top of the document(website)

Then you can scroll to the position you want

driver.execute_script('window.scrollTo(0, '+ posY +');')

This one will move the screen where the element is, some element needs to be waited for, so I would suggest some waiter function of this nature:

def waiter(token):
    # token is whatever you spect to be in the element like str("/") in a date field
    max_time = 60 #segs
    driver.switch_to.default_content()

    while token not in driver.find_element_by_xpath('//*[@Class="important"]'):
        time.sleep(1)
        max_time += -1
        if max_time == 0:
            raise Exception('Waited too long')

Made myself a small modules with js navigation, because even though selenium is powerfull sometimes it doesn't want to do it's thing on some websites.

EDIT: For completeness this snippet makes the element visible:

driver.execute_script(“arguments[0].style.visibility = 'visible';”, element)

This works because arguments[0] represents the element argument in execute_script

edited May 24, 2018 at 4:29

answered May 22, 2018 at 16:05

dmb

2962 silver badges9 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Mori shah Over a year ago

If you don't mind, could you please tell me how you added js_in . I want to run the code but js_in is undefined. Thanks!

dmb Over a year ago

@Morishah I've edited the code, js_in is the script you pass to execute_script, so I've renamed it as script

Mori shah Over a year ago

I have tried it and it works as long as a new window doesn't cover the next object in the loop. When I click the object, a new window that I want to get data from activates but it covers the next object and that's why I can't click the next object. I don't know how to address this issue. Here is the error: NoSuchElementException: no such element: Unable to locate element: ... Thanks!

dmb Over a year ago

@Morishah but that is what you asked. Thats a whole other question. For that you need to make sure that the element is visible, also viable by js

Mori shah Over a year ago

Can you tell me how I can make an element visible? Thanks!

TomH · Accepted Answer · 2018-05-24 04:43:11Z

0

Have you tried using the selenium actions class?

builder = ActionChains(driver)
builder.move_to_element(x).perform()

answered May 24, 2018 at 4:43

TomH

2,7692 gold badges20 silver badges34 bronze badges

Collectives™ on Stack Overflow

Scrolling to an element in Selenium

2 Answers 2

5 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

5 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related