3

I want to find element on page that only loads when user scroll to it. In order to scroll to it I have to find it. However in order to find it I have to scroll to it so it displays in html. Is there any workaround.

I tried finding elements by xpath to assure myself that there are no such elements findable.

channel_text = driver.find_element_by_xpath(f"//*[contains(text(), '{name_of_text_inside_tag}')]")

When I open the page in developer mode and search by xpath it findes none. However when I scroll to the element it findes it by the very same xpath.

As you can see in this picture I searched for channel named worldbuilding. As the channel was visible it found it. Here however the channel was not visible and element was not found.

from selenium import webdriver
from selenium.webdriver.common.keys import Keys

driver = webdriver.Chrome()

# get discord server
driver.get("https://discordapp.com/channels/393766374272663564")

# find channel on SCROLLABLE side menu
channel_text = driver.find_element_by_xpath(f"//*[contains(text(), 'channel-name')]")
print(channel_text) # prints empty list

Would appreciate any help.

3
  • 1
    There's no workaround: you should make a loop of find-scroll_a_page routine and break upon finding. Commented Jul 15, 2019 at 12:05
  • 1
    Create a loop that scrolls to the bottom of the page and then checks for the element. Commented Jul 15, 2019 at 12:08
  • Demo credentials? Commented Jul 16, 2019 at 10:20

2 Answers 2

4

You may create a loop that will scroll to the bottom of the page and search for the element.

The element is being loaded after the scroll, so you need to wait for it. This will make your script slower, but reliable. Create a wait object with a tolerable time delay. As you may need to scroll several times, the time allowed before timeout should be small. I'd suggest you use a wait object created specifically for this purpose, with a 5 or less second timeout.

You also need to set a limit so that it won't be looking forever.

from selenium.common.exceptions import NoSuchElementException, TimeoutException
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

# create a dedicated wait object to wait for a brief period for the elements to be created.
wait = WebDriverWait(driver, 5)
# (...)
# find channel on SCROLLABLE side menu

# limit the number of scrolls
count = 0
while count < 5:
    driver.execute_script("window.scrollTo(0,document.body.scrollHeight);")
    try:
        channel_text = wait.until(
            EC.presence_of_element_located(
             (By.XPATH, f"//*[contains(text(), 'channel-name')]")
            )
        )
        break
    except TimeoutException:
        pass
    count += 1
else:
    # do whatever must be done if the element is never found.
    pass
print(channel_text)
Sign up to request clarification or add additional context in comments.

5 Comments

i don't understad what you tried to achieve by !channel_tex and also if the element is not found on the first try it throws exception so the rest of the loop is useless.
The goal of !channel_text is to interrupt the while as soon as the channel_text is no longer empty.
With the try-except block there is no need for the !channel_text.
did it stop working? or was it that it worked before and then you updated the code and no longer works?
I tried your block of code on new file with things that are only needed for selenium and it simply never finds it. Problem is it is not casual scroll but it is element that is scrollable apparently.
-2

it worked for me, try this

wait.until(ExpectedConditions.presenceOfElementLocated(By.xpath("your locator")));

1 Comment

The question was about the Python implementation of Selenium bindings. The answer looks like a solution for another programming language though could be translated to Python.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.