Use selenium to navigate in joint selection/list elements

Question

I'm using beautifulSoup / selenium to do some webscraping and am hitting a wall with a certain dropdown select menu. The rough HTML is as follows:

<div class="selection-box" alt="selection" title="selection" role="select" tabindex="0">
<select id="select" style="display: none;">
    <option value="1">First</option>
    <option value="2">Second</option>
    <option value="3" selected="selected">Third</option>
</select>
<div class="current">Third</div>
<ul class="options" style="display: none;">
    <li class="search--option" alt="First option" title="First option" aria-label="First option" role="option" tabindex="0">First</li>
    <li class="search--option" alt="Second option" title="Second option" aria-label="Second option" role="option" tabindex="0">Second</li>
    <li class="search--option selected" alt="Third option" title="Third option" aria-label="Third option" role="option" tabindex="0">Third</li>
</ul>

When I'm working the menu through the browsers, it changes as follows:

the wrapper div class changes to "selection-box active"
the ul changes to "display: block"
once I pick a different option, those two get reversed again and the middle div and the selected li item change accordingly

I want to use selenium to select a certain option. So far, I tried the following:

    from selenium.webdriver.support.ui import Select
    drpBrand = driver.find_element(By.ID, "select");
    css = 'select#select' # css selector of the element
    js = """const data_options = Array.from(document.querySelectorAll('{css}'));
            data_options.forEach(a=>{{a.style='display:block;';}});""".format(css=css)
    driver.execute_script(js)
    drpBrand.select_by_visible_text("Third");

This is a best-of using various threads (element not visible: Element is not currently visible and may not be manipulated - Selenium webdriver, How to select a dropdown value in Selenium WebDriver using Java), but it still doesn't work. Any ideas? I assume I need to target the list as well (besides the select)?

The error is always

selenium.common.exceptions.ElementNotInteractableException: Message: element not interactable: Element is not currently visible and may not be manipulated

Thanks

wwe .com/superstars. Scroll down until you see the selection box (says CURRENT by default) — Berbatov
– Berbatov, Commented Jan 7, 2024 at 13:40

Yaroslavm · Accepted Answer · 2024-01-08 17:56:46Z

1

If I understood your goal correct (to select dropdown option), I suggest to emulate native user behaviour and rely on visible elements.

Dropdown element has selector .superstar-search--selection-box. You should wait for it's presence and click on it.

Dropdown options have selector .superstar-search--option . You should wait for visibility of this elements and filter them, for example, by containing text condition.

You can build more complex function than I wrote by defining dropdown open selector and dropdown option selector as function arguments.

If you have issue with Google Ads, I added example how to deal with it.

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC

driver = webdriver.Chrome()

driver.get("https://www.wwe.com/superstars")
wait = WebDriverWait(driver, 15)
driver.maximize_window()

def select_dropdown_option_by_text(text):
    dropdown = wait.until(EC.presence_of_element_located((By.CSS_SELECTOR, '.superstar-search--selection-box')))
    dropdown.click()
    options = wait.until(EC.visibility_of_all_elements_located((By.CSS_SELECTOR, '.superstar-search--option')))
    expected_option = [element for element in options if element.text.lower() == text][0]
    expected_option.click()
    wait.until(EC.invisibility_of_element(expected_option))

def remove_google_ads():
    return driver.execute_script("""
      function waitForElementAndRemove() {
        let element = document.querySelector('[id*=google_ads_iframe],[id*=ad_iframe]');
        if (element) {
            element.remove();
            console.log('Removed ad');
        } else {
           setTimeout(waitForElementAndRemove, 1000);
        }
    }

      waitForElementAndRemove();
    """)

remove_google_ads()
select_dropdown_option_by_text('all superstars')

edited Jan 8, 2024 at 17:56

answered Jan 7, 2024 at 22:59

Yaroslavm

4,84410 gold badges18 silver badges36 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Berbatov Over a year ago

Thanks for looking into this! When i run this, I get the following error: selenium.common.exceptions.ElementClickInterceptedException: Message: element click intercepted: Element <div class="superstar-search--selection-box" alt="Superstars" title="Superstars" role="select" tabindex= "0">...</div> is not clickable at point (462, 582). Other element would receive the click: <div id="google_ads_iframe_/4675/desktop/superstars_1__container__" style="border: 0pt none; width: 728px; height: 90p x;"></div> This is caused by the dropdown.click() when running the function manually line by line

Yaroslavm Over a year ago

@Berbatov that is another the issue that is not related to dropdown open. How to deal with random Google ads I described there stackoverflow.com/questions/77657693/… Also I updated snippet.

Collectives™ on Stack Overflow

Use selenium to navigate in joint selection/list elements

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related