How to handle elements inside Shadow DOM from Selenium

Sometimes the shadow root elements are nested and the second shadow root is not visible in document root, but is available in its parent accessed shadow root. I think is better to use the selenium selectors and inject the script just to take the shadow root:

def expand_shadow_element(element):
  shadow_root = driver.execute_script('return arguments[0].shadowRoot', element)
  return shadow_root

outer = expand_shadow_element(driver.find_element_by_css_selector("#test_button"))
inner = outer.find_element_by_id("inner_button")
inner.click()

To put this into perspective I just added a testable example with Chrome's download page, clicking the search button needs open 3 nested shadow root elements:

import selenium
from selenium import webdriver
driver = webdriver.Chrome()


def expand_shadow_element(element):
  shadow_root = driver.execute_script('return arguments[0].shadowRoot', element)
  return shadow_root

driver.get("chrome://downloads")
root1 = driver.find_element_by_tag_name('downloads-manager')
shadow_root1 = expand_shadow_element(root1)

root2 = shadow_root1.find_element_by_css_selector('downloads-toolbar')
shadow_root2 = expand_shadow_element(root2)

root3 = shadow_root2.find_element_by_css_selector('cr-search-field')
shadow_root3 = expand_shadow_element(root3)

search_button = shadow_root3.find_element_by_css_selector("#search-button")
search_button.click()

Doing the same approach suggested in the other answers has the drawback that it hard-codes the queries, is less readable and you cannot use the intermediary selections for other actions:

search_button = driver.execute_script('return document.querySelector("downloads-manager").shadowRoot.querySelector("downloads-toolbar").shadowRoot.querySelector("cr-search-field").shadowRoot.querySelector("#search-button")')
search_button.click()

later edit:

I recently try to access the content settings (see code below) and it has more than one shadow root elements imbricated now you cannot access one without first expanding the other, when you usually have also dynamic content and more than 3 shadow elements one into another it makes impossible automation. The answer above use to work a few time ago but is enough for just one element to change position and you need to always go with inspect element an ho up the tree an see if it is in a shadow root, automation nightmare.

Not only was hard to find just the content settings due to the shadowroots and dynamic change when you find the button is not clickable at this point.

driver = webdriver.Chrome()


def expand_shadow_element(element):
  shadow_root = driver.execute_script('return arguments[0].shadowRoot', element)
  return shadow_root

driver.get("chrome://settings")
root1 = driver.find_element_by_tag_name('settings-ui')
shadow_root1 = expand_shadow_element(root1)

root2 = shadow_root1.find_element_by_css_selector('[page-name="Settings"]')
shadow_root2 = expand_shadow_element(root2)

root3 = shadow_root2.find_element_by_id('search')
shadow_root3 = expand_shadow_element(root3)

search_button = shadow_root3.find_element_by_id("searchTerm")
search_button.click()

text_area = shadow_root3.find_element_by_id('searchInput')
text_area.send_keys("content settings")

root0 = shadow_root1.find_element_by_id('main')
shadow_root0_s = expand_shadow_element(root0)


root1_p = shadow_root0_s.find_element_by_css_selector('settings-basic-page')
shadow_root1_p = expand_shadow_element(root1_p)


root1_s = shadow_root1_p.find_element_by_css_selector('settings-privacy-page')
shadow_root1_s = expand_shadow_element(root1_s)

content_settings_div = shadow_root1_s.find_element_by_css_selector('#site-settings-subpage-trigger')
content_settings = content_settings_div.find_element_by_css_selector("button")
content_settings.click()

There is also ready to use pyshadow pip module, which worked in my case, below example:

from pyshadow.main import Shadow
from selenium import webdriver

driver = webdriver.Chrome('chromedriver.exe')
shadow = Shadow(driver)
element = shadow.find_element("#Selector_level1")
element1 = shadow.find_element("#Selector_level2")
element2 = shadow.find_element("#Selector_level3")
element3 = shadow.find_element("#Selector_level4")
element4 = shadow.find_element("#Selector_level5")
element5 = shadow.find_element('#control-button') #target selector
element5.click() 

I would add this as a comment but I don't have enough reputation points--

The answers by Eduard Florinescu works well with the caveat that once you're inside a shadowRoot, you only have the selenium methods available that correspond to the available JS methods--mainly select by id.

To get around this I wrote a longer JS function in a python string and used native JS methods and attributes (find by id, children + indexing etc.) to get the element I ultimately needed.

You can use this method to also access shadowRoots of child elements and so on when the JS string is run using driver.execute_script()

With selenium 4.1 there's a new attribute shadow_root for the WebElement class.

From the docs:

Returns a shadow root of the element if there is one or an error. Only works from Chromium 96 onwards. Previous versions of Chromium based browsers will throw an assertion exception.

Returns:

• ShadowRoot object or
• NoSuchShadowRoot - if no shadow root was attached to element

A ShadowRoot object has the methods find_element and find_elements but they're currently limited to:

• By.ID
• By.CSS_SELECTOR
• By.NAME
• By.CLASS_NAME

Shadow roots and explicit waits

You can also combine that with WebdriverWait and expected_conditions to obtain a decent behaviour. The only caveat is that you must use EC that accept WebElement objects. At the moment it's just one of the following ones:

• element_selection_state_to_be
• element_to_be_clickable
• element_to_be_selected
• invisibility_of_element
• staleness_of
• visibility_of

Example

e.g. borrowing the example from eduard-florinescu

from selenium.webdriver.support.ui import WebDriverWait

driver = webdriver.Chrome()
timeout = 10

driver.get("chrome://settings")
root1 = driver.find_element_by_tag_name('settings-ui')
shadow_root1 = root1.shadow_root

root2 = WebDriverWait(driver, timeout).until(EC.visibility_of(shadow_root1.find_element(by=By.CSS_SELECTOR, value='[page-name="Settings"]')))
shadow_root2 = root2.shadow_root

root3 = WebDriverWait(driver, timeout).until(EC.visibility_of(shadow_root2.find_element(by=By.ID, value='search')))
shadow_root3 = root3.shadow_root

search_button = WebDriverWait(driver, timeout).until(EC.visibility_of(shadow_root3.find_element(by=By.ID, value="searchTerm")))
search_button.click()

text_area = WebDriverWait(driver, timeout).until(EC.visibility_of(shadow_root3.find_element(by=By.ID, value='searchInput')))
text_area.send_keys("content settings")

root0 = WebDriverWait(driver, timeout).until(EC.visibility_of(shadow_root1.find_element(by=By.ID, value='main')))
shadow_root0_s = root0.shadow_root


root1_p = WebDriverWait(driver, timeout).until(EC.visibility_of(shadow_root0_s.find_element(by=By.CSS_SELECTOR, value='settings-basic-page')))
shadow_root1_p = root1_p.shadow_root


root1_s = WebDriverWait(driver, timeout).until(EC.visibility_of(shadow_root1_p.find_element(by=By.CSS_SELECTOR, value='settings-privacy-page')))
shadow_root1_s = root1_s.shadow_root

content_settings_div = WebDriverWait(driver, timeout).until(EC.visibility_of(shadow_root1_s.find_element(by=By.CSS_SELECTOR, value='#site-settings-subpage-trigger')))
content_settings = WebDriverWait(driver, timeout).until(EC.visibility_of(content_settings_div.find_element(by=By.CSS_SELECTOR, value="button")))
content_settings.click()

You can use the driver.executeScript() method to access the HTML elements and JavaScript objects in your web page.

In the exemple below, executeScript will return in a Promise the Node List of all <a> elements present in the Shadow tree of element which id is host. Then you can perform you assertion test:

it( 'check shadow root content', function () 
{
    return driver.executeScript( function ()
    {
        return host.shadowRoot.querySelectorAll( 'a' ).then( function ( n ) 
        {
            return expect( n ).to.have.length( 3 )
        }
    } )
} )     

Note: I don't know Python so I've used the JavaScript syntax but it should work the same way.

The downloaded items by google-chrome are within multiple #shadow-root (open).

Solution

To extract the contents of the table you have to use shadowRoot.querySelector() and you can use the following locator strategy:

• Code Block:

  driver = webdriver.Chrome(service=s, options=options)
  driver.execute("get", {'url': 'chrome://downloads/'})
  time.sleep(5)
  download = driver.execute_script("""return document.querySelector('downloads-manager').shadowRoot.querySelector('downloads-item').shadowRoot.querySelector('a#file-link')""")
  print(download.text)
  

I originally implemented Eduard's solution just slightly modified as a loop for simplicity. But when Chrome updated to 96.0.4664.45 selenium started returning a dict instead of a WebElement when calling 'return arguments[0].shadowRoot'.

I did a little hacking around and found out I could get Selenium to return a WebElement by calling return arguments[0].shadowRoot.querySelector("tag").

Here's what my final solution ended up looking like:

def get_balance_element(self):
        # Loop through nested shadow root tags
        tags = [
            "tag2",
            "tag3",
            "tag4",
            "tag5",
            ]

        root = self.driver.find_element_by_tag_name("tag1")

        for tag in tags:
            root = self.expand_shadow_element(root, tag)

        # Finally there.  GOLD!

        return [root]

def expand_shadow_element(self, element, tag):
    shadow_root = self.driver.execute_script(
        f'return arguments[0].shadowRoot.querySelector("{tag}")', element)
    return shadow_root

Clean and simple, works for me.

Also, I could only get this working Selenium 3.141.0. 4.1 has a half baked shadow DOM implementation that just manages to break everything.

Here's a generator for getting all the web elements from the shadow DOM:

def flatten_shadows(driver):
    from selenium.webdriver.common.by import By
    from selenium.webdriver.support.wait import WebDriverWait
    from selenium.webdriver.support import expected_conditions as EC

    def get_all_elements(driver, elements):
        for el in elements:
            shadow_root = driver.execute_script('return arguments[0].shadowRoot', el)
            if shadow_root:
                shadow_els = driver.execute_script('return arguments[0].shadowRoot.querySelectorAll("*")', el)
                yield from get_all_elements(driver, shadow_els)
            else:
                yield el

    els = WebDriverWait(driver, 10).until(EC.presence_of_all_elements_located((By.CSS_SELECTOR, '*')))
    yield from get_all_elements(driver, els)