0

I am using Selenium to find information in Kaggle. In order to do so, I am looking to scrape information on people solutions.

When I navigate to this url: https://www.kaggle.com/allen-institute-for-ai/CORD-19-research-challenge/tasks?taskId=882. There are people solutions at the end of the task.

enter image description here

I would like to click on those elements. From the page source, it doesn't look like pressing the solution should navigate to notebook of the solution (the < a href="/{usename} opens the user info tab, and not the notebook tab).

This is how the html of an element looks like: enter image description here

  <li class="sc-qQkIG hIKhtZ">
   <a href="/moghazy" target="_blank" rel="noopener" class="sc-pZMVu kiTgMN">
      <div class="sc-oTPjJ cCLmQH"><img src="https://storage.googleapis.com/kaggle-avatars/thumbnails/777390-kg.jpg" alt="Moghazy" class="sc-pckkE ljVeKr"><img src="/static/images/avatier/[email protected]" alt="expert" class="sc-pjIPr cvnVIi"></div>
   </a>
   <div class="sc-qYgLf hNZirj">
      <div class="sc-pRStN jTbOhh">
         <div class="sc-AxheI sc-fznWqX sc-qanuI frNLaF">COVID-19 Literature Ranking + Web Scraping</div>
      </div>
      <div class="sc-psdQm hrPHBT"><span class="sc-fzpkJw sc-fznzOf sc-oToFz dtnpxb"><a href="/moghazy" target="_blank" rel="noopener" class="sc-fzqMAW sc-fzoydu sc-pbJGu jFfyPB">Moghazy</a> · <span title="Sat May 16 2020 02:44:32 GMT+0300 (Israel Daylight Time)">6 days ago</span> · Notebook · Python · 4 Comments</span></div>
   </div>
   <div class="sc-pJsLC cBTgaO">
      <div class="sc-pZzGt eDxwsZ sc-pkjoF iehzxN">
         <button class="MuiButtonBase-root MuiButton-root sc-oTcWe sc-paYYD hlGLPZ MuiButton-outlined" tabindex="0" type="button" state="no-vote" data-testid="vote-button-icon">
            <span class="MuiButton-label label">
               <svg aria-hidden="true" focusable="false" data-prefix="fas" data-icon="angle-up" class="svg-inline--fa fa-angle-up fa-w-10 " role="img" xmlns="http://www.w3.org/2000/svg" viewBox="0 0 320 512">
                  <path fill="currentColor" d="M177 159.7l136 136c9.4 9.4 9.4 24.6 0 33.9l-22.6 22.6c-9.4 9.4-24.6 9.4-33.9 0L160 255.9l-96.4 96.4c-9.4 9.4-24.6 9.4-33.9 0L7 329.7c-9.4-9.4-9.4-24.6 0-33.9l136-136c9.4-9.5 24.6-9.5 34-.1z"></path>
               </svg>
            </span>
            <span class="MuiTouchRipple-root"></span>
         </button>
         <button class="MuiButtonBase-root MuiButton-root sc-oTcWe sc-pjUyM kZNtvl MuiButton-outlined Mui-disabled Mui-disabled" tabindex="-1" type="button" disabled="" state="no-vote"><span class="MuiButton-label label">14</span></button>
      </div>
   </div>
</li>

I find the elements using this code:

elements = driver.find_element_by_class_name("sc-oUaSW").find_elements_by_xpath(".//li")
for element in elements:
    element.click()

but I am getting the error:

Message:

element click intercepted: Element ... is not clickable

but when I click on it in the server, it is clickable.

What am I doing wrong?

1 Answer 1

1

There are several issues to address:

  1. Element is not clickable because it is not in focus. You need to move to it first.
  2. The click opens a new tab. This requires its own handling.
  3. Moving back to the original tab to click the next item.

This snippet should work:

main_window = driver.current_window_handle #this relates to issues 2, 3

elements = driver.find_element_by_class_name("sc-oUaSW").find_elements_by_xpath(".//li")
for element in elements:
    ActionChains(driver).move_to_element(element).click().perform()  #this addresses issue 1
    #element.click()
    driver.switch_to.window(driver.window_handles[1])   #this relates to issue 2
    WebDriverWait(driver, 10).until(EC.visibility_of_element_located((By.XPATH, "//a[@class='KernelViewerContext_KernelTitle-sc-rdaqnd chqxNN']"))) 
    #time.sleep(3)
    #do whatever you wish with the tab/task here
    driver.close()
    driver.switch_to.window(main_window)    # close tab and switch back to original browser tab (issue 3)

These imports are required for the above to work:

from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.common.action_chains import ActionChains
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.