Python selenium - get element by its xPath and access it

Question

I am using Python and Selenium to scrape a webpage, in some cases, I can't get it to work.

I would like to access the element with text 'PInt', which is the second link in the below code. The xPath for it (copied from Developer console) is: //[@id="submenu1"]/a[2]

      <div id="divTest" onscroll="SetDivPosition();" style="height: 861px;">
            <div class="menuTitle" id="title1">
                <a href="#" onclick="toggle(1);"> </a>
            </div>
            <div class="subtitle" id="submenu1">    
                <img src="images/spacer.gif" border="0" width="2px" height="12px">
                <a href="#" class="NormalBlueSmall" onclick="clickItem('area/search/mov/mov2','mov');">Mov</a><br>
                <img src="images/spacer.gif" border="0" width="2px" height="12px">
                <a href="#" class="NormalBlueSmall" onclick="clickItem('area/con/ExtInt/extInt','pIint');">PInt</a><br>
                <img src="images/spacer.gif" border="0" width="2px" height="12px">
                <a href="#" class="NormalBlueSmall" onclick="clickItem('GoToNew.asp?link=asw_cnt/SmanSwif.aspx','SMAN/SWIF');">SWAM / SWIF</a><br>
            </div>
...

A snippet of my code is:

try:
    res = driver.find_elements_by_link_text('PInt')
    print("res1:{}".format(res))

    res = driver.find_element(By.XPATH,'//*[@id="submenu1"]/a[3]')
    print("res:{} type[0]:{}".format(res,res[0]))
    itm1 = res[0]
    itm1.click()

I get the error:

Unable to locate element:
{"method":"xpath","selector":"//*[@id="submenu1"]/a[2]"}

My question is, how can I get the right xPath of the element or any other way to access the element?

UPDATE: This might be important, the issue with Message: invalid selector: Unable to locate an element with the xpath expression (and I've tried all proposed solutions) might be that this is after authenticate in the webpage (User + Pwd) before, everything works. I noticed that the url driver.current_url after login is static (asp page). Also this part I am trying to access in a frameset and frame

html > frameset > frameset > frame:nth-child(1)

Is the element inside an iframe? You may also have to wait for it to be available. — JeffC
– JeffC, Commented Mar 22, 2017 at 1:26
@JeffC it is not inside an iFrame but rather a <frameset ... <frame name = "menu" .. — Alg_D
– Alg_D, Commented Mar 22, 2017 at 20:17
You will need to switch to the frame then. You should be able to find some sample code. Try that and see if it works. — JeffC
– JeffC, Commented Mar 22, 2017 at 20:19

Alg_D · Accepted Answer · 2017-03-22 22:13:32Z

3

Thanks to @JeffC to point me in the right direction.

since the page has some frames, I manage to access the element first by switching to the right frame (using xPath) and then access the element.

driver.switch_to.default_content()
driver.switch_to.frame(driver.find_element_by_xpath('html / frameset / frameset / frame[1]'))

driver.find_element_by_xpath("//a[contains(text(),'PInt')]").click()

BTW, in case you want to run the script from a crontab,you need to setup a display:

30 5 * * * export DISPLAY=:0; python /usr/.../main.py

edited Mar 22, 2017 at 22:13

answered Mar 22, 2017 at 21:09

Alg_D

2,4307 gold badges37 silver badges68 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Onel Harrison · Accepted Answer · 2017-03-21 23:16:00Z

1

To see a full list of all the ways of selecting elements using selenium, you can read all about it in the documentation.

Using xpath:

res = driver.find_element_by_xpath(u'//*[@id="submenu1"]/a[2]')

Using css selector:

res = driver.find_element_by_css_selector('#submenu1 a:nth-of-type(2)')

edited Mar 21, 2017 at 23:16

answered Mar 21, 2017 at 22:45

Onel Harrison

1,33412 silver badges14 bronze badges

3 Comments

Onel Harrison Over a year ago

@Algina - I fixed a mistake with my css selector. Give that a try. Regarding the xpath, the error you're getting would mean that the element with the given xpath doesn't exist or the xpath isn't correct.

Alg_D Over a year ago

Thanks Onel. It's the same error: Message: no such element: Unable to locate element: {"method":"css selector","selector":"#submenu1 a:nth-of-type(2)"} Perhaps the page is somehow protected against this things... It's a 'secure' page..

Onel Harrison Over a year ago

Try one more thing. Select and click the main menu, and when the menu items show up, select and click the element you want to click. Let me know if that works - @Algina

Shoaib Akhtar · Accepted Answer · 2017-03-22 07:01:15Z

1

Try with any of the below xpath. Sometimes automatically generated xpath does not work.

//a[contains(text(),'PInt')]
or
//div[@id='submenu1']//a[contains(text(),'PInt')]

Also I would suggest you to set some wait time before clicking on above link in case if above xpath does not work

answered Mar 22, 2017 at 7:01

Shoaib Akhtar

1,4035 gold badges20 silver badges46 bronze badges

2 Comments

Jainish Kapadia Over a year ago

Try this as well //div[@id='submenu1']//following::a[contains(text(),'PInt')]

Alg_D Over a year ago

unfortunately any of the solutions works. Not sure how but might be the page preventing some scraping?

A.Sherif · Accepted Answer · 2017-03-21 22:44:31Z

0

To find xPath in chrome:

Right click the element you want
Inspect, which will open the developer window and highlight the selected element.
Right click the highlighted element and choose copy > copy by xpath

Here is a list of all the different ways to locate an element Locating Elements

answered Mar 21, 2017 at 22:44

A.Sherif

1442 silver badges8 bronze badges

1 Comment

Alg_D Over a year ago

Yes, that's what I did and git the xPath: //[@id="submenu1"]/a[2] that somehow seems to be invalid

Collectives™ on Stack Overflow

Python selenium - get element by its xPath and access it

4 Answers 4

Comments

3 Comments

2 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

Comments

3 Comments

2 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related