VBA Selenium FindElementByXPath doesnt find element

Question

I have written a VBA which opens a web link using selenium chrome web driver to scrape of data and I got several problems with which I need your advice on guys.

Code example and outcome 1: On error actived

    Sub test_supplements_store()
    Dim driver As New ChromeDriver
    Dim post As Object

    i = 1

    driver.Get "https://www.thesupplementstore.co.uk/brands/optimum_nutrition?page=4"
On Error Resume Next
    For Each post In driver.FindElementsByClass("desc")
        Cells(i, 1) = post.FindElementByTag("a").Attribute("title")
        Cells(i, 2) = Trim(Split(post.FindElementByClass("size").Text, ":")(1))
        Cells(i, 3) = post.FindElementByXPath(".//span[@class='now']//span[@class='pricetype-purchase-unit multi-price']//span[@class='blu-price blu-price-initialised']").Text
        Cells(i, 4) = post.FindElementByTag("a").Attribute("href")
        i = i + 1
    Next post
End Sub

Code example and outcome 2: On error deactivated

Sub test_supplements_store()
    Dim driver As New ChromeDriver
    Dim post As Object

    i = 1

    driver.Get "https://www.thesupplementstore.co.uk/brands/optimum_nutrition?page=4"
'On Error Resume Next
    For Each post In driver.FindElementsByClass("desc")
        Cells(i, 1) = post.FindElementByTag("a").Attribute("title")
        Cells(i, 2) = Trim(Split(post.FindElementByClass("size").Text, ":")(1))
        Cells(i, 3) = post.FindElementByXPath(".//span[@class='now']//span[@class='pricetype-purchase-unit multi-price']//span[@class='blu-price blu-price-initialised']").Text
        Cells(i, 4) = post.FindElementByTag("a").Attribute("href")
        i = i + 1
    Next post
End Sub

Code example and outcome 3: On error activated

Sub test_supplements_store()
    Dim driver As New ChromeDriver
    Dim post As Object

    i = 1

    driver.Get "https://www.thesupplementstore.co.uk/brands/optimum_nutrition"
On Error Resume Next
    For Each post In driver.FindElementsByClass("desc")
        Cells(i, 1) = post.FindElementByTag("a").Attribute("title")
        Cells(i, 2) = Trim(Split(post.FindElementByClass("size").Text, ":")(1))
        Cells(i, 3) = post.FindElementByXPath(".//span[@class='now']//span[@class='pricetype-purchase-unit multi-price']//span[@class='blu-price blu-price-initialised']").Text
        Cells(i, 4) = post.FindElementByTag("a").Attribute("href")
        i = i + 1
    Next post
End Sub

First example returns all of the 74 items from the website apart from price but in a very long period of time about two minutes.

Second example only returns title into the first cell of the sheet and pops the error out.

Third example returns only 21 but misses to return price of those items that do not have now label. Script runs very quickly, under 10 seconds.

Please advice on how to return all 74 items back together with title, size, price, href.

I'm not sure what you mean by that as the error snapshot is attached to the second example. First and third examples do not return any errors back. — Martin
– Martin, Commented Aug 7, 2017 at 7:45
Ok thanks. I haven't worked on VB, but this is the approach i have used to overcome staleness in java . stackoverflow.com/questions/45434381/… — santhosh kumar
– santhosh kumar, Commented Aug 7, 2017 at 8:01

SIM · Accepted Answer · 2017-08-07 10:11:01Z

1

The page you are dealing with has got lay-loading method applied. This is because all items don't load at a time; rather, it loads the rest when you scroll downmost. I used a small javascript function within the code and it solves the issue. I hope this is what the result you were looking for.

Sub test_supplements_store()
    Dim driver As New ChromeDriver
    Dim post As Object

    driver.Get "https://www.thesupplementstore.co.uk/brands/optimum_nutrition"
    On Error Resume Next

    Do While EndofPage = False
        PrevPageHeight = CurrentPageHeight
        CurrentPageHeight = driver.ExecuteScript("window.scrollTo(0, document.body.scrollHeight);var CurrentPageHeight=document.body.scrollHeight;return CurrentPageHeight;")
        driver.Wait 3000
        If PrevPageHeight = CurrentPageHeight Then
            EndofPage = True
        End If
    Loop

    For Each post In driver.FindElementsByXPath("//li[contains(@class,'prod')]")
        i = i + 1: Cells(i, 1) = post.FindElementByXPath(".//a").Attribute("title")
        Cells(i, 2) = Split(post.FindElementByXPath(".//p[@class='size']").Text, ": ")(1)
        Cells(i, 3) = post.FindElementByXPath(".//p[@class='price']//span[@class='now']//span|.//p[@class='price']//span[@class='dynamictype-single']").Text
        Cells(i, 4) = post.FindElementByXPath(".//a").Attribute("href")
    Next post
End Sub

edited Aug 7, 2017 at 10:11

answered Aug 7, 2017 at 7:58

SIM

22.5k6 gold badges45 silver badges116 bronze badges

Sign up to request clarification or add additional context in comments.

10 Comments

SIM Over a year ago

You have another requirement that I didn't notice. Using xpath will solve the price issue.

Martin Over a year ago

Unfortunately your code returns only prices for 21 first items of the page. Also I'm not sure how to return together regular and new price of the item.

SIM Over a year ago

I didn't tune up your price part. I tried to get all 74 items.

SIM Over a year ago

Just a moment. Get back to you.

SIM Over a year ago

Just follow the link. Very descriptive for a beginner like me and i think you will also find it useful. Btw, don't get lost to see some complicated stuffs cause you might never have any necessity to use them. zvon.org/comp/r/tut-XPath_1.html

|

Collectives™ on Stack Overflow

VBA Selenium FindElementByXPath doesnt find element

1 Answer 1

10 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

10 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related