Scraping HTML with selenium while clicking options [closed]

Question

Closed. This question needs debugging details. It is not currently accepting answers.

Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.

Closed 8 years ago.

Improve this question

I have got a script which I use to scrape off data from the websites using selenium.

    Sub Body_Building()
    Dim driver As New WebDriver, post As Object

    With driver
        .Start "chrome", "http://www.bodybuildingwarehouse.co.uk"
        .Get "/optimum-nutrition?limit=all"
    End With

    On Error Resume Next
    For Each post In driver.FindElementsByClass("grid-info")
        i = i + 1: Cells(i, 1) = post.FindElementByClass("product-name").Text
        Cells(i, 2) = post.FindElementByXPath(".//span[@class='regular-price']//span[@class='price']|.//p[@class='special-price']//span[@class='price']").Text
    Next post
End Sub

Would it be possible to scrape off data from this website using the same or similar technique so the outcome would be like below in the snapshot?

Please see the VBA working so it matched the desired outcome. Thank you SMth80

Sub optigura_scraper_v2()
    Dim driver As New ChromeDriver
    Dim elems As Object, post As Object

    driver.Get "https://www.optigura.com/uk/product/gold-standard-100-whey/"
    [A1:D1].Value = [{"Name","Flavor","Size","Price"}]

    Set elems = driver.FindElementsByXPath("//span[@class='img']/img")
    i = 2

    For n = 1 To elems.Count
        driver.FindElementsByXPath("//span[@class='img']/img")(n).Click
        driver.Wait 1000
        For Each post In driver.FindElementsByXPath("//div[@class='colright']//ul[@class='opt2']//label")
            Cells(i, 1) = driver.FindElementByXPath("//h1[@itemprop='name']").Text
            Cells(i, 2) = post.Text
            Cells(i, 3) = Split(driver.FindElementByXPath("//li[@class='active']//span[@class='img']/img").Attribute("alt"), "-")(1)
            Cells(i, 4) = driver.FindElementByXPath("//span[@class='price']").Text
            i = i + 1
        Next post
    Next n
End Sub

SIM · Accepted Answer · 2017-07-30 22:03:13Z

2

Check it out. This is certainly not the best technique. However, it will serve your purpose. Btw, the scraper will parse exactly how the data is displayed in that page.

Sub optigura_scraper()
    Dim driver As New ChromeDriver
    Dim elems As Object, post As Object

    driver.Get "https://www.optigura.com/uk/product/gold-standard-100-whey/"
    [A1:D1].Value = [{"Name","Price","Size","Flavor"}]

    Set elems = driver.FindElementsByXPath("//span[@class='img']/img")
    i = 2

    For N = 1 To elems.Count
        driver.FindElementsByXPath("//span[@class='img']/img")(N).Click
        driver.Wait 1000
        Cells(i, 1) = driver.FindElementByXPath("//h1[@itemprop='name']").Text
        Cells(i, 2) = driver.FindElementByXPath("//span[@class='price']").Text
        Cells(i, 3) = Split(driver.FindElementByXPath("//li[@class='active']//span[@class='img']/img").Attribute("alt"), "-")(1)
        For Each post In driver.FindElementsByXPath("//div[@class='colright']//ul[@class='opt2']//label")
            Cells(i, 4) = post.Text
            i = i + 1
        Next post
    Next N
End Sub

edited Jul 30, 2017 at 22:03

answered Jul 30, 2017 at 21:05

SIM

22.5k6 gold badges45 silver badges116 bronze badges

Sign up to request clarification or add additional context in comments.

9 Comments

Martin Over a year ago

I'm just thinking would it be possible to return size for each item? In general it looks very good.

Martin Over a year ago

Also what do you use to read web page source? F12 on Chrome or Ctrl+U option?

SIM Over a year ago

Clicking right button and selecting inspect element.

SIM Over a year ago

Hey, Martin. Check the update script. Size included. It was a bit difficult to deal with.

Martin Over a year ago

Thank you very much for helping me out with selenium. I have amended the VBA a little so it matched the desired outcome.

|

Collectives™ on Stack Overflow

Scraping HTML with selenium while clicking options [closed]

1 Answer 1

9 Comments

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

9 Comments

Linked

Related