InnerText returns empty for a specific span class

Question

I am trying to retrieve regular (126,37€) and reduced (101,10€) price information from this website.

Simplified HTML code looks like this:

<div class="vw-productFeatures ">
  <ul class="feature-list -price-container">
    <li class="feature -price">
      <span class="value">126,37</span>
    </li>
  </ul>
  <ul class="feature-list vw-productVoucher">
    <li class="voucher-information">Mit Code
      <span class="voucher-reduced-price">101,10</span>
    </li>
  </ul>
</div>

So, I basically go step by step (div class -> ul class -> li class -> span class) and get the innerText at the end.

I am able to get the regular price, however, spanclass.innerText of reduced price returns empty.

This is the code I am working with:

Function getHTMLelemFromCol(HTMLColIn As MSHTML.IHTMLElementCollection, tagNameIn As String, classNameIn As String) As MSHTML.IHTMLElement
    Dim HTMLitem As MSHTML.IHTMLElement

    For Each HTMLitem In HTMLColIn
        If (HTMLitem.tagName = tagNameIn) Then
            If (HTMLitem.className = classNameIn) Then
                Set getHTMLelemFromCol = HTMLitem
                Exit For
            End If
        End If
    Next HTMLitem
End Function
Function getPrice(webSite As String, divClass As String, ulClass As String, liClass As String, spanClass As String) As String
    Dim XMLPage As New msxml2.XMLHTTP60
    Dim HTMLDoc As New MSHTML.HTMLDocument
    Dim HTMLitem As MSHTML.IHTMLElement
    Dim HTMLObjCol As MSHTML.IHTMLElementCollection

    XMLPage.Open "GET", webSite, False
    XMLPage.send
    HTMLDoc.body.innerHTML = XMLPage.responseText

    Set HTMLObjCol = HTMLDoc.getElementsByClassName(divClass)
    Set HTMLitem = getHTMLelemFromCol(HTMLObjCol, "DIV", divClass)          ' Find the div class we are interested in first
    Set HTMLitem = getHTMLelemFromCol(HTMLitem.Children, "UL", ulClass)     ' Find the ul class we are interested in
    Set HTMLitem = getHTMLelemFromCol(HTMLitem.Children, "LI", liClass)     ' Find the li class we are interested in
    Set HTMLitem = getHTMLelemFromCol(HTMLitem.Children, "SPAN", spanClass) ' Find the span class we are interested in

    getPrice = HTMLitem.innerText
End Function
Sub Run()
    Dim webSite As String, divClass As String, ulClass As String, liClass As String, spanClass As String, regularPrice As String, reducedPrice As String

    webSite = "https://www.rakuten.de/produkt/msi-b450-tomahawk-max-atx-mainboard-4x-ddr4-max-64gb-1x-dvi-d-1x-hdmi-14-1x-usb-c-31-2843843890"
    divClass = "vw-productFeatures "

    ' Get the regular price
    ulClass = "feature-list -price-container"
    liClass = "feature -price"
    spanClass = "value"
    regularPrice = getPrice(webSite, divClass, ulClass, liClass, spanClass)

    ' Get the reduced price
    ulClass = "feature-list vw-productVoucher -hide"
    liClass = "voucher-information"
    spanClass = "voucher-reduced-price"
    reducedPrice = getPrice(webSite, divClass, ulClass, liClass, spanClass)

    Debug.Print "Regular price: " & regularPrice
    Debug.Print "Reduced price: " & reducedPrice
End Sub

The output I am getting:

Regular price: 126,37
Reduced price:

Debugger shows that it is able to find the correct span class, but it does not have any attribute (including innerText) that has the price information.

How can I get the reduced price information?

I am not an expert at this, but I do not see a reason to include -hide in your ulClass variable. It seems like that would cause it to not match anything. — braX
– braX, Commented Feb 2, 2020 at 20:20
Actually, when I "inspect", ul class seems to be feature-list vw-productVoucher. However, there is no match then. When I look at the page source, and search for feature-list vw-productVoucher, I can see that it is feature-list vw-productVoucher -hide. — vitruvius
– vitruvius, Commented Feb 2, 2020 at 20:31

Ryan Wildry · Accepted Answer · 2020-02-13 04:12:22Z

Sometimes when much of the page's content is dependent on API calls, it is easier to use browser automation.

It's non-ideal from a performance perspective, but faster to get operational, and works in a pinch. The alternative approach is to monitor the web traffic between you and the server, and see if you can emulate the web requests to get the reduced price. This would be faster, but may take a bit of time to figure out how this works.

There are trade-offs for each approach to consider. Below is some Internet Explorer Automation code that is working for me to retrieve the data I believe you are after.

Declare Sub Sleep Lib "kernel32" (ByVal dwMilliseconds As Long)

Sub GetReducedPrice()
    Dim text As String

    With CreateObject("internetexplorer.application")
        .navigate "https://www.rakuten.de/produkt/msi-b450-tomahawk-max-atx-mainboard-4x-ddr4-max-64gb-1x-dvi-d-1x-hdmi-14-1x-usb-c-31-2843843890"
         Do While .Busy And .readyState <> 4: DoEvents: Loop
         Sleep 1000 ' wait a little bit too
         text = .document.querySelector(".voucher-reduced-price").innerText
        .Quit
    End With

    Debug.Print "the reduced price is: " & text
End Sub

Result is:

the reduced price is: 101,10

Sers · Accepted Answer · 2020-02-02 21:26:56Z

0

There's no -hide class for reduce price:

ulClass = "feature-list vw-productVoucher"

You can use simple selectors to get both prices with querySelector (example) instead of complex methods with unnecessary iterations.

regularPrice = HTMLDoc.querySelector(".-price .value").innerText
reducedPrice = HTMLDoc.querySelector(".voucher-reduced-price").innerText

Update: Vaucher is https://tags.tiqcdn.com/utag/rakuten/main/prod/utag.js here and calculated based on product_shop_id and dates.

edited Feb 2, 2020 at 21:26

answered Feb 2, 2020 at 20:28

Sers

12.3k2 gold badges14 silver badges33 bronze badges

2 Comments

vitruvius Over a year ago

Thanks for your answer. Did you run this code? I'm still getting an empty response for reduced price.

Sers Over a year ago

It's empty, reduced price coming from another place. Probably from JavaScript (tags.tiqcdn.com/utag/rakuten/main/prod/utag.js)

Collectives™ on Stack Overflow

InnerText returns empty for a specific span class

2 Answers 2

Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related