C# Web Scraping: Reading dynamically load ajax content by web browser control

Question

This is how i tried. i load a web site into web browser control. the web site load more data when user scroll down.

This web site load data dynamically by ajax. i try to read all dynamic H3 tag loaded by ajax but my code did not work. not able to understand what i am missing in my code.

here is my code

private void BrowserTest_Load(object sender, EventArgs e)
{
    webBrowser1.Navigate("https://www.pinterest.com/pin/517210338432366716/");
}

private void webBrowser1_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
{
    if (webBrowser1.ReadyState == WebBrowserReadyState.Complete)
    {
        HtmlElement elm = webBrowser1.Document.GetElementById("h3"); // Get "abc" element by ID
        //Console.WriteLine("elm.InnerHtml(DocumentCompleted):" + elm.InnerHtml);
        if (elm != null)
        {
            elm.AttachEventHandler("onpropertychange", new EventHandler(handler));
        }
    }
}

private void handler(Object sender, EventArgs e)
{
    HtmlElement div = webBrowser1.Document.GetElementById("h3");
    if (div == null) return;
    String contentLoaded = div.InnerHtml;
}

private void btnScrollDown_Click(object sender, EventArgs e)
{
    if (webBrowser1.Document != null)
    {
        webBrowser1.Document.Window.ScrollTo(0, webBrowser1.Document.Body.ScrollRectangle.Height);
    }
}

Looking for suggestion how to achieve my goal. thanks

Noldor · Accepted Answer · 2020-07-10 21:10:42Z

1

I would choose a more different way for this;

scroll document to bottom
wait 100ms (or 200ms, 500ms, your choice..)
count total loaded grid elements in document
repeat this from step 1; until; if loaded grid elements count does not change for last 5 seconds. in that case it is probably end of all items, so get all grid elements in the document.

answered Jul 10, 2020 at 21:10

Noldor

1,1829 silver badges23 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Indi_Rain Over a year ago

thanks for reply. how to count total loaded grid elements in document which load by ajax. when data loaded by which i am not being able to capture programmatically....that is getting big obstacle. can you please tell me how to read data from web browser control whose content load dynamically by ajax?

Noldor Over a year ago

@Indi_Rain you can retrieve current updated elements in a document at any time. With my method, you don't need to know if ajax is loading or loaded new elements into page or not.. you just keep scrolling browser to the bottom, it will load data itself.. and you will be checking document regularly to see if new elements are being added or not.. after each scroll job, read elements from webBrowser.Document.GetElements(ById/TagName) .. like in a while loop or with Timers.. just keep counting elements after each scroll job, and check if they keep increasing or not..

Indi_Rain Over a year ago

if you get some time can you please run the code and tell me reason why i am not getting loaded data by ajax. i understood there is some problem in my code but i do not know what to fix. can you help me with code sample.

Collectives™ on Stack Overflow

C# Web Scraping: Reading dynamically load ajax content by web browser control

here is my code

1 Answer 1

3 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

here is my code

1 Answer 1

3 Comments

Your Answer

Sign up or log in

Post as a guest

Related