How can I parse dynamic content from a web page?

Question

I try to get a proxy list from this url:

This would be cool, but port number is dynamic JavaScript content. How can I get JavaScript-generated content from this page? I have jsoup and djNativeSwing but I want do this in background thread.

JWebBrowser webBrowser = new JWebBrowser();
webBrowser.navigate("http://spys.ru/en/free-proxy-list/");
System.out.println(webBrowser.getHTMLContent());

this code returns a Null result. Help please.

check out htmlUnit I think it is the one thing that you need. Thanks. — Dhruvenkumar Shah
– Dhruvenkumar Shah, Commented Aug 23, 2012 at 17:57
i think this is not for me. My code must be simple and crossplatform. As i know HTMLUnit - must have SYSTEM path - it's trouble — Trump
– Trump, Commented Aug 23, 2012 at 18:13
No you do not need system path. I created web crawler using HTMLUnit which works on http protocol. so yes it does work properly. Let me know if you want more help. I can write a formal answer if you want. — Dhruvenkumar Shah
– Dhruvenkumar Shah, Commented Aug 23, 2012 at 18:23
I'm try to use this code: final WebClient webClient = new WebClient(); final HtmlPage page = webClient.getPage("spys.ru/free-proxy-list1/RU/"); System.out.println(page.toString()); — Trump
– Trump, Commented Aug 23, 2012 at 18:31

Henrik Karlsson · Accepted Answer · 2012-08-23 18:02:14Z

2

The webbrowser hasn't finnished loading when you call the getHtmlContent() method. Use something like this instead:

JWebBrowser webBrowser = new JWebBrowser();
webBrowser.navigate("http://spys.ru/en/free-proxy-list/");
webBrowser.addWebBrowserListener(new WebBrowserListener(){
   public void loadingProgressChanged(WebBrowserEvent e){
       if(e.getWebBrowser().getLoadingProgress()==100)
            System.out.println(webBrowser.getHTMLContent());
   }
}
/* Note: I wrote this in the comment field without any testing,
   you probably have to make the webBrowser final. */

JavaDocs is your friend!

answered Aug 23, 2012 at 18:02

Henrik Karlsson

5,7634 gold badges28 silver badges43 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Trump Over a year ago

Thanks for fast replay, but i nothig to get from this code. And how to not building all abstrcat class for event.

Henrik Karlsson Over a year ago

2 secs, I'll try to get the lib and test it out

Henrik Karlsson Over a year ago

Sorry, I'm not familiar enough with the library. Even if I use a timer to wait for a long period of time, webBrowser.getHTMLContent() returns null for some reason

Trump Over a year ago

Do yo have any ideas? I'm try to use this example to and so it is

Collectives™ on Stack Overflow

How can I parse dynamic content from a web page?

1 Answer 1

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related