5

I know there are already some similar questions here. But I do not want to build a browser in Java, I only want to see the source code fully generated (or "rendered"). As if I look at the generated DOM in the browser. Does anybody know a tool for that?

I had a look at Cobra and HtmlUnit, but they dont seem to be able to render more complex websites correctly. Especially if there are AJAX calls adding content to the site after it has loaded. I really need a tool that does the same as a browser does, without the actual display of it. Do I have to remote control a browser in the end?

Does anybody has experience with that?

A very similar question but without any satisfying answeres can be found here.

Improve this question
5
  • do you need to see rendered HTML or the source code? Commented Jan 31, 2012 at 16:16
  • I dont need to see the visual rendered page, I need to see the generated source code. Like firebug shows it when you look at the DOM. Commented Jan 31, 2012 at 16:23
  • Have you looked at Selenium? I don't know exactly how you could use it to do what you're asking, but there might be a backdoor api for one of the browser drivers. Commented Jan 31, 2012 at 16:29
  • If you need only the rendered HTML giving the URL, you can try docs.oracle.com/javase/tutorial/networking/urls/readingURL.html Commented Jan 31, 2012 at 17:07
  • i love to know also a solution for this without using a browser specially with the elements generated by javascript Commented Feb 1, 2012 at 10:54

3 Answers 3

Reset to default
1

I don't believe that a library exists that does scraping of the asynchronous calls after the page is loaded.

My recommendation is:

  1. Get the HTML of a page using Cobra or a similar library.
  2. Parse the source for AJAX requests. (for example, the ajax call will have a URL parameter and a "data" JSON string you can use for the request)
  3. For each AJAX call, make another request to the URL parameter you captured.
  4. Append the result from each AJAX call to the source of your HTML from the original page.

It's not a perfect solution and it will not help you in the scenarios that require the user to trigger an event. Also your code for capturing the URLs for the AJAX events will differ depending on what javascript library the website is using to make its async calls.

Hope that helps.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

I already do that in many cases, but its quite inconvenient in more complex cases. Thanks anyway!
0

Selenium do some what similar to this. You need to install selenium remote control on your machine. Then you can pass url request to the selenium. Selenium will open a browser a render the html/dhtml page metioned in the url. After that you can get the entire dom by querying to the selenium. you can do all these thing by coding

http://seleniumhq.org/ please note: You need to install either slenium webdriver or selenium remote control, not selenium ide.

Improve this answer

9 Comments

Thanks a lot for that hint, that might indeed be the solution in the end. But I would prefer not to actually launch a browser. It will probably be too slow... I need to evaluate it for a while.
Launching browser is really weird.May be you can use some browser engines instead of browser. But in that case, i think you can't use selenium which may increase the over head. I think it is not easy to generate the entire dom by reading from the url. There will be stuff like dynamic manupulation of done in the body onload or aftet the ajax requests.
Yes. I also just realised from another post stackoverflow.com/a/9088811/518587 that selenium supports "headless" processing, but then uses HtmlUnit again. And HtmlUnit does a good job, but seems to be slow and sometimes not working as well as the common browsers.
selenium RC(remote control) is typicaly very slow. If your target is webcrawling, then this is not the right way. If getting the rendered Dom is the core of your project, then you could go from some javascript rendering engines.
Yes, I need the speed. Do you know any javascript rendering engines that integrate into java well?
|
0

I have to answer this myself... In the end the best solution I found was actually HtmlUnit. It is just too slow for my needs. So I built my own tool, that of course needs manual setup to call the required links. But thus it does not have to wait for any js timeouts or alike, but parses the requested information from the page and does the desired calls. Its a lot of manual work, but it looks like there is no other solution that works fast enough.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.