get a browser rendered html+javascript

Question

I need a comandline tool (or Javascript/PHP, but i think commandline is the one way) for render and get the rendered content of URL, but the important its I need to renderer the Javascript not only the CSS/Html/images.

For example command like: "renderengine http://www.google.es outputfile.html" and the content of the web (parsed html and javascript executed) isa saved in outputfile.html.

I need this because i need to take the result of a full javascript website like grooveshark, the site load all using javascript/ajax and the crawlers dont find nothing, only basic HTML empty template (because is loaded after using ajax/javscript)

Exists any browser engine for linux with support to Javascript (for example V8) that output the result for save in files?

I'm not sure 'render' is the word I would use here if you want to save it as html, as render is more associated with taking code and outputting pictures or sounds. What you want is more related to saving the temporary modifications of the html/css/js state then a render of anything. — Jules Gagnon-Marchand
– Jules Gagnon-Marchand, Commented Sep 10, 2013 at 13:32

DisappointedByUnaccountableMod · Accepted Answer · 2021-02-15 10:29:22Z

10

Selenium : very complete solution with bindings in many languages
puppeteer : headless Chrome API, usable in NodeJS or as a command-line tool
HTtrack : command-line tool
Apache Notch & webmagic : open source Java web crawlers
pholcus : "distributed & high concurrency" web crawler written in Go
Xvfb a display server implementing the X11 display server protocol, without showing any screen output. I have used it successfully with Travis CI and Protractor as an example. Alternative: XDummy
~~PhantomJS (first suggested by nvuono) : can export the rendered page as non-HTML (pdf, png...).~~ PhantomJS development is suspended until further notice (more details). Closely related: SlimerJS, CasperJS

And there are many Python web scraping libraries:

edited Feb 15, 2021 at 10:29

DisappointedByUnaccountableMod

6,8444 gold badges21 silver badges23 bronze badges

answered Jan 10, 2014 at 19:31

Lucas Cimon

2,0633 gold badges27 silver badges38 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

J.T Over a year ago

Hi there from 2018. Are there any new tools available? PhantomJS website is not accessible, Xvfb last update 2010 - looks outdated. Thanks!

Lucas Cimon Over a year ago

@J.T. I updated the links to reflect a little more the current options in 2018. PhantomJS is still used a lot, I think their website is just temporarily down

nvuono · Accepted Answer · 2013-09-10 13:27:28Z

6

Try phantomjs from www.phantomjs.org and you can easily modify the included rasterize.js to export the rendered HTML. It's based on webkit and does full evaluation of your target site's javascript, allowing you to adjust timeouts or execute your own code first if you wish. I personally use it to save hardcopy HTML file version of fully-rendered knockout.js templates.

It executes javascript so I just did something like this and saved the console output to a file:

var markup = page.evaluate(function(){return document.documentElement.innerHTML;});
console.log(markup);
phantom.exit();

answered Sep 10, 2013 at 13:27

nvuono

3,39330 silver badges28 bronze badges

2 Comments

Zenth Over a year ago

Im try it, and work perfect!! var page = require('webpage').create(); page.open('page.es', function (status) { if (status !== 'success') { console.log('Unable to load the address!'); phantom.exit(); } else { window.setTimeout(function () { page.viewportSize = { width: 1000, height: 800 }; console.log(page.content); phantom.exit(); }, 400); } }); The timeout its need because the ajax load after certain second animation, but, its perfect!

Gilles Quénot Over a year ago

CasperJS is a suspended/dead project

Collectives™ on Stack Overflow

get a browser rendered html+javascript

2 Answers 2

2 Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related