I need to write a scraper in Java + Groovy..
I was wondering if something able to parse HTML documents and select the informations I need through simple CSS selectors (instead that going through the whole document tree and manually select what I need) exists? Something like Nokogiri for Ruby, just to give you the idea of what I need..
thanks in advance!
I do something like this by loading a page with Qt Webkit and including JQuery.
It's a hack but works well for my use case. I needed a solution that requires no configuration - just sudo apt-get install libqt4-webkit and you're ready to go.
If you can be backed by a browser (as in use a browser to render and create the pages), selenium would be perfect. this would have the added benefit of having full support for Ajax websites.
If not, something like webdriver would probably work.
I've only used Selenium.
I use Selenium RC + jQuery for screen scraping.
Example code: HERE
While I use PHP as the client, but you can implement it using any language you like (as long as it can talk to Selenium RC).
I have tried several CSS selector libraries before, but honestly, the best parser is your browser, Selenium RC approach is not fast but superb reliable.
