Web crawler to fetch ajax based link without browser

Question

Is there a simple way to execute ajax and retrieve ajax results without doing the old way with an automate web browser or without external library ? Is there some advance algorithm that it does this ?

arachnode.net · Accepted Answer · 2013-02-28 20:17:47Z

2

There isn't. The JS code needs to be compiled locally and then executed locally.

Code: https://svn.arachnode.net/svn/arachnodenet/trunk/Renderer/ UN/PW : Public/Public

You can get the path from an SVN client, remove the Arachnode.* references and have a FAST JavaScript interpreter. It uses mshtml.dll, which powers AxShDocVw.dll which powers the .NET WebBrowser control and is about 6 times faster than the .NET control.

https://svn.arachnode.net/svn/arachnodenet/trunk/Renderer/HtmlRenderer.cs is the file to peruse first.

Thanks, Mike

answered Feb 28, 2013 at 20:17

arachnode.net

7915 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

OakNinja · Accepted Answer · 2013-02-16 08:33:09Z

0

I dont know of any way to do it without any external library, but Htmlunit is great (written in java) and there is a c# wrapper you can try here: https://github.com/HtmlUnit/NHtmlUnit

answered Feb 16, 2013 at 8:33

OakNinja

2,34618 silver badges20 bronze badges

Comments

abhinav · Accepted Answer · 2013-02-21 17:57:21Z

0

Try nutch. In general all content which is not restricted by robots.txt, and does not require any input from user, is crawlable by crawlers.

answered Feb 21, 2013 at 17:57

abhinav

1,2821 gold badge11 silver badges27 bronze badges

Collectives™ on Stack Overflow

Web crawler to fetch ajax based link without browser

3 Answers 3

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related