I want to make a java script crawler that gets a assignment from a php server and than index the page. but the sites I want to crawl are external and I cant reach them with javascript does anyone have a solution with a other assignment language or a javascript solution.
I thought of using greasemonkey to crawl a page when a new site is loaded.
You can't reach them because of browser security restrictions- i believe this is termed cross site scripting. This is the type of job for the PHP application, not the client.
If you need the client to do the work then request same origin policy bypass permission from the client. See here stackoverflow.com:can-i-disable-sop-same-origin-policy-on-any-browser-for-development
You can achieve the same thing with a java applet.
jquery.xdomainajax.js is capable of loading external page.
Here is a simple code to load webpage
$(document).ready(function(){
$('#test').load('http://abc.com', '', function(response, status, xhr) {
if (status == 'error') {
var msg = "Sorry but there was an error: ";
$(".content").html(msg + xhr.status + " " + xhr.statusText);
}
});
});
It can work with cross domain.
