Scraping a dynamically loading website with php curl

Question

I am new to scraping and have scrapped two websites formally. But the problem appeared to me when I tried to scrape dynamic loading websites. When the website is rendered with JavaScript, I am unable to scrape the contents of the website then.

Is there any way I can scrape the contents of that website using php curl or any other client related to PHP?

This is what I have done so far :

$link = "https://www.glassdoor.com/Job/jobs.htm?suggestCount=0&suggestChosen=false&clickSource=searchBtn&typedKeyword=android+developer&sc.keyword=android+developer&locT=N&locId=192&jobType=";

$ch = curl_init();
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch,CURLOPT_URL,$link);
curl_setopt($ch,CURLOPT_RETURNTRANSFER,1);
curl_setopt($ch, CURLOPT_USERAGENT, "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US) AppleWebKit/525.13 (KHTML, like Gecko) Chrome/0.A.B.C Safari/525.13");
$data = curl_exec($ch);


$document = new DOMdocument();
libxml_use_internal_errors(true);
$document->loadHTML($data);
$elements = $document->getElementsByTagName("div");

foreach($elements as $element){
  	echo $element->nodeValue."<br>";;
}

Do some research please ... google.com/search?q=scrape+website+populated+by+javascript — C3roe
– C3roe, Commented Mar 1, 2018 at 12:22

Faraz Irfan · Accepted Answer · 2018-03-14 07:57:19Z

3

You need headless browser for this, you can use PHP Wrapper for PhantomJS , here is the link http://jonnnnyw.github.io/php-phantomjs/. This will solve your problem. It has following features:

Load webpages through the PhantomJS headless browser
View detailed response data including page content, headers, status code etc.
Handle redirects
View javascript console errors

Hope this helps.

answered Mar 14, 2018 at 7:57

Faraz Irfan

1,3263 gold badges10 silver badges17 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Scraping a dynamically loading website with php curl

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related