Can't figure out how to scrape Google's search results using PHP

Question

I am trying to scrape Google search results using PHP.

I tried using @file_get_contents(http://www.google.com/search?hl=en&q=test) but it does not work. It only works with http://www.google.com.

I tried using curl instead. Here's my function:

function my_fetch($url,$user_agent='Mozilla/4.0 (compatible; MSIE
5.01; Windows NT 5.0)')  { 
    $ch = curl_init(); 
    curl_setopt ($ch, CURLOPT_URL, $url); 
    curl_setopt ($ch, CURLOPT_USERAGENT, $user_agent); 
    curl_setopt ($ch, CURLOPT_HEADER, 0); 
    curl_setopt ($ch, CURLOPT_RETURNTRANSFER, 1); 
    curl_setopt ($ch, CURLOPT_REFERER, 'http://www.google.com/'); 
    $result = curl_exec ($ch); 
    curl_close ($ch); 
    return $result;  }

$googleContent = my_fetch("http://www.google.com/search?hl=en&q=test");
echo $googleContent;

The result is

302 Moved
The document has moved here.

With a link to here: http://www.google.com/sorry/?continue=http://www.google.com/search%3Fhl%3Den%26q%3Dtest

Is there any way to crawl the search results using PHP without having to learn the API?

Scraping Google search results is against their TOS. Use the Custom Search API instead. — serk
– serk, Commented Oct 24, 2011 at 4:06
The current search API is covered at code.google.com/apis/customsearch/v1/reference.html and needs an API key. As search companies ultimately want to make money, they don't make search result pages easy to scrape. — Phil Lello
– Phil Lello, Commented Oct 24, 2011 at 4:08
Using the Search API is not useful for getting accurate rankings and the amount of data is heavily restricted, even the quite expensive commercial sort is kinda useless for a larger amount of data. Regarding the TOS, you do not accept the TOS by accessing Google and you can reject it in a written statement if you accepted it before (like when using a Google account) not that it plays a role, if you do not cause trouble Google will not hunt you for scraping them. There is a opensource PHP project at scraping.compunect.com which scrapes Google reliable. I guess my answer comes too late:) — John
– John, Commented May 9, 2014 at 2:00

Peter O. · Accepted Answer · 2012-02-05 22:11:13Z

6

Your problem is Google does a redirect. You need to add

CURLOPT_FOLLOWLOCATION => true

edited Feb 5, 2012 at 22:11

Peter O.

33.1k14 gold badges86 silver badges97 bronze badges

answered Feb 5, 2012 at 16:56

Just Something

611 silver badge2 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Blake K Peterson · Accepted Answer · 2011-10-28 12:15:34Z

0

What are you trying to scrape? There are numerous ways of getting SERPS without breaking Google's TOS.

I've used RSS feeds from Search engines in the past - I think you can add a date filter so you don't end up with the same results reach time.

answered Oct 28, 2011 at 12:15

Blake K Peterson

4714 silver badges14 bronze badges

1 Comment

Tony Paternite Over a year ago

I am trying to scrape the ratings. I don't know any way of getting the rating data using the API or RSS feeds.

Collectives™ on Stack Overflow

Can't figure out how to scrape Google's search results using PHP

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related