I scrape the google ranks with this script:
<?php
include("phpQuery-onefile.php");
$url = "https://www.google.com/search?hl=en&output=search&start=1&q=shoes";
$ch = curl_init($url);
curl_setopt($ch,CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$page = curl_exec($ch);
curl_close($ch);
$doc = phpQuery::newDocument($page);
echo $doc['#ires cite'];
?>I get the URL addresses from the Google Snippet (the one under the title in green). He used a library called phpQuery that automates the process. I search for all the "cites" within the Div with the id #ires.
This works perfect the problem starts with the mobile version:
<?php
include("phpQuery-onefile.php");
$url = "https://www.google.com/search?hl=en&output=search&start=1&q=shoes";
$ch = curl_init($url);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Linux; Android 7.1; Mi A1 Build/N2G47H) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/58.0.3029.83 Mobile Safari/537.36');
curl_setopt($ch,CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$page = curl_exec($ch);
curl_close($ch);
$doc = phpQuery::newDocument($page);
echo $doc['#ires cite'];
?>I add the Android agent to the Curl and Google shows the mobile version that has a different styles. Below the title does not appear the address, but the address appears in a similar breadcrumb format.
I can check the positions but not store the complete URL So I would have to get it from the href.
That's my problem, I can't get the complete Url, someone can help me?
I leave a piece of source code of the page of Google mobile version. (snippet)
Thanks in advance
<div>
<a class="C8nzq BmP5tf" href="/url?sa=t&source=web&rct=j&url=https://m.6pm.com/shoes&ved=2ahUKEwjIn8mcosDgAhXxwVkKHT1gDF04ARAWMAB6BAgFEAE&usg=AOvVaw1j59lxT1UHTjMA9CBFqgkQ" oncontextmenu="google.ctpacw.cm(this)">
<div aria-level="3" role="heading" class="MUxGbd v0nnCb">Shoes on Sale | 6pm</div>
<div class="zbELhe MUxGbd lyLwlc aLF0Z">
<span class="qzEoUe">https://m.6pm.com › shoes</span>
</div>
</a>
</div>