cURL weird status codes when checking URL

Question

I'm checking for the presence of a xml site map on different URLs. If I supply a URL example.com/sitemap.xml, and it has a 301 to www.example.com/sitemap.xml, I get a 301 obviously. If www.example.com/sitemap.xml doesnt exist, I wont see the 404. So, if I get a 301, I execute another cURL to see if a 404 returns for www.example.com/sitemap.xml. But, for reason, I get random 404 and 303 status codes.

private function check_http_status($domain,$file){

        $url = $domain . "/" . $file;

        $curl = new Curl();

        $curl->url = $url;
        $curl->nobody = true;
        $curl->userAgent = 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.1) Gecko/20060601 Firefox/2.0.0.1 (Ubuntu-edgy)';
        $curl->execute();
        $retcode = $curl->httpCode();

        if ($retcode == 301 || $retcode == 302){

            $url = "www." . $domain . "/" . $file;

            $curl = new Curl();
            $curl->url = $url;
            $curl->nobody = true;
            $curl->userAgent = 'Mozilla/5.0 (X11; U; Linux i686; en-US; rv:1.8.1.1) Gecko/20060601 Firefox/2.0.0.1 (Ubuntu-edgy)';
            $curl->execute();
            $retcode = $curl->httpCode();

        }

        return $retcode;

    }

Can you provide the site you are checking?

Jake N
– Jake N

2012-11-29 15:42:25 +00:00
Commented Nov 29, 2012 at 15:42 — Jake N
– Jake N, Commented Nov 29, 2012 at 15:42
nytimes.com/sitemap.xml

madphp
– madphp

2012-11-29 15:44:39 +00:00
Commented Nov 29, 2012 at 15:44 — madphp
– madphp, Commented Nov 29, 2012 at 15:44

Kami · Accepted Answer · 2012-11-29 15:45:26Z

2

Have a look at the list of response codes returned - http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html.

Usually a web browser will automatically handle these, but as you are doing things manually with curl, you need to understand what each response means. The 301 or 302 means that you should use the alternative url supplied to access the resource. This may be a simple as addin www to the request but it also may be more complex as a redirect to a different domain altogather.

The 303 means that you are using a POST attempt to access the resource, and should use GET.

answered Nov 29, 2012 at 15:45

Kami

19.5k4 gold badges53 silver badges65 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

madphp Over a year ago

I may have figuired it out. If i just do followLocation, it will return the status code for the redirect location.

user1703809 · Accepted Answer · 2012-11-29 15:42:34Z

0

Well, when you receive a 301 or 302 you should use the location found in the response, not just assume another location and try that.

As you can see in this example, the response from the server contains the new location of the file. Use that for your next request: http://en.wikipedia.org/wiki/HTTP_301#Example

answered Nov 29, 2012 at 15:42

user1703809

3 Comments

madphp Over a year ago

You can assume the location for sitemap.xml. If its not there, I want to know.

madphp Over a year ago

actually, i take that back. I just asked a colleauge, and he said sitemap.xml can be called something else. Never occured to me. I suppose, I will insert possible locations into an array, but for each one, I would like to get a true status. I have no idea what a 303 is.

user1703809 Over a year ago

Don't do that (insert possible locations into an array). Just get the new location from the response. This way you will have less tries and be more efficient and your script will also behave according to the http protocol. Asumig the server returns the true location for the redirect, which it should.

youssefr · Accepted Answer · 2013-12-11 16:39:41Z

"followLocation" works very well. Here is how I implemented it:

$url = "http://www.YOURSITE.com//"; // Assign you url here.

$ch = curl_init(); // initialize curl.
curl_setopt($ch, CURLOPT_URL, $url); // Pass the URL as the option/target.
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); // 0 will print html. 1 does not.
curl_setopt($ch, CURLOPT_HEADER, 0); // Please curl, inlude the header in the output.
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1); // ..and yes, follow what the server sends as part of the HTTP header.

$response_data = curl_exec($ch); // execute curl with the target URL.
$http_header = curl_getinfo($ch); // Gets information about the last transfer i.e. our URL
// Print the URLs that are not returning 200 Found.
if($http_header['http_code'] != "200") {
    echo " <b> PAGE NOT FOUND => </b>"; print $http_header['http_code'];
}
// print $http_header['url']; // Print the URL sent back in the header. This will print the page to wich you were redirected.
print $url; // this will print the original URLs that you are trying to access

curl_close($ch); // we are done with curl; so let's close it.

Collectives™ on Stack Overflow

cURL weird status codes when checking URL

3 Answers 3

1 Comment

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related