PHP Curl gets 403 error, but browser from same machine can request page?

Question

I've got this script working with generally no problems. I say generally, because while it retrieves pages from CNN.com, allrecipes.com, reddit.com, etc - when I point it towards at least one URL (foxnews.com), I get a 403 error instead.

As you can see, I've set the user agent to the same as my machine's browser (that was necessitated by sending a request to Facebook's homepage, which returned a message that the browser wasn't supported).

So, basically wondering what step(s) I need to take to have as many sites as possible recognize the CURL request as coming from a real, actual browser, rather than 403'ing it.

    $ch = curl_init();
    $timeout = 5;
    curl_setopt($ch, CURLOPT_URL, $this->url);
    curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
    curl_setopt($ch, CURLOPT_HEADER, 1);
    curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
    curl_setopt($ch, CURLOPT_USERAGENT,'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_3) AppleWebKit/602.4.8 (KHTML, like Gecko) Version/10.0.3 Safari/602.4.8');
    curl_setopt($ch, CURLOPT_FRESH_CONNECT, 1);
    curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);

Obsidian Age · Accepted Answer · 2017-03-20 23:18:28Z

2

Fox News appears to be blocking access to their website from any request passing a USERAGENT. Simply removing the USERAGENT string works fine for me:

$ch = curl_init();
$timeout = 5;
curl_setopt($ch, CURLOPT_URL, $this->url);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ch, CURLOPT_HEADER, 1);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, $timeout);
curl_setopt($ch, CURLOPT_FRESH_CONNECT, 1);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1);

Hope this helps! :)

answered Mar 20, 2017 at 23:18

Obsidian Age

42.6k11 gold badges54 silver badges76 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Lucas Krupinski Over a year ago

Thanks! I had figured setting the USERAGENT to a real browser string wouldn't cause an issue like that, live and learn I guess :)

Lucas Krupinski Over a year ago

Still getting wierd issues... Fox works, but NYTimes needs cookies, so now I'm setting cookies. I figured I would go back to fox with a user agent (like a browser) and accepting cookies, but that doesn't resolve it. I'm really curious about how CURL can be made to look like a real live browser. Metatags also seem to be less consistent without a user agent, but I can't speak to that factually, just speculating.

Collectives™ on Stack Overflow

PHP Curl gets 403 error, but browser from same machine can request page?

1 Answer 1

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related