PHP cURL methods time out on some URLs, but command line always works

Question

When I attempt to use PHP's cURL methods for SOME URLs, it times out. When I use the commandline for the same URL, it works just fine.

I am using AWS and have a t2.medium box running the php-55 apache libraries from yum.

Here is my PHP code:

function curl($url) {
$ch = curl_init();
curl_setopt($ch, CURLOPT_URL, $url);
curl_setopt($ch, CURLOPT_AUTOREFERER, true);
curl_setopt($ch, CURLOPT_USERAGENT, 'Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36');
curl_setopt($ch, CURLOPT_HEADER, true);
curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);
curl_setopt($ch, CURLOPT_VERBOSE, true);
curl_setopt($ch, CURLOPT_FOLLOWLOCATION, true);
curl_setopt($ch, CURLOPT_MAXREDIRS, 2);
curl_setopt($ch, CURLOPT_HTTPHEADER, array(
    'Accept-Language: en-us'
));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 5);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);
curl_setopt($ch, CURLOPT_IPRESOLVE, CURL_IPRESOLVE_V4);
$fh = fopen('/home/ec2-user/curllog', 'w');
curl_setopt($ch, CURLOPT_STDERR, $fh);
$a = curl_exec($ch);
curl_close($ch);
fclose($fh);
$headers = explode("\n",$a);
var_dump($headers);
var_dump($a);
exit;

        return $result;
}

So here is call that works just fine:

curl('http://www.google.com');

And this returns the data for the homepage of google.

However, I try another URL:

curl('http://www.trulia.com/profile/agent-1391347/overview');

And I get this in the curllog:

[ec2-user@central Node]$ cat ../curllog
* Hostname was NOT found in DNS cache
*   Trying 23.0.160.99...
* Connected to www.trulia.com (23.0.160.99) port 80 (#0)
> GET /profile/agent-1391347/overview HTTP/1.1
User-Agent: Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36
Host: www.trulia.com
Accept: */*
Accept-Language: en-us

* Operation timed out after 10002 milliseconds with 0 bytes received
* Closing connection 0

If I run this from the command line:

curl -s www.trulia.com/profile/agent-1391347/overview

It IMMEDIATELY returns (within 1 second) with NO output. This is expected. However when I run this:

curl -sL www.trulia.com/profile/agent-1391347/overview

It returns the page properly, just as I would want.

So, what is wrong with my curl?

PHP 5.5.20

Here is the cURL bit from my phpinfo():

curl

cURL support => enabled
cURL Information => 7.38.0
Age => 3
Features
AsynchDNS => Yes
CharConv => No
Debug => No
GSS-Negotiate => No
IDN => Yes
IPv6 => Yes
krb4 => No
Largefile => Yes
libz => Yes
NTLM => Yes
NTLMWB => Yes
SPNEGO => Yes
SSL => Yes
SSPI => No
TLS-SRP => No
Protocols => dict, file, ftp, ftps, gopher, http, https, imap, imaps, ldap, ldaps, pop3, pop3s, rtsp, scp, sftp, smtp, smtps, telnet, tftp
Host => x86_64-redhat-linux-gnu
SSL Version => NSS/3.16.2 Basic ECC
ZLib Version => 1.2.7
libSSH Version => libssh2/1.4.2

Are you running the command line from the same machine as the php code? if not you may want to compare the curl version. — Danny Yeshurun
– Danny Yeshurun, Commented Jun 27, 2017 at 18:41
what's the output of terminal curl --version ? and what's the output of <?php var_dump(curl_version()); ? — hanshenrik
– hanshenrik, Commented Jun 27, 2017 at 23:18
IRC snippet: i.imgur.com/bOwxcS3.png - ... and the guy that answered, is the author of (lib-)curl — hanshenrik
– hanshenrik, Commented Jun 27, 2017 at 23:21
@hanshenrik < curl --version curl 7.51.0 (x86_64-redhat-linux-gnu) libcurl/7.51.0 NSS/3.28.4 zlib/1.2.8 libidn2/0.16 libpsl/0.6.2 (+libicu/50.1.2) libssh2/1.4.2 ...SNIP... php > var_dump(curl_version()); array(9) { ["version_number"]=> int(471808) ["age"]=> int(3) ["features"]=> int(2000797) ["ssl_version_number"]=> int(0) ["version"]=> string(6) "7.51.0" ["host"]=> string(23) "x86_64-redhat-linux-gnu" ["ssl_version"]=> string(10) "NSS/3.28.4" ["libz_version"]=> string(5) "1.2.8" ["protocols"]=> array(22) { SNIP } } — Shawn Tolidano
– Shawn Tolidano, Commented Jun 30, 2017 at 4:06
@ShawnTolidano that is php-cli. is php-cli where you have the problem? or do you experience the problem in an apache mod_php? or in a php-fcgi? if its the latter, does your php-cli and your php-whatever have the same version? — hanshenrik
– hanshenrik, Commented Jun 30, 2017 at 7:47

Sagar V · Accepted Answer · 2017-06-30 12:08:57Z

9

+50

I have checked your function curl() It seems fine. No need to change anything in the function. What should you need to do is just pass the URL as it is as parameter no need to change HTTPS to HTTP

curl('http://www.trulia.com/profile/agent-1391347/overview');

Reason:

You already told curl to don't verify the SSL

curl_setopt($ch, CURLOPT_SSL_VERIFYPEER, false);

Let me know if you need any explanation.

edited Jun 30, 2017 at 12:08

Sagar V

12.5k8 gold badges46 silver badges71 bronze badges

answered Jun 30, 2017 at 6:55

Sahil Patel

1,6402 gold badges17 silver badges35 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

The Unix Janitor Over a year ago

Ah, nothing like turning security off! :-)

hakre · Accepted Answer · 2017-06-30 20:24:42Z

The verbose output shows a clear timeout problem:

Operation timed out after 10002 milliseconds with 0 bytes received

This signals a problem with your network setup. These are harder to locate, this can be on your own end (e.g. in context of the webserver or the PHP executable) or on the other end. Both places are possible to a certain extend, however the server accepts both requests even if they have different request headers, so it is more likely that this is execution context related which is also how you generally describe it.

Check if there are any restrictions on security and other networking layers regarding performing those requests via PHP. E.g. try a different server image if you're not so into system administration and trouble-shooting. From what is shared in your question, this is hard to say what exactly causes your timeout.

Chris Stringer · Accepted Answer · 2015-02-06 18:02:25Z

4

Try increasing the timeout values in the following lines:

curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 5);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);

Those are pretty short timeout values - the CURLOPT_TIMEOUT specifically limits the entire execution time, try giving larger values:

curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 15);
curl_setopt($ch, CURLOPT_TIMEOUT, 30);

answered Feb 6, 2015 at 18:02

Chris Stringer

1351 silver badge6 bronze badges

1 Comment

Shawn Tolidano Over a year ago

It runs instantaneously from the command line - under 1 second. I should not need 10 seconds in PHP to do what takes 1 on the CLI. I did try it with double values though and it still times out. I ran on a different EC2 instance and it worked perfectly.

Sagar V · Accepted Answer · 2017-06-27 14:37:37Z

3

You have 2 VARIABLES

curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 5);
curl_setopt($ch, CURLOPT_TIMEOUT, 10);

The first one, CURLOPT_CONNECTTIMEOUT is maximum amount of time allowed to make connection to the server`

You can disable it by setting it to 0.

That is

curl_setopt($ch, CURLOPT_CONNECTTIMEOUT, 0);

But it is not a good method if you are in a production environment because it will never timeout.

Now CURLOPT_TIMEOUT

From PHP Documentation

The maximum number of seconds to allow cURL functions to execute.

Set it to some higher value

curl_setopt($ch, CURLOPT_TIMEOUT, 20); // 20 Seconds.

answered Jun 27, 2017 at 14:37

Sagar V

12.5k8 gold badges46 silver badges71 bronze badges

2 Comments

alzee Over a year ago

My particular problem, and why I put the bounty on it, isn't answered by the timeouts. Look at the first statement in the post -- the issue is that it times out only sometimes when called from PHP, but never when called from the command line -- even when used on the same URL both times.

tangxinfa Over a year ago

I measure from my pc, it takes 10+ seconds with curl command, most of the time used on data transfer. So it means we triggered the cdn's speed limit(by cloudfront.net for this example). You should figure out how to bypass the speed limit, or just set a larger CURLOPT_TIMEOUT

Collectives™ on Stack Overflow

PHP cURL methods time out on some URLs, but command line always works

4 Answers 4

1 Comment

Comments

1 Comment

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

4 Answers 4

1 Comment

Comments

1 Comment

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related