php-java-bridge-users Mailing List for PHP/Java Bridge

> In any case, this isn't due to bugs in PHP or the PHP/Java Bridge, but
> due to your misunderstanding of how HTTP chunked encoding works and what
> fgets() and feof() do.

I comfortably admit I am new to downloading very large files in PHP, and 
I'm trying to learn.

I found an approach that worked...mostly. But it caused PHP to crash 
under JavaBridge and Tomcat. An experienced Java programmer saw the logs 
and said it was due to a class error. I contacted this mailing list and 
explained what was happening and was told that it was down to a feature 
not yet supported in JavaBridge. Now I'm being told it's not down to that.

I don't have any understanding of HTTP chunked encoding works. I'm not 
using this header in my code and your references to it were the first 
that I had seen. I'll read up on this and see how I can use it given 
what you say. And looking at the implementation of chunked encoding will 
give me more information on how fgets() and feof() work generally, and 
in the context of chunked encoding.

Again, many thanks for helping out. It is appreciated.


Mark...

On Sun, 2010-02-07 at 15:19 +0700,
php...@li... wrote:
> I'm just trying to download the response in a way 

How else do you want to download the files efficiently, if not via
Transfer-Encoding: Chunked?


> The downloads only fail 'sometimes', so I don't understand how some of
> the explanations 

Given your code, the possibility that PHP reads "over the edge" depends
on the file size. If the file is large it is more likely that this will
happen.

In any case, this isn't due to bugs in PHP or the PHP/Java Bridge, but
due to your misunderstanding of how HTTP chunked encoding works and what
fgets() and feof() do.


Regards,
Jost Bökemeier

> If the data you receive is encoded in HTTP/1.1 chunks

I don't believe it is, but I don't know a lot about these things, so I 
might be wrong.

There's nothing special happening at the server side. It's just a GET 
request and response. I'm just trying to download the response in a way 
that doesn't cause me memory problems with files that can be 100MBs in size.

At present, the code below is producing an adequate result, ie: it's not 
failed in testing yet. The downloads only fail 'sometimes', so I don't 
understand how some of the explanations offer here fit to what's 
happening. The retry improves the chance of the downloading completing. 
If a file does not complete after the retries, the client handles this, 
and the downloads are retried again later.

Thanks thanks for everyone's advice though. Some of it is over my head 
as a PHP newbie, but I'm understanding and learning other parts of what 
is being said.

Cheers,


Mark...


           $download_attempt = 1;
           do {
             $fs = fsockopen($host, 80, $errno, $errstr, 30);

             if (!$fs) {
               $this->writeDebugInfo("FAILED ", $errstr . '(' . $errno . 
')');
             } else {
               $out = "GET $file HTTP/1.1\r\n";
               $out .= "Host: $host\r\n";
               $out .= "Connection: Close\r\n\r\n";
               fwrite($fs, $out);

               $fm = fopen ($temp_file_name, "w");
               stream_set_timeout($fs, 30);

               while(!feof($fs) && ($debug = fgets($fs)) != "\r\n" ); // 
ignore headers

               while(!feof($fs)) {
                 $contents = fgets($fs, 4096); // Chunk download
                 fwrite($fm, $contents);
                 $info = stream_get_meta_data($fs);
                 if ($info['timed_out']) {
                   break;
                 }
               }
               fclose($fm);
               fclose($fs);

               if ($info['timed_out']) {
                 // Delete temp file if fails
                 unlink($temp_file_name);
                 $this->writeDebugInfo("FAILED on attempt " . 
$download_attempt . " - Connection timed out: ", $temp_file_name);
                 $download_attempt++;
                 if ($download_attempt < 5) {
                   $this->writeDebugInfo("RETRYING: ", $temp_file_name);
                 }
               } else {
                 // Move temp file if succeeds
                 $media_file_name = str_replace('temp/', 'media/', 
$temp_file_name);
                 rename($temp_file_name, $media_file_name);
                 $this->writeDebugInfo("SUCCESS: ", $media_file_name);
               }
             }
           } while ($download_attempt < 5 && $info['timed_out']);

Hi Peter,

> The apache server sends garbage if you send http 1.0 but expect http
> 1.1.

are we talking about this apache feature which can stream bytes to the
client?

This feature is quite useful on C level, you can simply read() the bytes
as apache generates them. But it is difficult to handle in PHP or Bea
WebLogic for example, because their network implementations use a buffer
which reads data eagerly. 

If the PHP network buffer contains 4 bytes and fread() requests 20
bytes, PHP tries to read 16 bytes from the network, instead of simply
returning the 4 bytes available so far. I don't know why PHP and other
JEE servers implement their network buffers this way. It might be more
efficient for normal HTTP connections, but it is bad for data packets
which don't have an explicit length; in PHP you must call fread($sock,
1), if you don't want to run into a dead lock. Or you must unblock the
stream and call stream_select() for each packet. Very low tech...


Regards,
Jost Bökemeier

Hello!

The apache server sends garbage if you send http 1.0 but expect http 1.1.

The following code will work:

$sock = fsockopen(...);
fwrite($sock, "GET /foo.php HTTP/1.1\r\nHost: localhost\r\n\r\n");
while(fgets($sock)!="\r\n");
while($len=hexdec(fgets($sock))) {
  for($str=fread($sock, $len); $len>strlen($str); $str.=fread($sock, $len-strlen($str)));
echo $str;
}

Peter