PHP: Receive and store large binary data

Question

My PHP script is receiving large data (100 - 500 MB) from a client. I want my PHP script run fast, without using too much memory.

To save traffic, I don't use Base64 or form data. I send binary data directly in a POST request.

The data consists of two parts: 2000 Bytes header, and the rest, that has to be stored as a file on the server.

$fle = file_get_contents("php://input",FALSE,NULL,2000);
file_put_contents("file.bin", $fle);

The problem is, that file_get_contents ignores the offset parameter, and reads the data from byte 0. Is there any better way to do it?

** I don't want to read the whole data and slice off the last N-2000 bytes, as I am afraid it would use too much memory.

First of all don't create another variable -$fle- file_put_contents("file.bin", file_get_contents("php://input",FALSE,NULL,2000)); — Accountant م
– Accountant م, Commented Jan 7, 2019 at 18:21
@Accountantم still reads everything into memory before flushing it to disk. — Sammitch
– Sammitch, Commented Jan 7, 2019 at 18:32
@Sammitch Yes I know, this will just prevent creating another copy of the data in the memory. great answer you post by the way, I did similar buffer before — Accountant م
– Accountant م, Commented Jan 7, 2019 at 18:39
@Accountantم ah, I misread your previous comment. It should be 1x with or without the variable assignment. — Sammitch
– Sammitch, Commented Jan 7, 2019 at 19:04

Sammitch · Accepted Answer · 2019-01-07 18:15:58Z

3

Use the lower-level file IO functions and read/write a little bit at a time.

$bufsz = 4096;

$fi = fopen("php://input", "rb");
$fo = fopen("file.bin", "wb");

fseek($fi, 2000);

while( $buf = fread($fi, $bufsz) ) {
  fwrite($fo, $buf);
}

fclose($fi);
fclose($fo);

This will read/write in 4kB chunks.

answered Jan 7, 2019 at 18:15

Sammitch

32.5k7 gold badges58 silver badges92 bronze badges

Sign up to request clarification or add additional context in comments.

5 Comments

Jaquarh Over a year ago

Things to note with this is your execution time - ensure you have set this in your php ini. Great answer with explanation though! Didn't know you could specify the size to read consistently, always used filesize() as the second arg. Great to know for future reference.

Ivan Kuckir Over a year ago

BTW. is there a simple way to read N bytes from $fi ? In my case, fread() often returns less than the value of the second parameter.

Sammitch Over a year ago

@IvanKuckir that should only ever happen once per file, at most, and only ever at the end of the file when it's not an even multiple of the buffer size. This is fine. Any other observed case is a different problem that needs to be addressed.

Ivan Kuckir Over a year ago

@Sammitch I ran your code with $bufsz = 1000000; (which is larger than the size of my file). The file is saved, but it never reads more than 8192 bytes at once.

Sammitch Over a year ago

@IvanKuckir php://input is a special stream and may have some peculiarities associated with it. Unless you have some special requirement to hold ~1MB of data in memory for some reason, a buffer size of 8192 bytes or smaller should work just fine.

Collectives™ on Stack Overflow

PHP: Receive and store large binary data

1 Answer 1

5 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

5 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related