0

In laravel, I have a query that searches thousands of rows in a database, and I trying to compile that into a CSV file. In attempt to reduce memory usage, I get 500 rows at a time and output the CSV.

    $callback = function () use ($query) {
      $file = fopen('php://output', 'w');

      $query->chunk(500, function ($rows) use ($file) {
        foreach ($rows as $key => $row) {
          fputcsv($file, array_map(...$rows...));
        }
        log_info("Memory used " . memory_get_usage());
      });

      fclose($file);
    };

    $headers = [ ... ];

    return response()->stream($callback, 200, $headers);

The actual query is a bit more complex, and involves getting related models which also need to be rehydrated. When I run this, it begins generating the CSV file and, after a while, runs out of memory. This is in my log

(59): Memory used 17208328;
(59): Memory used 25105328;
(59): Memory used 30601328;
...
(59): Memory used 127380496;
(59): Memory used 129352584;
(59): Memory used 131207672;
[2025-11-23 23:50:15] qa.ERROR: Allowed memory size of 134217728 bytes exhausted (tried to allocate 16384 bytes) 

What I Tried

I tried putting the following inside the chunk loop, hoping that it would free memory. It had no effect on the memory consumption.

        unset($rows);
        flush();
        gc_collect_cycles();
4
  • Since you are writing to the output it's possible you have buffered output on. Try running ob_end_clean() before you start streaming the response to see if that makes a difference Commented Nov 24 at 8:30
  • I added ob_end_clean() but no luck. I would think that flush() would release that memory Commented Nov 24 at 9:39
  • How much rows are in the CSV? Since you are writing to php://output, this writes to the output buffer (keyword being buffer), so if the output exceeds 128MB, it will trip the memory limit (but 128MB is a lot of text). Commented Nov 24 at 15:46
  • There are about 11k rows when it runs out of memory, but the file size is only 4mb, considerably less than the 128 mb memory limit. Also, it is streaming, so once that data is released on the wire, it shouldn't also still be in memory. Commented 2 days ago

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.