Php laravel memory limit error

Question

I am making currently migration from one database to another, project is on laravel so I am creating laravel command for this. I have one table with about 700000 records. I have created function with LIMIT and transactions to optimize query but still getting out of memory error from PHP. Here is my code:

ini_set('memory_limit', '750M');  // at beginning of file

$circuit_c = DB::connection('legacy')->select('SELECT COUNT(*) FROM tbl_info');
$count = (array) $circuit_c[0];
$counc = $count['COUNT(*)'];
$max =  1000;
$pages = ceil($counc / $max);

    for ($i = 1; $i < ($pages + 1); $i++) {
        $offset = (($i - 1) * $max);
        $start = ($offset == 0 ? 0 : ($offset + 1));
        $infos = DB::connection('legacy')->select('SELECT * from tbl_info LIMIT ' . $offset . ', ' . $max);
        DB::connection('mysql')->transaction(function() use ($infos) {
            foreach ($infos as $info) {
                $validator = Validator::make($data = (array) $info, Info::$rules);
                if ($validator->passes()) {
                    if ($info->record_type == 'C') {
                        $b_user_new = Info::create($data);
                        unset($b_user_new);

                    }
                }
                unset($info);
                unset($validator);
            }
        });
        unset($infos);
    }

Error is this:

user@lenovo /var/www/info $ php artisan migratedata
PHP Fatal error:  Allowed memory size of 786432000 bytes exhausted (tried to allocate 32 bytes) in /var/www/info/vendor/laravel/framework/src/Illuminate/Database/Grammar.php on line 75

Error is show after importing about 50000 records.

Then increase how much memory PHP is allowed to use... If you have run out of memory then you have simply run out; buy more memory, or find a more efficient way to migrate your data. — Sverri M. Olsen
– Sverri M. Olsen, Commented Apr 26, 2015 at 9:16
I need optimization answer, because I have already optimized my functions using LIMIT and OFFSET, may be exists some other efficient way. — Serghei
– Serghei, Commented Apr 26, 2015 at 9:34
Can you try doing this in chunks of 100 instead 1000, per transaction? — N.B.
– N.B., Commented Apr 26, 2015 at 10:21
How about within DB::connection('mysql')->transaction(function()..., instead of use($infos)` you simply move $infos = DB::connection... in the transaction closure? It looks like some references still exist so PHP isn't freeing any memory at all. — N.B.
– N.B., Commented Apr 26, 2015 at 11:47

Margus Pala · Accepted Answer · 2015-04-26 10:14:56Z

1

There is kind of a "memory leak" in here. You need to find out which of the variables is hogging all of this memory. Try this function to debug and see which variable keep on growing constantly

function sizeofvar($var) {
  $start_memory = memory_get_usage();
  $tmp = unserialize(serialize($var));
  return memory_get_usage() - $start_memory;
}

Once you know what variable is taking all the memory then you can start implementíng appropriate measures.

answered Apr 26, 2015 at 10:14

Margus Pala

8,7339 gold badges45 silver badges53 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

Serghei Over a year ago

I have tried but I cannot find memory leaks, variables are all time all the same size

I c:96 Count c:128 Circuit c:840 Count:464 Pages:96 Max:96 Start:96 Offset:96 Infos:7120136 Info:7488 Info data:7392 New Info:28312 PHP Fatal error:  Allowed memory size of 786432000 bytes exhausted (tried to allocate 5 bytes) in /var/www/info/app/commands/MigrateDataCommand.php on line 358

Margus Pala Over a year ago

What about sizeofvar(DB::connection('legacy')) ?

Serghei Over a year ago

Error: [PDOException] You cannot serialize or unserialize PDO instances

Serghei · Accepted Answer · 2015-04-26 13:38:53Z

1

Found the answer, laravel caches all queries, so just: DB::connection()->disableQueryLog();

answered Apr 26, 2015 at 13:38

Serghei

4226 silver badges19 bronze badges

Collectives™ on Stack Overflow

Php laravel memory limit error

2 Answers 2

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related