PHP memory limit not reached but OOM killer kills php script

Question

I'm executing long running php script (PHP 7.3) which is parsing XML's in a loop and adds data to MySQL. I've set memory limit to 512MB with ini_set('memory_limit', '512M') inside the script. Problem is that after parsing about half of xml's OOM killer kills php with error:

[Mon May 6 16:52:09 2019] Out of memory: Kill process 12103 (php) score 704 or sacrifice child
[Mon May 6 16:52:09 2019] Killed process 12103 (php) total-vm:12540924kB, anon-rss:12268740kB, file-rss:3408kB, shmem-rss:0kB

I've tried debugging the code with memory_get_usage and php_meminfo. They both show that script do not exceed 100MB of memory at start and end of each loop (xml's have the same size). I'm already unsetting all possible vars at end of each loop.

It looks like PHP used 12.5GB of RAM inspite of the 0.5GB memory limit in the script. I'm expecting PHP to throw a fatal memory exhausted error if memory limit is reached but it never happens.

Any ideas how can I debug this problem?

You may be better off looking into how to optimise the loading of the data - there are ways of loading XML as a stream if necessary, or how you insert the data. — Nigel Ren
– Nigel Ren, Commented May 8, 2019 at 6:33

Andrei Nikolaenko · Accepted Answer · 2022-08-13 18:51:35Z

1

I have recently met this problem and the trick was to allow PHP as little memory as possible, say 8MB. Otherwise the system triggered an out-of-memory error before PHP did. It did not provide any info so I did not know which part of the script was causing it.

But with 8MB memory limit I got a PHP exception with line number and additional info. It was trying to allocate a rather big chunk of memory (about 200k) whereas nothing in my script was demanding it.

Following the line number it became immediately obvious that one of the functions went recursive thus causing infinite memory consumption.

edited Aug 13, 2022 at 18:51

answered Dec 27, 2021 at 21:41

Andrei Nikolaenko

1,0741 gold badge7 silver badges15 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Zoe - Save the data dump · Accepted Answer · 2019-05-10 14:47:05Z

0

I have worked importing large xml files, and the require more memory that most people expect. is not only the memory use for the file, but related variables and processes, as @NigelRen suggested, the best way to handle large xml is reading the file in parts. Here is a simple example that I hope can give you an idea on how to do this.

    $reader = new \XMLReader(); 
    //https://www.php.net/manual/en/book.xmlreader.php

    // path to the file, the LIBXML_NOCDATA will help if you have CDATA in your 
    // content
    $reader->open($xmlPath, 'ISO-8859-1', LIBXML_NOCDATA); 

    while ($reader->read()) {
        if ($reader->nodeType == XMLReader::ELEMENT) {

                try {

                    $xmlNode = new \SimpleXMLElement($reader->readOuterXml());
                    // do what ever you want

                } catch (\Throwable $th) {
                  // hanlde error
               }
        }
    }

edited May 10, 2019 at 14:47

Zoe - Save the data dump

28.4k22 gold badges130 silver badges163 bronze badges

answered May 10, 2019 at 14:46

Eli

9829 silver badges21 bronze badges

2 Comments

Ziom Ziomalski Over a year ago

I'm already splitting one large XML into many smaller ones. My script is looping every part and parsing it. The problem is when I have really large XML (1.5GB) which is splitted into 52 parts. Server runs out of memory after parsing about half of the parts. The main questions are: 1. why memory limit exhausted error is not triggered? 2. what exactly is causing server RAM to exhaust?

Eli Over a year ago

If I understood you right, you are still reading the entire files at once, but instead of reading one large file, you are reading small files, so you may still be accumulating a lot of memory. I read this article that may help you plumbr.io/blog/memory-leaks/…. TLTR: linux may be killing your process if the memory is getting too low, that is why you don't get the php error of exhausted memory.

Collectives™ on Stack Overflow

PHP memory limit not reached but OOM killer kills php script

2 Answers 2

Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related