0

I've been using simplexml_load_file to parse xml file with 15,000 records, it was working ok, but then when I tried to work with many files each of them is 15,000 it gave me this errorPHP 

Fatal error:  Allowed memory size of 134217728 bytes exhausted (tried to allocate 64 bytes)

Not sure what to do, The following is sample of what I'm doing,

$xml = simplexml_load_file($file)
       or die("Error: Cannot create object");

foreach($xml->children() as $events){
                foreach($events->load as $load){
                        $record = $load->loadrecord['record']."    ";
                        if ($load->loadrecord['record'] == "test"){
                                foreach($events->receiveds as $received){
                                $release = $received->received['release'];
                                }
                                foreach($events->sender as $sender){
                                $test1 = $sender['test1'];
                                $test2 = $sender['test2'];
                                $test3 = $sender['test3'];
                                $test4 = $sender['test4'];
                                }
                                foreach($events->screens as $filter){
                                $record = $filter->filter['record'];
                                }
                        }
}

Do I need to free something after the parsing is completed, please note that the issue happens when have many files, I tried with two files and there was no problem

Improve this question
3
  • There is clearly more to this code than what you're showing us, and you're not giving us much to go on. Best I can tell you given what's you've included is that you can use unset() on the variables once you're done with them to remove them from memory. Commented Nov 2, 2012 at 21:41
  • Where within the code do you get the memory error? In other words, are you able to even load the document in the first line, or are you able to execute part way through the script before you see the error? Commented Nov 2, 2012 at 21:49
  • 1
    have you even explored google? If you arent going to post enough code, double your memory limit atleast. ini_set('memory_limit', '256M'); Commented Nov 2, 2012 at 21:58

4 Answers 4

Reset to default
2

DON'T use simplexml for large files.. Use XML DOM object.

You may use some advanced tool like SAX or XMLReader or other third-party to parse the data.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

+1. SimpleXML loads the whole thing into memory. No wonder it blows up when you give it a big file.
1

Finally found the problem, you need to unset after each iterate, for example you need to unset the $xml here

$xml = simplexml_load_file(file_name )
foreach($xml->children() as $logs){
do_stuff
unset($xml);
}

What I had before was like this 

$xml = simplexml_load_file(file_name )
foreach($xml->children() as $logs){
do_stuff
}
unset($xml);

wouldn't really found it without your guidance

Improve this answer

Comments

0

You could consider using a cron job to process these files, one-by-one, then store their output somewhere and retrieve the output when it is finished.

This of course relies on you not needing the result immediately. If you did, you could begin this process and use AJAX requests to check for when it is done and to grab the final output.

Obviously, the needs and requirements may mean that's not feasible. Raising your memory limit is an option but typically not if you are on a shared hosting platform. It also just means you bypass the issue, rather than solving it (i.e. if the number of records increases again, the problem will come back)

Edit: misread question, modified answer to suit.

Improve this answer

5 Comments

Actually this is the code, All I'm doing is I have for loop going over a dir and start processing the file and put everything in a database, I didn't put it because it has nothing to do with the problem
IF that's all then I see no reason you couldn't build a little queue for them, process one file, add it to DB, then end the script and another gets started automatically if your cron detects there are new files to process. Instead of trying to do them all at once in one script.
same problem again, Now it's throwing this error after 10 files instead of 5
Are you still using the for loop?
What you want to do is put your processing script into a smaller script, add a new cron job for that, run it every 5 minutes or whatever is appropriate. Then 1 file gets processed every 5 minutes and you never run out of memory (maybe)
0

Unfortunately for you, the SimpleXML class loads the whole XML file into memory. This will obviously cause you issues if you give it a large file.

Instead, you'll need to use the XMLReader class. This class reads the XML one element at a time, and throws it away after reading. This means you have a minimum of data in memory at any one time.

A convenient way to use this class is by wrapping it in an Iterator class. This means you can use foreach() to loop through the elements just as if it were all loaded at once.

Here's a link to a great example of an Iterator class for XMLReader. When I had exactly this issue, I found this class very helpful. I had to make a couple of minor tweaks to suit my needs, but it pretty much worked first time. I hope it works for you too.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.