2

I am programming in PHP and I have an include file that I have to, well, include, as part of my script. The include file is 400 MB, and it contains an array of objects which are nothing more than configuration settings for a larger project. Here is an example of the contents:

$obj = new obj();
    $obj->name = "myname";
    ....

$objs[$obj->name] = $obj->name;
{repeat}
....
return $objs;

This process repeats itself 40,000 times and ultimately is 650,000 lines long (the file is generated dynamically of course.)

If I am simply trying to load a file that is 400MB, why then would memory usage increase to 6GB? Even if the file is loaded into memory, wouldn't only take up 400 MB of RAM?

Improve this question
6
  • 4
    Why, oh why would you use a 400MB script? Can't you generate a simple datafile you can read in a streaming fashion? Or database? Commented Feb 28, 2013 at 8:36
  • CodesInChaos and @cHao: I am new to this, so I am sure there is a better way! I am actually trying to get each of these into the database. I wanted to include them, loop through them, and then each object would be a separate configuration setting in the database. Although a tad off topic, in the comments if you know of a better way, please advise or provide a link that can help me understand a bit better. I thought about one file for each object, but then I have to loop through 40k files, and that isn't working out that well either! Thanks! Commented Feb 28, 2013 at 8:52
  • 2
    This usually happens when you allocate memory (with new()) without deleting it accordingly. Then even a few lines of code soon can consume all the memory available. Commented Feb 28, 2013 at 9:07
  • 1
    @user658182: Looping through 40k files should actually be less demanding memorywise; you can insert the stuff from each file into the DB, and then forget a bunch of it. But ideally, you wouldn't be doing this with PHP scripts at all. If you had a data file you read in instead, you could read it line by line, and PHP wouldn't have to compile it. Commented Feb 28, 2013 at 16:33
  • Instead of a 400MB PHP file that consumes 6GB, you need a 4kb php file that uses a binary search to search through a sorted 6GB file for what you want (if you do this, then DO NOT load the entire 6GB file into memory, rather jump around the hard disk until you get down to like a 2kb segment that you can read into memory and continue the search in memory). Commented Jul 7, 2017 at 21:37

3 Answers 3

Reset to default
7

There's not just the file. There's the parse tree that gets generated from parsing the file, the bytecode it gets compiled to, the memory taken up by the variables defined in it, etc. 6GB still sounds like a lot, but for a 650,000 line file it doesn't surprise me much.

At some point along the way, someone should probably have thought of using a database. :) Unless you're using every item in that file every time, loading half a million lines of stuff to pull out a few things is incredibly wasteful.

Improve this answer
1
  • Ah, that's very informative, thank you. I did not know so much extra overhead was involved. I thought it was simply a 1 to 1 ratio, or at least, until I started manipulating it. Commented Feb 28, 2013 at 8:54
5

File size is the size of the code that is executed.

Memory usage is the size of the contents of all the variables created by the execution of the code.

An algorithm computing pi may take less than 10 lines of code and need billions of bytes just to store the result.

Improve this answer
3
  • -1: does not answer the question Commented Feb 28, 2013 at 17:43
  • 1
    @sparkleshy - I would be happy to learn from your answer then. Commented Feb 28, 2013 at 18:22
  • 2
    +1: answers the question correctly, also for handling criticism politely Commented Mar 1, 2013 at 0:31
3

A script is typically a fixed size. It is read in and effectively executed line by line. This is your file size.

While the script is running, it can say to the computer "store this information which I have generated for me", and the computer dutifully gives it a bit of memory. This is your memory size.

Your script can ask repeatedly for more and more places to store the information it has generated, often storing the same or similar things in many different places. This can take up many megabytes, and even gigabytes. The amount of storage (in memory) is almost totally independent of script size, although the same script will normally use the same amount each time it is run.

Improve this answer
1
  • -1: does not answer the question Commented Feb 28, 2013 at 17:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.