Putting length to 0 for OutOfMemoryException of StringBuilder

Question

I have a block of code which gets a file and appends require data as shown below:

var srBuilder = new StringBuilder();
using (var file = new StreamReader(document.FullSourcePath, Encoding.ASCII))
            {
                while (!file.EndOfStream)
                {                 
                    var bytess = new char[numBytes];
                    file.ReadBlock(bytess , 0, bytess.Length);
                    srBuilder.Append(buff);
                }

                document.Document += srBuilder.ToString();  ////Exception occures here
            }

But when file is more than 200 MB then its throwing OutofMemoryException.

What i thought is to make length of string builder to zero as below:

 while (!file.EndOfStream)
        {  
            srBuilder.Length = 0;      //// Here         
            var bytess = new char[numBytes];
            file.ReadBlock(bytess , 0, bytess.Length);
            srBuilder.Append(buff);
        }

Is it best solution or anything else is required?

Your code does not make sense to me. Can you explain in plain English what you want to happen? — nvoigt
– nvoigt, Commented Mar 10, 2016 at 10:49
At 32 bits it is very difficult to have CONTIGUOUS space for a string of 400mb (your 200mb of ASCII file in .NET will be a 400mb unicode string, with each character 2 bytes). Try at 64 bits (should load without problems) or try finding a different algorithm where you don't need loading a 200mb file in memory. — xanatos
– xanatos, Commented Mar 10, 2016 at 10:54
In general you don't load a full file of unknown size in memory. You load it one row at a time, you parse and use that row, you forget of that row and go to the next row. — xanatos
– xanatos, Commented Mar 10, 2016 at 11:02
Setting the length to 0 is effectively clearing the stringbuilder's contents. You cannot do that, as you're then simply throwing away data. That's the problem with copy-pasting code you don't understand. If it is a "requirement" that the entire text file needs to be loaded in-memory at once (which is not a real requirement, but might be your interpretation of it), then you have a problem: you can not reliably do this. Explain what you do with document.Document afterwards that makes you think you need this, then a real solution can be suggested. — CodeCaster
– CodeCaster, Commented Mar 10, 2016 at 11:12

nvoigt · Accepted Answer · 2016-03-10 10:53:33Z

3

I don't know why you made it that complicated. All you need is a single line:

document.Document += File.ReadAllText(document.FullSourcePath, Encoding.ASCII);

If this throws an exception, then yes, maybe you don't have enough memory.

answered Mar 10, 2016 at 10:53

nvoigt

77.8k28 gold badges103 silver badges156 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Neel Over a year ago

i had 50 rounds of testing and it threw exception once. is it problem? i had some debuggers in that case

CodeCaster · Accepted Answer · 2016-03-10 11:32:27Z

2

Your ultimate goal is to assign a 200 MB ASCII file to the document.Document member variable, which I assume is of type string.

Long story short: you cannot do this, and need to reconsider your approach.

One way or another, this requirement of yours will require 400 MB of contiguous memory (given strings are char[] arrays of UTF-16 characters, weighing 2 bytes per character), which is nigh impossible to obtain in a 32 bit process, which the OutOfMemoryException that the runtime throws is trying to tell you.

Any "trick" you're going to find is going to have its drawbacks.

Reading the file in chunks doesn't matter, as the end result is the same: the final string will still require 400 MB of contiguous memory.
Setting the StringBuilder's Length to 0, found through Googling "C# StringBuilder OutOfMemoryException", will make the exception go away, but only cause the last chunk of the file to be assigned to document.Document. It's safe to say you don't want that as well.
Changing the project to run only on 64 bit machines might be an option, but most likely won't (hardware and other dependencies), and is a dirty workaround that hides the actual problem, and that is like applying a band-aid to a blown-off leg: you're ignoring the fatal design mistake. One day someone is going to feed your app a 2 GB file, and you're back to square one.

You need to stream the file, and process it line by line.

foreach (string line in File.ReadLines(filename))
{
    // process line
}

answered Mar 10, 2016 at 11:32

CodeCaster

153k24 gold badges237 silver badges287 bronze badges

4 Comments

Neel Over a year ago

by the way what if i still need everything in document.Document then?

Neel Over a year ago

thing is whole code is already written and i am just there to handle 200 MB file case :/

CodeCaster Over a year ago

Somebody gave me a 5 liter bucket and tells me to go get 200 liter of water with it. What do I tell them?

CodeCaster Over a year ago

My point is: whatever it is you're doing with document.Document must be changed to support streaming. You simply cannot assign 400 MB of data to it.

Collectives™ on Stack Overflow

Putting length to 0 for OutOfMemoryException of StringBuilder

2 Answers 2

1 Comment

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related