.NET heaps filled up with string object -> OutOfMemoryException

Question

I'm constantly (every 30-60 min) getting a System.OutOfMemoryException in my Windows Service. The service's job is to loop though 6 directories which contains data files which the service datawashes to a common XML data format.

These 6 folders contains 5-10.000 files each, so the total number of files is about 45.000 and new files is added duing the day. There is added about 1-2000 new files a day. The files is between 4KB and 500KB.

Each data file is washed to the common XML data format through the XElement object.

I have used RedGates ANTS Memory Profiler on the service and the objects which are using the most memory is string (about 90.000.000 bytes) and XElement (about 51.000.000 bytes).

In the Memory Profiler, when i trace, what is using the string object, i can see that it's mostly (93%) the XElement object which is using the string object.

The server have 6 cpu's and 6GB of RAM, so i can't see why i'm getting the OutOfMemoryException. If i look at the Windows Service in the Processes it's MAX use of RAM have been 1.2GB.

I have read that .NET garbage collector doesn't clear the string object because the string object is stored in a intern table. Could this be the error, if so what can i do about it?

The code below shows how i'm looping through the files. As you can see i have also tried to take 20 files at a time. This just pushes the OutOfMemoryException a few hours, so the service will run for 4-5 hours instead of 30-60 min.

Why do i can the OutOfMemoryException?

private static void CheckExistingImportFiles(object sender, System.Timers.ElapsedEventArgs e)
    {
        CheckTimer.Stop();
        var dir = Directory.GetFiles(RawDataDirectory.FullName, "*.*", SearchOption.AllDirectories);

        List<ManualResetEvent> doneEvents = new List<ManualResetEvent>();
        int i = 0;
        //int doNumberOfFiles = 20;

        foreach (string existingFile in Directory.GetFiles(RawDataDirectory.FullName, "*.*", SearchOption.AllDirectories))
        {
            if (existingFile.EndsWith("ignored") || existingFile.EndsWith("error") || existingFile.EndsWith("importing"))
            {
                //if (DateTime.UtcNow.Subtract(File.GetCreationTimeUtc(existingFile)).TotalDays > 5)
                //  File.Delete(existingFile);
                //continue;
            }

            StringBuilder fullFileName = new StringBuilder().Append(existingFile);

            if (!fullFileName.ToString().ToLower().EndsWith("error") && !fullFileName.ToString().ToLower().EndsWith("ignored") && !fullFileName.ToString().ToLower().EndsWith("importing"))
            {
                File.Move(fullFileName.ToString(), fullFileName + ".importing");
                fullFileName = fullFileName.Append(".importing");

                ImportFileJob newJob = new ImportFileJob(fullFileName.ToString());

                doneEvents.Add(new ManualResetEvent(false));

                ThreadPool.QueueUserWorkItem(newJob.Run, doneEvents.ElementAt(i));
                i++;
            }

            //if (i > doNumberOfFiles)
            //{
            //    i = 0;
            //    doNumberOfFiles = 20;
            //    break;
            //}
        }
        i = 0;
        WaitHandle.WaitAll(doneEvents.ToArray());

        CheckTimer.Start();
    }

Your usage of StringBuilder is redundant. There is not benefits IOW. — leppie
– leppie, Commented Feb 19, 2012 at 11:18
ImportFileJob takes the datafile and uses a xlst stylesheet to transform the datafile to the common XML data file. — Poku
– Poku, Commented Feb 19, 2012 at 11:23
I used a List<string> in the app which caused the service to do outofmemoryexception after 3-5 min. ANTS memory profiler told me that it was the List<string> which was the reason. Change that with List<StringBuilder> solved that issue. But the exception kept coming, just not that fast, so i tried to replace all strings with StringBuilder, because i read that string objects are not collected by the .NET GC. — Poku
– Poku, Commented Feb 19, 2012 at 11:26
Could you show the code for ImportJob as well? There are no XDocuments in this method, so I'm guessing any problems resulting in 51MBs' worth of XDocuments are hidden there. — Avner Shahar-Kashtan
– Avner Shahar-Kashtan, Commented Feb 19, 2012 at 11:27

FrantzX · Accepted Answer · 2012-02-19 11:22:47Z

2

Directory.GetFiles(RawDataDirectory.FullName, "*.*", SearchOption.AllDirectories);

This returns an array. If the directories have as many files as you state, these will be very large arrays, large enough to be placed in the Large Object Heap. Mutliple massive arrays there could easily cause a OutOfMemoryException. It doesn't help that the following line

var dir = Directory.GetFiles(RawDataDirectory.FullName, "*.*", SearchOption.AllDirectories);

has the variable 'dir' that isn't doing anything. The large array is created twice per method execution.

answered Feb 19, 2012 at 11:22

FrantzX

621 silver badge4 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

L.B · Accepted Answer · 2012-02-24 10:13:40Z

1

As Avner Shahar-Kashtan already stated, I also think that the problem is in ImportJob(you haven't shown us its code).

Even so, you can still make some optimizations.

You don't have to load all file names at once. It can be done dir by dir as below

IEnumerable<string> GetAllFiles(string dirName)
{
    var dirs = Directory.GetDirectories(dirName);

    foreach (var file in Directory.GetFiles(dirName))
        yield return file;

    foreach (var dir in dirs) //recurse
        foreach (var file in GetAllFiles(dir)) 
            yield return file;
}

And by using TPL, you can reduce the number of ManualResetEvents created (and their forgotten Dispose()s)

Parallel.ForEach(GetAllFiles(RawDataDirectory.FullName) , file =>
{
    //ImportFileJob newJob = new ImportFileJob(file);
    //newJob.Run
    Console.WriteLine(file);
});

BTW, you should also see CountdownEvent

edited Feb 24, 2012 at 10:13

answered Feb 19, 2012 at 12:00

L.B

116k20 gold badges189 silver badges229 bronze badges

2 Comments

Poku Over a year ago

ImportFileJob class does alot of diffenrent things and the XElement class is used alot, so you are rigth that the problem could be here. Shouldn't the .NET garbage collector clean up the XElement object?

L.B Over a year ago

there may be millions of reasons. I don't know your code. But there is no reason for suspecting from XElement or Garbage Collector unless you don't release resources (like files)

Avner Shahar-Kashtan · Accepted Answer · 2012-02-19 11:22:08Z

1

I can spot a couple of easy optimizations right off the bat.

You use a lot of fullFileName.ToString().ToLower().EndsWith("ignored") calls. These have a lot of overheads, since you always take the given string and create a new, lower-case string.

Instead, you should use the Endswith (or Contains) overloads that allow for a case-insensitive comparison:

fullFileName.ToString()
  .EndsWith("ignored", StringComparison.CurrentCultureIgnoreCase)

Also, I don't think your StringBuilders are helping in this case. StringBuilders are mostly useful when you're building multi-part strings and don't want the overhead of creating several intermediate strings while you're composing them. It seems that all your string concatenations here are always using only two strings - the base name and the new suffix - so I'm not sure it's actually saving you any time or memory.

answered Feb 19, 2012 at 11:22

Avner Shahar-Kashtan

14.8k3 gold badges42 silver badges66 bronze badges

1 Comment

TomTom Over a year ago

-1. You are right, but none of that should cause hthat issue - those string are discarded immediately.

Jared Shaver · Accepted Answer · 2012-02-19 12:55:07Z

1

Instead of using a timer and looping over all the contents of the folders you could use a FileSystemWatcher: http://msdn.microsoft.com/en-us/library/system.io.filesystemwatcher.aspx

That way your program is notified of the exact files that changed and you don't even have to allocate the memory for the arrays of files that you don't care about.

answered Feb 19, 2012 at 12:55

Jared Shaver

1,3398 silver badges12 bronze badges

Comments

user45886 · Accepted Answer · 2012-02-19 11:23:43Z

0

You call fullFileName.ToString().ToLower() three times in your If statement. Cache this string value in a local variable and use that your if statement (saves you three temporary strings).

Try using XmlWriter and not XDocument. XDocument is in-memory object graph so for large datasets it might not be the most performant (you hold the whole thing in memory until you write it out to disc as a whole). With XmlWriter you can typically stream off to a file buffer element by element, memory footprint would be far less demanding.

Not sure how much work each import is, but have you tried a thread per directory instead of per file?

answered Feb 19, 2012 at 11:23

user45886

Comments

Manish Basantani · Accepted Answer · 2012-02-19 12:20:31Z

As others suggested,

1) reduce string manipulation.

It appears your directory is returning "too many" file names (strings) so that needs attention.

2) your line 'var dir = Directory.GetFiles(RawDataDirectory.FullName, "*.*", SearchOption.AllDirectories);' seems to be redundant. It looks like you are not using it. So, remove this code, it is holding lot of string references.

3) IF possible, iterate through the files returned from the directory in chunks (say 10K). So this would need writing a code that splits List to List>, and then clear the references held by the inner list as you iterate through the outer loop. Somthing like,

foreach(List<List<string>> fileNamesInChunk in GetFilesInChunk(directoryName)){
     foreach(var fileName in fileNamesInChunk){
     //Do the processing.
     }
     fileNamesInChunk.Clear(); //This would reduce the working set as you proceed.
}

Hope this would help.

Collectives™ on Stack Overflow

.NET heaps filled up with string object -> OutOfMemoryException

6 Answers 6

Comments

2 Comments

1 Comment

Comments

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

6 Answers 6

Comments

2 Comments

1 Comment

Comments

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related