System.OutOfMemoryException: Exception of Type 'System.OutOfMemory' was thrown

Question

private List<T> ReadCurrentFile(string currentExtractedFile, PurgingDetails purgingParams)
{
    List<T> thinLogDoList = new List<T>();
    using (StreamReader sr = new StreamReader(currentExtractedFile))
    {
        string currentLine = string.Empty;
        Dictionary<string, string> ColumnNamesDictionary = null;
        while ((currentLine = sr.ReadLine()) != null)
        {
            if (currentLine.IsNotNullOrEmpty() && currentLine.Contains("Æ"))
            {
                string[] columnNames = currentLine.Split(new char[] { 'Æ' });
                ColumnNamesDictionary = FillColumnNameDictionary(columnNames);

                if (CheckForValidConditions(ColumnNamesDictionary, purgingParams))
                {
                    thinLogDoList.Add(FillThinLogDO(ColumnNamesDictionary));
                }
            }
        }
    }
    return thinLogDoList;
}

(Above code is for Reading a File and adding data to the List by filling the object.) The function is reading file of size 10 MB which is inside a zip file, first I am extracting the zip files, then reading the data, using this function and storing it into List and then deleting the extracted zip files. It is working for approximately 6L(6,00,000) Data but above that data it throws exception. I want to read More data 10L(10,00,000) how should I do that ?

It is quite confusing if you use numbers in indian formatting — xanatos
– xanatos, Commented May 5, 2015 at 11:46
correct, you wouldn't really be able to do this. one way would be to do a direct yield / return an IEnumerable<T> or maybe even use a Dictionary perhaps. I have had massive perf improvements and less problems using a Dictionary. — Ahmed ilyas
– Ahmed ilyas, Commented May 5, 2015 at 11:47

Zoran Horvat · Accepted Answer · 2015-05-05 11:51:04Z

2

Do not return a list. Instead, use yield return to just run through the data:

private IEnumerable<i1LogThinDO> ReadCurrentFile(string currentExtractedFile,
                                                 PurgingDetails purgingParams)
{
    using (StreamReader sr = new StreamReader(currentExtractedFile))
    {
        string currentLine = string.Empty;
        Dictionary<string, string> ColumnNamesDictionary = null;
        while ((currentLine = sr.ReadLine()) != null)
        {
            if (currentLine.IsNotNullOrEmpty() && currentLine.Contains("Æ"))
            {
                string[] columnNames = currentLine.Split(new char[] { 'Æ' });
                ColumnNamesDictionary = FillColumnNameDictionary(columnNames);

                if (CheckForValidConditions(ColumnNamesDictionary, purgingParams))
                {
                    yield return FillThinLogDO(ColumnNamesDictionary);
                }
            }
        }
    }
}

This way, the ball is in the caller's yard. The caller must be able to process the data returned from this method without keeping them all in memory. This could mean that you have to redesign the calling methods as well, but it would bring a huge cut down in memory footprint of the application if you could do all the processing without keeping the data in memory.

answered May 5, 2015 at 11:51

Zoran Horvat

11.4k3 gold badges36 silver badges45 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

Mr. Monster Over a year ago

Thanks @Zoran, but your solution is also not working.

Zoran Horvat Over a year ago

Are you calling ToList() or something similar on this method? This solution won't work if any caller of this method materializes the list. You have to fix callers to run in foreach loop and never call ToList() or something like that.

Mr. Monster Over a year ago

Still Exception is thrown, how should i handle it ?

Zoran Horvat Over a year ago

Is the exception thrown in this function, or at the caller? If OutOfMemory exception is thrown then someone in the application is definitely converting this stream of objects into a list, and that is precisely what shouldn't be done.

Collectives™ on Stack Overflow

System.OutOfMemoryException: Exception of Type 'System.OutOfMemory' was thrown

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related