Out of Memory Exception Entity Framework When Saving

Question

I'm looping through many different objects in entity framework, grabbing some related data and saving that data. I'm opening a new thread for each item since it takes a while to do the work in the method. However, I'm getting out of memory exceptions a couple thousand items into the loop. I'm trying to figure out where this problem is occurring and how to keep the multi-thread functionality, but fix the error. Any help is greatly appreciated. Thank you.

    foreach (var prod in products)
    {
        try
        {
            new Thread(() =>
            {
                ParsePage(prod);
            }).Start();


            counter++;
            Console.WriteLine(counter);
        }
        catch (Exception ex)
        {
            string message = ex.Message;
        }
    }

    public void ParsePage(Product p)
    {
        //New Repository
        IDataRepository repo = new DataRepository();

        //Grab related page report
        PageReport pr = repo.PageReports.Where(c => c.ThePageType == "Product").FirstOrDefault(c => c.ThePageTypeID == p.ProductID);

        //Do some stuff

        //Save
        repo.SavePageReport(pr);

    }

Is that DataRepository IDisposable? If so, try wrapping it in a using statement. — dbc
– dbc, Commented Nov 12, 2014 at 18:19
Does it contain disposable members like a DbContext? If so, maybe it should be disposable. — dbc
– dbc, Commented Nov 12, 2014 at 18:24
As a sidenote you should consider using Tasks instead of Threads — Mohnkuchenzentrale
– Mohnkuchenzentrale, Commented Nov 12, 2014 at 18:40

dbc · Accepted Answer · 2014-11-12 19:11:46Z

You may be running out of memory because each thread has a default stack size of 1 MB. 2000 started threads implies 2 GB of stack, which may be the max on your system:

Memory Limits for Applications on Windows*

32-bit

Static data - 2GB

Dynamic data - 2GB

Stack data - 1GB (the stack size is set by the linker, the default is 1MB. This can be increased using the Linker property System > Stack Reserve Size)

Note that on 32-bit Windows, the sum of all types of data must be 2GB or less. The practical limit is about 1.75GB due to space used by Windows itself

64-bit

Static data - 2GB

Dynamic data - 8TB

Stack data - 1GB (the stack size is set by the linker, the default is 1MB. This can be increased using the Linker property System > Stack Reserve Size)

See also this article by Raymond Chen and this stackoverflow question for more information.

As KroaX said, the newer task and older threadpool libraries are designed to handle situations like this. They offer an additional level of abstraction between threads and tasks, allowing you to easily create thousands of lightweight tasks that will be assigned to threads as they become available.

Daniel Persson · Accepted Answer · 2014-11-12 18:36:04Z

1

If you're using the same DbContext for the entire operation, its cache may cause the Out of memory error. I had a similar situation when parsing and storing large amounts of data from XML files.

You should make your repository disposable, as dbc suggests, and then wrap it in a using clause:

public void ParsePage(Product p)
{
    using (var repo = new DataRepository())
    {
        //Grab related page report
        PageReport pr = repo.PageReports.Where(c => c.ThePageType == "Product").FirstOrDefault(c => c.ThePageTypeID == p.ProductID);

        //Do some stuff

        //Save
        repo.SavePageReport(pr);
    }
}

Info on how to implement the Disposable pattern can be found here

answered Nov 12, 2014 at 18:36

Daniel Persson

2,2311 gold badge18 silver badges26 bronze badges

Collectives™ on Stack Overflow

Out of Memory Exception Entity Framework When Saving

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related