Linq2Sql out of memory exception

Question

I'm experiencing OutOfMemory exceptions in my application, when fetching from the database. It is a C# .Net application using Linq2Sql.

I have tried using GC.GetTotalMemory() to see how much memory is taken up before and after the call to the database. This gives a nice although not quite accurate picture of what is going on. When I look in the Windows Task Manager I can see that the Peak Working Set is not smaller when fetching the data in a paged manner using the following code:

public static void PreloadPaged()
{
    int NoPoints = PointRepository.Count();
    int pagesize = 50000;
    int fetchedRows = 0;

    while (fetchedRows < NoPoints)
    {
        PreloadPointEntity.Points.AddRange(PointRepository.ReadPaged(pagesize, fetchedRows));
        PointRepository.ReadPointCollections();
        PreloadPointEntity.PointCollections.Count());
        fetchedRows += pagesize;
    }
}


private static List<PointEntity> ReadPaged(int pagesize, int fetchedRows)
{
    DataModel dataContext = InstantiateDataModel();
    var Points = (from p in dataContext.PointDatas
                select p.ToEntity());

    return Points.Skip(fetchedRows).Take(pagesize).ToList();
}

I guess it's the Linq2Sql code that is using up the memory and not reusing it or freeing it afterwards, but what can I do to get the memory foot print down?

I have observed that it uses 10 times as much memory to fetch the data as it does to store them in my list of enties. I have considered invoking the garbage collector, but I would rather avoid it.

50000 is just my page size chosen for experimental purpose, I'm prefetching a grid of coordinates which all have to be processed. So I just store them in a list of entities and access them from there in stead of having to make a new DB connection every time I need another coordinate. — Keller
– Keller, Commented Jun 21, 2012 at 10:56

Jason Meckley · Accepted Answer · 2012-06-21 10:57:24Z

2

you are retrieving way too much data and storing it in memory, that's why you are getting an OOM exception.

1 of 2 things is occurring:

you are loading an excessive amount of data when the user will only view a subset of the results and/or this is a 1st attempt at "caching" data.
you do need all this data, but are using the wrong technology (Linq2Sql) to access the data.

if it's the first, you need to either

load smaller chunks of data (20-50 records, not 50K or everything)
if this is only for display purposes, then query a projection of what's needed, rather than the entity itself.

if it's the second than use an ETL tool designed to manage large amounts of data. I prefer Rhino.ETL but SSIS also works.

answered Jun 21, 2012 at 10:57

Jason Meckley

7,6111 gold badge26 silver badges46 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Keller Over a year ago

Maybe Linq2SQL is not the best framework to use for the amounts of data, I'll look into the ETL tools you suggest here, thanks.

Collectives™ on Stack Overflow

Linq2Sql out of memory exception

1 Answer 1

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related