How to prevent System.OutOfMemoryException when retrieving large amounts of data in c#

Question

I have a C# program that populates a dictionary with 4Million Guids at startup. I keep getting an Exception of type 'System.OutOfMemoryException' error at this point.

Example:

using (reportingconn)
{
    var initialrowkeys = reportingconn.Query("select rowkey from table”);
}

Can anyone give me some ideas how best to retrieve large amounts data from SQL into a dictionary?

Do you really need all 4 million at once? are you looking for a specific one? — Brad
– Brad, Commented Mar 29, 2013 at 19:05
Why do you need to populate 4 million guids at startup? That is the problem here. — TheGeekYouNeed
– TheGeekYouNeed, Commented Mar 29, 2013 at 19:05
I love this question - classic case of something grizzled old programmers would never ask and young programmers rarely have to consider (stream your data, small RAM). — Chris Moschini
– Chris Moschini, Commented Mar 29, 2013 at 19:06
@Brad I need to use those guids to retrieve data from Azure table storage n a loop. I do not want to keep hitting the database every-time. — user1526912
– user1526912, Commented Mar 29, 2013 at 19:10
Databases are meant to store large amounts of data and be able to handle large numbers of queries. I'd be more worried about a query returning millions of results than hitting the database for data on-demand. — jugg1es
– jugg1es, Commented Mar 29, 2013 at 19:18

iefpw · Accepted Answer · 2013-03-29 19:05:51Z

5

Use SqlDataReader() this will read row by row, instead of SqlDataAdapter.Fill.

answered Mar 29, 2013 at 19:05

iefpw

7,08215 gold badges59 silver badges83 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

user1526912 · Accepted Answer · 2013-03-29 20:35:19Z

0

It seems like there is a 2GB memory limitation on C# data dictionary. A programmer friend told me to change my console application from 32 bit to 64 bit to prevent that limitaion. That seems to have resolved my problem.

answered Mar 29, 2013 at 20:35

user1526912

17.5k18 gold badges62 silver badges98 bronze badges

1 Comment

Brad Over a year ago

I would guess the limit is more like 2 GB of contiguous addressable memory. Some Good reading

Collectives™ on Stack Overflow

How to prevent System.OutOfMemoryException when retrieving large amounts of data in c#

2 Answers 2

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related