System.OutOfMemoryException using ThreadPool

Question

I'm experiencing an unexpected behaviour while running some very simple code.

First of all I'm using Visual Studio 2015 on a i7-4770 CPU, 32Gb memory (22,6 free)

My sample code:

        int length = 10;

        for (int i = 0; i < length; i++)
        {
            int j = i;
            //ThreadPool.QueueUserWorkItem(ThreadProc_CLR, j);
            Task.Factory.StartNew(() => { ThreadProc_CLR(j); });
        }

        public void ThreadProc_CLR(object parameter)
        {
            int i = Convert.ToInt32(parameter);

            byte[] data = new byte[1000000000];
            new Random().NextBytes(data);
            System.Security.Cryptography.SHA1.Create().ComputeHash(data);

        }

What I do not understand is why if I run my code using

Platform Target: Any CPU
Prefer 32-bit checked

I get a System.OutOfMemoryException after allocating the byte[] buffer for the 3rd or 4th time

If I uncheck "Prefer 32-bit" everything works smoothly. I've googled around looking for any docs explaining possible limitations but I haven't found any.

32-bit means you can't allocate more than 2GB of RAM. Your code tries to allocate 10GB of RAM. — Panagiotis Kanavos
– Panagiotis Kanavos, Commented Jan 25, 2016 at 12:59
You're creating SHA1 instances without disposing of them when you've finished. That's probably leaky. — spender
– spender, Commented Jan 25, 2016 at 13:01
@PanagiotisKanavos is completly correct - each iteration takes 1 billion bytes = 1 GB. — user57508
– user57508, Commented Jan 25, 2016 at 13:01
Related: How much memory a x86 excutable can use on a 64bit os? — Heinzi
– Heinzi, Commented Jan 25, 2016 at 13:02

Yacoub Massad · Accepted Answer · 2016-01-25 13:13:24Z

2

It seems that you are working on a 64-bit OS.

When you compile with "Any CPU" and the "Prefer 32-bit" setting is turned on. Then the process will execute as 32-bit process even if the OS is 64-bit.

32-bit processes can have a virtual address space of size 2GB (can be 4GB using the LARGEADDRESSAWARE setting). This is a lot less than what you need for your application to run. Your application seems to allocate about 10GB, and the cryptography API is also going to consume some memory.

When you uncheck the "Prefer 32-bit" setting, the process will run as a 64-bit process. This gives you a virtual address space of 8 TB.

edited Jan 25, 2016 at 13:13

answered Jan 25, 2016 at 13:01

Yacoub Massad

27.9k2 gold badges42 silver badges68 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Stefano Over a year ago

So basically "prefer 32-bit" means "force 32-bit". Understood, thanks for your feedback

spender · Accepted Answer · 2016-01-25 13:05:04Z

0

You're not disposing of your HashAlgorithm instances, so they're probably sitting around consuming memory.

//...
using(var ha = System.Security.Cryptography.SHA1.Create())
{
    ha.ComputeHash(data);
}

edited Jan 25, 2016 at 13:05

answered Jan 25, 2016 at 13:04

spender

121k36 gold badges245 silver badges369 bronze badges

3 Comments

Heinzi Over a year ago

While this is technically correct, it isn't the source of the problem the OP is facing.

spender Over a year ago

@Heinzi: I beg to differ. If OP is running out of memory after a number of passes, then yes, there is enough memory available in a 32-bit process to do what they want. Freeing it up will, most likely, avoid the necessity of switching to 64bit. Simply switching to 64-bit without understanding why the code chums up memory seems short-sighted to me.

Heinzi Over a year ago

You might have overlooked that the OP uses threading: the 10 instances with 1GB each are started in parallel. Thus, the 32 bit address space is never enough (unless, by coincidence, the scheduler totally serializes those 10 threads, but this is very unlikely), even if those instances correctly dispose their memory.

Collectives™ on Stack Overflow

System.OutOfMemoryException using ThreadPool

2 Answers 2

1 Comment

3 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

3 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related