C# OutOfMemoryException creating ZipOutputStream using SharpZipLib

Question

I keep getting a very annoying OutOfMemory exception on the following code.

I'm zipping a lot of small files (PDF, each being 1.5mb approx).

At first I was getting the exception afer 25 files zipped, which doesn't seem like a massing archive.

Setting up the size of the ZipEntry somehow helped since now I manage to get up to 110 files zipped (I'm debugging under visual studio)

Here's my code, maybe there's something wrong with it.

Any help would be greatly appreciated.

Thanks

    public static MemoryStream Zip(Dictionary<string, byte[]> files)
    {
        var outputMemStream = new MemoryStream();

        var zipStream = new ZipOutputStream(outputMemStream);

        zipStream.SetLevel(9);
        foreach (var file in files)
        {
            zipStream.PutNextEntry(new ZipEntry(file.Key.FmtValidFileName())
                {
                    Size = file.Value.Length
                });
            zipStream.Write(file.Value, 0, file.Value.Length);
            zipStream.Flush();
        }           
        zipStream.Finish();
        outputMemStream.Position = 0;
        return outputMemStream;
    }

Every class in there that implements IDisposible needs to be put into a using block. If you have a Visual Studio version that supports it, you can turn on static code analysis so you get warnings if you forget to do it properly. — nvoigt
– nvoigt, Commented Oct 17, 2014 at 4:45
It doesn't help I'm afraid. The OutOfMemoryException happens within the loop on files. I can't put the outputMemStream inside a using because I'm using it as the return. I tried to use buffer instead of writing the file entirely but it doesn't help either, the problem is not the size of each file I'm zipping but the total size of all files. — Etienne
– Etienne, Commented Oct 17, 2014 at 4:54
You should create a small, self contained example. For all we know, FmtValidFileName could consist of throw new OutOfMemoryException();. Prepare an example that people can copy and compile themselves using best practices and it will be a lot easier to help you. — nvoigt
– nvoigt, Commented Oct 17, 2014 at 5:00
FmtValidFileName is an extension method for a string to remove any fuzzy character from the filename that is being zipped so it's not the problem. Also I cannot give more than the code extract I've given, it's the smallest I got, I'm running it against a Dictionary<string, byte[]> files consisting of 160 records, each byte[] being approx 1434167 byte which is 1.3Mb more or less and it throws the OutOfMemory after approx a 100 files have been zipped. — Etienne
– Etienne, Commented Oct 17, 2014 at 5:09

Bob Kaufman · Accepted Answer · 2021-02-15 22:15:22Z

2

As always, a concise-but-complete code example would go a long way toward helping you get good answers.

That said, you might want to consider using the (relatively new) System.IO.Compression.ZipArchive class in .NET. It is potentially less buggy and/or more reliable than third party libraries (though I admit SharpZipLib is reasonably well-respected :) ).

More important, you can instantiate a new ZipArchive object with the ZipArchiveMode.Create value, which will cause the compressed data to be written directly to the stream rather than being cached in-memory. In this mode, out-of-memory errors should be non-existent, no matter how much data or how many archive items you're trying to create.

EDIT: One more thing: to ensure completely against out-of-memory problems, make sure that whatever .zip implementation you are using, you write directly to the disk. Writing to a temporary in-memory MemoryStream will of course impose limitations on your process that otherwise need not occur.

edited Feb 15, 2021 at 22:15

Bob Kaufman

12.9k16 gold badges83 silver badges112 bronze badges

answered Oct 17, 2014 at 5:35

Peter Duniho

71k7 gold badges113 silver badges149 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Etienne Over a year ago

Thanks I will try that but I've got to ask what the.. is wrong with my code sample ? imho it is both concise and complete.

Peter Duniho Over a year ago

A complete code example is one that I or anyone else can copy into an empty project and then compile and run without adding anything else. It's tricky to do in some cases, such as this one, because the example has to include or generate test data needed to reproduce the problem. But while your code example is nicely concise, it's not complete.

Etienne Over a year ago

Rarely seen any of those in StackOverflow but fair enough I can totally understand that people willing to help won't spend any time in generating the input necessary to reproduce the problem with my code.

Peter Duniho Over a year ago

I agree they are rare. There are a lot of really poorly-asked questions on SO. And I agree the one you posted here is one of the better ones. But at the end of the day, given that there's not even a guarantee of success, few if any people will spend much time trying to flesh an existing example out so that it runs and reproduces the problem. So you're left with purely speculative answers, which are never as good as to-the-point, fully-informed ones. :)

Etienne Over a year ago

Using ZipArchive didn't change, but I think my problem is that the IIS server running from Visual Studio is 32 bit and I'm reaching the natural memory limit.

|

Community · Accepted Answer · 2017-05-23 10:25:58Z

0

I gave up trying to use MemoryStream even though being on a 64bit system with 16gb of memory I should have been safe on that side.

The relevant topic I found was: OutOfMemoryException while populating MemoryStream: 256MB allocation on 16GB system

And using a temporary file to write/read the data instead of memory.

edited May 23, 2017 at 10:25

CommunityBot

11 silver badge

answered Oct 17, 2014 at 8:23

Etienne

1,1161 gold badge12 silver badges22 bronze badges

Collectives™ on Stack Overflow

C# OutOfMemoryException creating ZipOutputStream using SharpZipLib

2 Answers 2

6 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related