Difficulty reading large file into byte array

Question

I have a very large BMP file that I have to read in all at once because I need to reverse the bytes when writing it to a temp file. This BMP is 1.28GB, and I'm getting the "Out of memory" error. I can't read it completely (using ReadAllBytes) or using a buffer into a binary array because I can't initialize an array of that size. I also can't read it into a List (which I could then Reverse()) using a buffer because halfway through it runs out of memory.

So basically the question is, how do I read a very large file backwards (ie, starting at LastByte and ending at FirstByte) and then write that to disk?

Bonus: when writing the reversed file to disk, do not write the last 54 bytes.

Editing the question to include the .NET framework version you're using would help since, beginning from version 4, you can access Memory-Mapped file through managed code. — s.m.
– s.m., Commented Apr 8, 2013 at 8:51

Vico · Accepted Answer · 2017-11-13 03:55:58Z

4

With a StreamReader object, you can Seek (place the "cursor") to any particular byte, so you can use that to go over the entire file's contents in reverse.

Example:

const int bufferSize = 1024;
string fileName = 'yourfile.txt';

StreamReader myStream = new StreamReader(fileName);
myStream.BaseStream.Seek(bufferSize, SeekOrigin.End);

char[] bytes = new char[bufferSize];
while(myStream.BaseStream.Position > 0)
{
    bytes.Initialize();
    myStream.BaseStream.Seek(bufferSize, SeekOrigin.Current);
    int bytesRead = myStream.Read(bytes, 0, bufferSize);
}

edited Nov 13, 2017 at 3:55

Vico

1182 silver badges14 bronze badges

answered Apr 8, 2013 at 8:45

Roy Dictus

33.3k9 gold badges64 silver badges77 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Aeon2058 Over a year ago

Thanks for this! Instead of the SeekOrigin.Current, I just looped through the file using the Position option, read forward the number of bytes I needed, then reversed the buffer before writing it to disk with a Filestream Append. Works great!

Tigran · Accepted Answer · 2013-04-08 08:45:34Z

4

You can not normally handle so big files in .NET, due the implied memory limit for CLR applications and collections inside them neither for 32 nor for 64 platform.

For this you can use Memory Mapped File, to read a file directly from the disk, without loading it into the memory. One time memory mapping created move the reading pointer to end of the file and read backwards.

Hope this helps.

answered Apr 8, 2013 at 8:45

Tigran

62.3k8 gold badges90 silver badges124 bronze badges

Comments

Memoizer · Accepted Answer · 2013-04-08 08:45:40Z

2

You can use Memory Mapped Files.

http://msdn.microsoft.com/en-us/library/vstudio/dd997372%28v=vs.100%29.aspx

Also, you can use FileStream and positioning on necessary position by stream.Seek(xxx, SeekOrigin.Begin) (relative position) or Position property (absolute position).

answered Apr 8, 2013 at 8:45

Memoizer

2,2991 gold badge17 silver badges15 bronze badges

Collectives™ on Stack Overflow

Difficulty reading large file into byte array

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related