Java Read File Larger than 2 GB (Using Chunking)

Question

I'm implementing a file transfer server, and I've run into an issue with sending a file larger than 2 GB over the network. The issue starts when I get the File I want to work with and try to read its contents into a byte[]. I have a for loop :

for(long i = 0; i < fileToSend.length(); i += PACKET_SIZE){
    fileBytes = getBytesFromFile(fileToSend, i);

where getBytesFromFile() reads a PACKET_SIZE amount of bytes from fileToSend which is then sent to the client in the for loop. getBytesFromFile() uses i as an offset; however, the offset variable in FileInputStream.read() has to be an int. I'm sure there is a better way to read this file into the array, I just haven't found it yet.

I would prefer to not use NIO yet, although I will switch to using that in the future. Indulge my madness :-)

What is your file transfer server written with? Do you use servlets? If so, just use the HttpServletResponse.getOutputStream() method and stream the file contents directly to the output stream...you don't have to do any chunking work youself. — Michael
– Michael, Commented Nov 24, 2011 at 14:31
I'm using Sockets and ServerSockets, which as far as I know, can only send byte[]. Am I missing something? — Eliezer
– Eliezer, Commented Nov 24, 2011 at 15:01
No, you're not using servlets :) I think I see your problem now. Because the files are so big, an int becomes too small. Let me write an answer — Michael
– Michael, Commented Nov 24, 2011 at 15:05
You are most likely to find the most efficient buffer size is around 32 KB (The size of your L1 cache) — Peter Lawrey
– Peter Lawrey, Commented Nov 24, 2011 at 15:10

Michael · Accepted Answer · 2011-11-24 15:22:58Z

4

It doesn't look like you're reading data from the file properly. When reading data from a stream in Java, it's standard practice to read data into a buffer. The size of the buffer can be your packet size.

File fileToSend = //...
InputStream in = new FileInputStream(fileToSend);
OutputStream out = //...
byte buffer[] = new byte[PACKET_SIZE];
int read;
while ((read = in.read(buffer)) != -1){
  out.write(buffer, 0, read);
}
in.close();
out.close();

Note that, the size of the buffer array remains constant. But-- if the buffer cannot be filled (like when it reaches the end of the file), the remaining elements of the array will contain data from the last packet, so you must ignore these elements (this is what the out.write() line in my code sample does)

edited Nov 24, 2011 at 15:22

answered Nov 24, 2011 at 15:12

Michael

35.5k17 gold badges79 silver badges112 bronze badges

Sign up to request clarification or add additional context in comments.

6 Comments

Eliezer Over a year ago

Thanks a lot! This snippet is really gonna come in handy :)

user207421 Over a year ago

@Eliezer And choosing your own 'packet size' is futile. The data will be buffered into the local socket send buffer, sent to the receiver in MTU-sized chunks, and then be buffered into the socket receive buffer. All traces of your 'packet size' disappear before the data even gets onto the wire. Just think of the buffer above as a file reading buffer. 32k is a good size as Peter Lawrey suggested.

Michael Over a year ago

@EJP Someone suggested to me that 4096 was a good buffer size when reading/writing to disk because that's the block size that many file systems use. 32k sounds good too though. :)

user207421 Over a year ago

@Michael That was true about 20 years ago. Disk subsystems have so much caching in them these days that cluster sizes are fairly irrelevant. It's more about getting enough data in one hit to keep the output buffer full. Many years ago I dealt with a printer spooler that printed one line at a time. It hogged the CPU because it got scheduled for every line, so under heavy load the system ran slowly, and the printer ran very slowly. We switched to a spooler that wrote 8192 byte chunks, which only got scheduled 132/8192 as often: the system ran faster and the printer ran much faster.

Eliezer Over a year ago

@EJP So I would initialize the buffer straight to 32k or would I have to do something like socket.setSendBufferSize(32k) first? I use PACKET_SIZE so in case that value ever has to change for whatever reason, it wouldn't be a big deal. Something I carried over from school after having it drilled into me :)

|

EdChum · Accepted Answer · 2012-06-22 22:22:45Z

-1

Umm, realize that your handling of the variable i is not correct..

Iteration 0: i=0
Iteration 1: i=PACKET_SIZE
...
...
Iteration n: i=PACKET_SIZE*n

edited Jun 22, 2012 at 22:22

EdChum

397k204 gold badges836 silver badges583 bronze badges

answered Nov 24, 2011 at 14:39

user173438

1711 gold badge2 silver badges8 bronze badges

2 Comments

Eliezer Over a year ago

I'm not exactly sure what you're saying. Care to explain?

Eliezer Over a year ago

It's exactly the value I would have wanted it to be if I didn't have the correct answer above. Why don't you tell me why that would be a problem instead of being cryptic?

Collectives™ on Stack Overflow

Java Read File Larger than 2 GB (Using Chunking)

2 Answers 2

6 Comments

2 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

6 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Related