Continuously write to file in Java efficiently

Question

I have a program that's generating a bunch of output I want to to save to a text file. I don't necessarily know when the program is going to end, and I want to have as little impact on performance as possible. Right now I am buffering my lines into a Deque, then dumping that entire Deque into the file every 100 lines (number was arbitrary). I currently using the Files.write() method.

I would ideally like to open the file when the program begins and close the file when the program ends, but I don't know when the program is going to end, so I might fail to close the file. Is that even an issue? Should I do the buffering myself with the Deque, or should I use BufferedWriter or something else similar?

Erwin Bolwidt · Accepted Answer · 2019-12-05 03:00:21Z

1

If you were content to risk losing the last 100 lines from a Deque when the application crashes, then you can also use a BufferedWriter and flush() it every 100 lines.

Flushing it has the effect of writing it to the operating system.

Technically, the operating system can also keep data in memory, but that also happens with a FileOutputStream without a buffer. FileOutputStream.getFD().sync() forces the OS buffers to disk as well, if you are concerned about losing power or about an OS crash.

answered Dec 5, 2019 at 3:00

Erwin Bolwidt

31.3k15 gold badges59 silver badges81 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

multitaskPro Over a year ago

Do you think the BufferedWriter is the preferred solution to using the Deque? Also yeah I am willing to lose one "buffer"''s worth of data

Erwin Bolwidt Over a year ago

As always, it depends. If you keep them in a Deque, you can still refer to the data as lines. You might want to do something with the data before writing, such as removing duplicate lines, etc. You can't do that anymore when you have lost the abstraction of lines by writing to a Writer. On the other hand, using a Writer is less complex so it makes your code easier to maintain, reason about, optimize, etc. so if you don't need the complexity of the Deque then I wouldn't use it.

multitaskPro Over a year ago

okay sounds good. Related question, is there any danger to leaving a file open for a long time? Assuming I eventually close it. In this case, a long time means a couple hours

Erwin Bolwidt Over a year ago

No, not an issue at all. Most OSes keep your executable's executable file open for the entire lifetime of a process.

Collectives™ on Stack Overflow

Continuously write to file in Java efficiently

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related