4

I have a function in my Main thread which will write some data to disk. I don't want my Main thread to stuck (High Latency of Disk I/O) and creating a new thread just to write is an overkill. I have decided to use ExecutorService. 

ExecutorService executorService = Executors.newFixedThreadPool(3);

   Future future = executorService.submit(new Callable<Boolean>() {
    public Boolean call() throws Exception {
      logger.log(Level.INFO, "Writing data to disk");
      return writeToDisk();
    }
  });

writeToDisk is the function which will write to disk

Is it a nice way to do? Could somebody refer better approach if any?

UPDATE: Data size will be greater than 100 MB. Disk bandwidth is 40 MBps, so the write operation could take couple of seconds. I don't want calling function to stuck as It has to do other jobs, So, I am looking for a way to schedule Disk I/O asynchronous to the execution of the calling thread.

I need to delegate the task and forget about it!

Improve this question
2
  • This question might be too broad as you are still very unspecific about your exact requirements. We don't know anything about your disk configuration and the size of the chunks being written for example. Commented Nov 13, 2014 at 8:21
  • Are you aware of AsynchronousFileChannel? Commented May 8, 2020 at 10:23

3 Answers 3

Reset to default
10

Your code looks good anyways I've used AsynchronousFileChannel from new non-blocking IO. The implementation uses MappedByteBuffer through FileChannel. It might give you the performance which @Chris stated. Below is the simple example:

public static void main(String[] args) {
    String filePath = "D:\\tmp\\async_file_write.txt";
    Path file = Paths.get(filePath);
    try(AsynchronousFileChannel asyncFile = AsynchronousFileChannel.open(file,
                        StandardOpenOption.WRITE,
                        StandardOpenOption.CREATE)) {

        asyncFile.write(ByteBuffer.wrap("Some text to be written".getBytes()), 0);
    } catch (IOException e) {
        e.printStackTrace();
    }
}
Improve this answer
Sign up to request clarification or add additional context in comments.

3 Comments

+1 for remembering about this api. Do however measure its performance. The last time that I investigated AsynchronousFileChannel, it was implemented under the covers using a separate thread. And is thus equivalent (but perhaps more convenient) to creating the thread oneself. But that is of course open to change with versions of the JVM.
hmmmm, if it is async, doing a try with resource such that it closes before it may have completed seems a little off? The main() method should exit before the Future returned is completed (otherwise, it is just synchronous)
@DeanHiller I guess that's a good reason not to use try-with-resource in that case: an AsynchronousFileChannel could be considered as a lower-level object, when a "resource" would be higher-level (thus with easier accessors).
3

There are two approaches that I am aware of, spin up a thread (or use a pool of threads as you have) or memory map a file and let the OS manage it. Both are good approaches, memory mapping a file can be as much as 10x faster than using Java writers/streams and it does not require a separate thread so I often bias towards that when performance is key.

Either way, as a few tips to optimize disk writing try to preallocate the file where possible. Resizing a file is expensive. Spinning disks do not like seeking, and SSDs do not like mutating data that has been previously written.

I wrote some benchmarks to help me explore this area awhile back, feel free to run the benchmarks yourself. Amongst them is an example of memory mapping a file.

Improve this answer

2 Comments

Sorry I don't have any experience with memory mapped writes, does it mean that I will simply link a data structure with a file and keep writing to it? OS(or JVM?) will simply write it into disk at its leisure? If yes then It seems very convenient.
@MangatRai that is correct. A memory mapped file hooks into the memory management parts of the OS and hardware, making writing to memory equivalent to writing to disk. It does mean that you have to be happy working with bytes, and a corrupt disk will result in killing the entire process (no IOException). However the advantages are speed, convenience and the ability to survive the death of the process and still keep the data and fast interprocess comms (not that you need that here).
3

I would agree with Õzbek to use a non-blocking approach. Yet, as pointed by Dean Hiller, we cannot close the AsynchronousFileChannel before-hand using a try with resources because it may close the channel before the write has completed.

Thus, I would add a CompletionHandler on asyncFile.write(…,new CompletionHandler< >(){…} to track completion and close the underlying AsynchronousFileChannel after conclusion of write operation. To simplify the use I turned the CompletionHandler into a CompletableFuture which we can easily chain with a continuation to do whatever we want after write completion. The final auxiliary method is CompletableFuture<Integer> write(ByteBuffer bytes) which returns a CompletableFuture of the final file index after the completion of the corresponding write operation. I placed all this logic in an auxiliary class AsyncFiles that can be used like this:

Path path = Paths.get("output.txt")
AsyncFiles
    .write(path, bytes)
    .thenAccept(index -> /* called on completion from a background thread */)
Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.