2

I know they say premature optimization is the root of all evil... but it's about that time.

I have a slow, but working procedure that performs the following operations:

  1. Read chunk (sequential) from file.input
  2. Transform chunk
  3. Write (append) transformed chunk to file.output

file.input and file.output end up being in the same size ballpark (10-100+ GB). A chunk is typically about 10K. The transform step is just a conversion between proprietary formats. For discussion's sake, we can consider it to be computationally on par with a real-time compression algorithm.

These steps are currently done in a single thread.

My question: How do I make this better performing?

I realize this will never get "fast" based on the pure volume of data being processed, but I have to believe there are some relatively simple and standard techniques to get this faster.

I've tried adding buffering to reading step (1). That is, reading in much larger blocks than the chunk size and reading from the buffer. This helped. However, I'm a bit stuck on the if there's anything that can be done for transform step (2) and the appending (3).

According to Resource Monitor, my CPU usage fluctuates between 30-45% and Disk I/O has some sustained periods of low usage.

I'm using C# with a a bunch of P/invoke interop to native libraries.

Improve this question

5 Answers 5

Reset to default
2

First off you should verify that all steps take a large portion of the spent time - if the bottleneck turns out to be one step only, we're looking at the wrong picture here.

Disk I/O may take time during which the CPU is available for other tasks, so a multithreaded solution could improve performance. Perhaps you could use one thread to keep filling a buffer up to 10 chunks of processable data, one that processes any chunk loaded by the first, and another that simply writes any chunk that has finished processing. A pair of AutoResetEvents could then inform each thread that the previous thread has finished processing one or more chunks.

Improve this answer
Sign up to request clarification or add additional context in comments.

Comments

2

First, don't add threads to this scenario, they will do nothing to better your performance. (of course, if your application has an UI interface you could add a thread to perform your intensive work and let the user continue with other tasks)

Second, try to minimize the IO on disk. Read as many bytes you can and write all togheter. Third, probably your best bet is to optimize your transformation work.

Improve this answer

4 Comments

+1. I would guess that optimize transformation is last thing to do, but your guess as good as mine without measurements.
Wouldn't the single thread wait for disk I/O to complete? When talking 100GB files the disk I/O portion may be significant?
Well, but the OP says that Disk Usage is low and he reads bunches of 10K instead the CPU is not irrelevant. Adding threads will result in a far more complex scenario of synchronizations between the first step and the second step. And a thread adds to the CPU usage
+1 it should be easy measuring total transformation time versus total time spent
1

Your main issue will be the I/O. The fact that your alternating between 2 files makes things a little worse.

  • buffer both Read and Write. Large actions are better and in this case fewer actions is also better.

Since there also is a CPU intensive part, I would try using threads. To minimize waiting I would use a pipeline: ReaderThread -> Queue -> ProcescessingThread -> Queue -> WriterThread

Improve this answer

5 Comments

Shouldn't writes be already buffered? (that is, I'm not explicitly flushing each time a chunk is written?)
Yes, but normally not by much. It might be enough to use one of the constructors that takes a bufferSize parameter. On both streams.
I'm going to test out this approach approach shortly; is there a name for this pattern?
Pipeline or 2x Producer/Consumer. You'll want a ConcurrentQueue or better a BlockingQueu. Limit the queues so that there is always a supply of work-to-be-done w/o filling up memory.
+1 & answer. This approach shaved a significant amount of execution time
1

If you are on .NET 4, you could try MemoryMappedFile and MemoryMappedViewAccessor

Improve this answer

1 Comment

And how exactly would that speed things up? Aside from the problems of emulating a stream.
1

Measure

  • 1 (simple sequential read to nowhere),
  • 3 (simple sequential write of random bytes)
  • 1+3 (read and immediately copy in somewhat reasonably sized blocks 16K-64k-512k-1Mb)
  • 2 (just calculations)

Than look at your numbers and see if overlapping can bring any sizable benefits. 45% CPU load is not too bad, so you may be better off just keep it as is.

The best you can get is Max(1, 2, 3) so plan you optimizations accordingly.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.