Java write to file using multi-threading doesn't speedup execution

Question

I have asked this question some time ago to get an idea of speeding up image writing to file.I basically have an OpenGL app.At the end of each render loop I save the frame buffer into image file.

   fbo.readFrame();

   glReadPixels(0, 0, _viewportWidth, _viewportHeight, GL_RGBA, GL_UNSIGNED_BYTE, _data);
   _data.rewind(); //ByteBuffer
   new TImageExporter(ImageExporter.TGA, "renderings/", _data, _viewportWidth, _viewportHeight, true, _frameCount++).run();

The TImageExporter extends Thread and inside the "run()" method I execute writing to File. To my surprise The render loop for 50 frames takes almost the same time as if I use a single thread version of the Image Exporter. (3293 milliseconds -multi-threaded and 3437 milliseconds using single threaded).What do I do wrong here? That is the code inside TImageExporter:

public void export() {
    _pixels = new int[_width * _height];
    int bindex;
    int plenght = _pixels.length;
    // convert RGB data in ByteBuffer to integer array
    for (int i = 0; i < plenght; ++i) {
        bindex = i * 4;  //use 3 for RGB
          //// here write the pixels to RGBA/////
         ............
         .......................

    }

    _image.flush();
    _image.setRGB(0, 0, _width, _height, _pixels, 0, _width);
    _image = ImageUtils.flipY(_image);

    renderByFormatType();

}

private void renderByFormatType() {


    String formattedString = String.format(_formatString, _frameCount);

    if (_exportType.equals(TGA)) {
        try {
           writeTGA(_image, new File(_renderPath + "screenshot_test" + formattedString + ".tga"), _flipX);
        } catch (IOException ex) {
            Logger.getLogger(TImageExporter.class.getName()).log(Level.SEVERE, null, ex);
        }
    }




}

@Override
public void run() {


        export();



}

UPDATE: People have asked here if I write to the same file.No, each thread writes to a completely new file.

UPDATE1: Set a global static variable that holds array of BufferedImage.Now each new TImageExporter writes the image data to that array to a different index.All I got is 3038 milliseconds instead of 3437 when writing directly to Disk.

do you copy that _data ByteBuffer before passing it to the thread? — jtahlborn
– jtahlborn, Commented Aug 1, 2012 at 17:52
also, have you run your code through a profiler to see where the bottleneck is? — jtahlborn
– jtahlborn, Commented Aug 1, 2012 at 17:53

kosa · Accepted Answer · 2012-08-01 17:43:11Z

3

Multi-threading will not speed up file transfer because:

making a non-thread safe file write will overwrite some of the information the other threads are writing when 1 thread goes to save it.

Disk I/O is your bottle neck. The best thing to do it write what you want to into a memorystream (someone edit and give documentation i can't find it real quick) and allow it to write to the disk.. this will allow constant disk i/o without worrying about threads

edited Aug 1, 2012 at 17:43

kosa

66.7k15 gold badges135 silver badges170 bronze badges

answered Aug 1, 2012 at 17:40

Matt Westlake

3,6818 gold badges44 silver badges83 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Michael IV Over a year ago

So you say it is better for me to write it into ByteBuffers and at the end of rendering save those to files.And if I have like 10000 frames to write? Can it blow up the memory ?Can I have a static or global array to hold the BufferedImage instances which are created in each thread inside TImageWriter?

Matt Westlake Over a year ago

yes it can blow up the memory.. Remember your constraints an Int is 4 bytes, a char is 2 bytes. The more you can write to the disk at one time the better. If you want, make a program that generates a random int, and saves it to a file. Keep a count and see how many times it does it in a minute. Then generate 1000 ints (4K worth of memory) and save those to the file for 1 minute and you'll be surprised how much better your disk I/O is

Byter · Accepted Answer · 2012-08-01 17:40:01Z

2

Even though your code is multi threaded ; since all you Threads are trying to access the same file..it wont make much difference..because in this scenario even though lot of threads are running; at one instant all except 1 will be waiting to acquire write access to file.

answered Aug 1, 2012 at 17:40

Byter

1,1321 gold badge7 silver badges12 bronze badges

2 Comments

Michael IV Over a year ago

They don't access the same file.Each one writes into new file."_frameCount++" takes care of it .

Matt Westlake Over a year ago

@MichaelIV They still use the same disk.. Disk I/O is your enemy here. The more files you try to write to, the more down time going from file to file writing to the disk.

user207421 · Accepted Answer · 2012-08-01 22:27:45Z

-1

Why do you think it will be faster? The disk isn't multithreaded.

answered Aug 1, 2012 at 22:27

user207421

312k45 gold badges324 silver badges493 bronze badges

2 Comments

Matt Westlake Over a year ago

this makes no since at all... threads don't even exist when it comes to physical drives.

user207421 Over a year ago

@MattWestlake Exactly my point. Multiple threads in the CPU doesn't extend as far as the disk, so why would they do the I/O any faster. The multiple threads are sequentialized at the disk, if not before. There is no point to them.

Collectives™ on Stack Overflow

Java write to file using multi-threading doesn't speedup execution

3 Answers 3

2 Comments

2 Comments

2 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

2 Comments

2 Comments

2 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related