Java 8 - Most effective way to merge List<byte[]> to byte[]

Question

I have a library that returns some binary data as list of binary arrays. Those byte[] need to be merged into an InputStream.

This is my current implementation:

public static InputStream foo(List<byte[]> binary) {
    byte[] streamArray = null;
    binary.forEach(bin -> {
        org.apache.commons.lang.ArrayUtils.addAll(streamArray, bin);
    });
    return new ByteArrayInputStream(streamArray);
}

but this is quite cpu intense. Is there a better way?

Thanks for all the answers. I did a performance test. Those are my results:

Function: 'NicolasFilotto' => 68,04 ms average on 100 calls
Function: 'NicolasFilottoEstSize' => 65,24 ms average on 100 calls
Function: 'NicolasFilottoSequenceInputStream' => 63,09 ms average on 100 calls
Function: 'Saka1029_1' => 63,06 ms average on 100 calls
Function: 'Saka1029_2' => 0,79 ms average on 100 calls
Function: 'Coco' => 541,60 ms average on 10 calls

I'm not sure if 'Saka1029_2' is measured correctly...

this is the execute function:

private static double execute(Callable<InputStream> funct, int times) throws Exception {
    List<Long> executions = new ArrayList<>(times);

    for (int idx = 0; idx < times; idx++) {
        BufferedReader br = null;
        long startTime = System.currentTimeMillis();
        InputStream is = funct.call();
        br = new BufferedReader(new InputStreamReader(is));
        String line = null;
        while ((line = br.readLine()) != null) {}
        executions.add(System.currentTimeMillis() - startTime);
    }

    return calculateAverage(executions);
}

note that I read every input stream

those are the used implementations:

public static class NicolasFilotto implements Callable<InputStream> {

    private final List<byte[]> binary;

    public NicolasFilotto(List<byte[]> binary) {
        this.binary = binary;
    }

    @Override
    public InputStream call() throws Exception {
        ByteArrayOutputStream baos = new ByteArrayOutputStream();
        for (byte[] bytes : binary) {
            baos.write(bytes, 0, bytes.length);
        }
        return new ByteArrayInputStream(baos.toByteArray());
    }

}

public static class NicolasFilottoSequenceInputStream implements Callable<InputStream> {

    private final List<byte[]> binary;

    public NicolasFilottoSequenceInputStream(List<byte[]> binary) {
        this.binary = binary;
    }

    @Override
    public InputStream call() throws Exception {
        return new SequenceInputStream(
                Collections.enumeration(
                        binary.stream().map(ByteArrayInputStream::new).collect(Collectors.toList())));
    }

}

public static class NicolasFilottoEstSize implements Callable<InputStream> {

    private final List<byte[]> binary;
    private final int lineSize;

    public NicolasFilottoEstSize(List<byte[]> binary, int lineSize) {
        this.binary = binary;
        this.lineSize = lineSize;
    }

    @Override
    public InputStream call() throws Exception {
        ByteArrayOutputStream baos = new ByteArrayOutputStream(binary.size() * lineSize);
        for (byte[] bytes : binary) {
            baos.write(bytes, 0, bytes.length);
        }
        return new ByteArrayInputStream(baos.toByteArray());
    }

}

public static class Saka1029_1 implements Callable<InputStream> {

    private final List<byte[]> binary;

    public Saka1029_1(List<byte[]> binary) {
        this.binary = binary;
    }

    @Override
    public InputStream call() throws Exception {
        byte[] all = new byte[binary.stream().mapToInt(a -> a.length).sum()];
        int pos = 0;
        for (byte[] bin : binary) {
            int length = bin.length;
            System.arraycopy(bin, 0, all, pos, length);
            pos += length;
        }
        return new ByteArrayInputStream(all);
    }

}

public static class Saka1029_2 implements Callable<InputStream> {

    private final List<byte[]> binary;

    public Saka1029_2(List<byte[]> binary) {
        this.binary = binary;
    }

    @Override
    public InputStream call() throws Exception {
        int size = binary.size();
        return new InputStream() {
            int i = 0, j = 0;

            @Override
            public int read() throws IOException {
                if (i >= size) return -1;
                if (j >= binary.get(i).length) {
                    ++i;
                    j = 0;
                }
                if (i >= size) return -1;
                return binary.get(i)[j++];
            }
        };
    }

}

public static class Coco implements Callable<InputStream> {

    private final List<byte[]> binary;

    public Coco(List<byte[]> binary) {
        this.binary = binary;
    }

    @Override
    public InputStream call() throws Exception {
        byte[] streamArray = new byte[0];
        for (byte[] bin : binary) {
            streamArray = org.apache.commons.lang.ArrayUtils.addAll(streamArray, bin);
        }
        return new ByteArrayInputStream(streamArray);
    }

}

You could use an InputStream with reads from the List<Byte[]> without producing a byte[] of everything first. — Peter Lawrey
– Peter Lawrey, Commented Oct 6, 2016 at 8:22
What about my second proposal based on SequenceInputStream ? — Nicolas Filotto
– Nicolas Filotto, Commented Oct 6, 2016 at 14:10
for micro-benchmark, use JMH openjdk.java.net/projects/code-tools/jmh — Nicolas Filotto
– Nicolas Filotto, Commented Oct 6, 2016 at 14:14
@NicolasFilotto I've edited the answer to include the InputStream variant. — jaecktec
– jaecktec, Commented Oct 13, 2016 at 12:43

Community · Accepted Answer · 2017-05-23 11:48:21Z

5

You could use a ByteArrayOutputStream to store the content of each byte arrays of your list but to make it efficient, we would need to create the instance of ByteArrayOutputStream with an initial size that matches the best as possible with the target size, so if you know the size or at least the average size of the array of bytes, you should use it, the code would be:

public static InputStream foo(List<byte[]> binary) {
    ByteArrayOutputStream baos = new ByteArrayOutputStream(ARRAY_SIZE * binary.size());
    for (byte[] bytes : binary) {
        baos.write(bytes, 0, bytes.length);
    }
    return new ByteArrayInputStream(baos.toByteArray());
}

Another approach would be to use SequenceInputStream in order to logically concatenate all the ByteArrayInputStream instances representing one element of your list, as next:

public static InputStream foo(List<byte[]> binary) {
    return new SequenceInputStream(
        Collections.enumeration(
            binary.stream().map(ByteArrayInputStream::new).collect(Collectors.toList())
        )
    );
}

The interesting aspect of this approach is the fact that you have no need to copy anything, you only create instances of ByteArrayInputStream that will use the byte array as it is.

To avoid collecting the result as a List which has a cost especially if your initial List is big, you can directly call iterator() as proposed by @Holger, then we will simply need to convert an iterator into an enumeration which can be done with IteratorUtils.asEnumeration(iterator) from Apache Commons Collection, the final code would then be:

public static InputStream foo(List<byte[]> binary) {
    return new SequenceInputStream(
        IteratorUtils.asEnumeration(
            binary.stream().map(ByteArrayInputStream::new).iterator()
        )
    );
}

edited May 23, 2017 at 11:48

CommunityBot

11 silver badge

answered Oct 6, 2016 at 8:25

Nicolas Filotto

45.2k11 gold badges97 silver badges129 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

jaecktec Over a year ago

is there a way to utilize java8 parralel?

Holger Over a year ago

@Coco: with the SequenceInputStream approach there is no need for any parallel processing as there’s no actual work to perform.

Holger Over a year ago

@Nicolas Filotto: the workload per list element is so tiny that there won’t be any benefit, regardless of how big the list is. The point is, Collectors.toList() never benefits from parallel processing as the workload of the merging step(s) is exactly as big as the maximum savings of the previous concurrent accumulation(s). The only thing that benefits from parallel processing then, is the concurrent execution of the lightweight ByteArrayInputStream::new. That won’t compensate the thread management overhead.

Holger Over a year ago

If you have large lists and want improve performance, don’t collect into a List at all. Instead, use binary.stream().map(ByteArrayInputStream::new).iterator() and implement a lightweight Enumeration wrapping the returned Iterator…

score 4 · Accepted Answer · 2016-10-06 09:01:37Z

4

Try this.

public static InputStream foo(List<byte[]> binary) {
    byte[] all = new byte[binary.stream().mapToInt(a -> a.length).sum()];
    int pos = 0;
    for (byte[] bin : binary) {
        int length = bin.length;
        System.arraycopy(bin, 0, all, pos, length);
        pos += length;
    }
    return new ByteArrayInputStream(all);
}

Or

public static InputStream foo(List<byte[]> binary) {
    int size = binary.size();
    return new InputStream() {
        int i = 0, j = 0;
        @Override
        public int read() throws IOException {
            if (i >= size) return -1;
            if (j >= binary.get(i).length) {
                ++i;
                j = 0;
            }
            if (i >= size) return -1;
            return binary.get(i)[j++];
        }
    };
}

edited Oct 6, 2016 at 9:01

answered Oct 6, 2016 at 8:37

user4910279

1 Comment

user4910279 Over a year ago

@NicolasFilotto But this allocates byte array only once.

Collectives™ on Stack Overflow

Java 8 - Most effective way to merge List<byte[]> to byte[]

2 Answers 2

4 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related