I have an array of several Uint8Arrays. Something similar to this:
[Uint8Array(16384), Uint8Array(16384), Uint8Array(16384), Uint8Array(16384), Uint8Array(16384), Uint8Array(8868)]
How do I merge/concatenate/join (not sure what the right term is) them to a single ArrayBuffer?
The key here is that the output I need must be an ArrayBuffer.
You can use the set method. Create a new typed array with all the sizes.
Example:
var arrayOne = new Uint8Array([2,4,8]);
var arrayTwo = new Uint8Array([16,32,64]);
var mergedArray = new Uint8Array(arrayOne.length + arrayTwo.length);
mergedArray.set(arrayOne);
mergedArray.set(arrayTwo, arrayOne.length);
Alternative: Convert your typed array in "normal" arrays. concat it and create a type array of it again.
In your case (solution):
let myArrays = [new Uint8Array(16384), new Uint8Array(16384), new Uint8Array(16384), new Uint8Array(16384), new Uint8Array(16384), new Uint8Array(8868)];
// Get the total length of all arrays.
let length = 0;
myArrays.forEach(item => {
length += item.length;
});
// Create a new array with total length and merge all source arrays.
let mergedArray = new Uint8Array(length);
let offset = 0;
myArrays.forEach(item => {
mergedArray.set(item, offset);
offset += item.length;
});
// Should print an array with length 90788 (5x 16384 + 8868 your source arrays)
console.log(mergedArray);
offset = should be offset +=. And a subjective remark: you could use forEach for the second loop too (mergedArray.set(item,offset);offset+=item.length;)
set documentation: developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/…Just treat it like any normal array:
var concatArray = new Uint8Array([ ...Uint8Array1, ...Uint8Array2, ...Uint8Array3 ]);
... in JS copies each element from the original arrays / object, hence slow. TBH I didn't like the fact that the code we write in JS/TS has to be either long or slow.
There has an easy way:
await new Blob([a1,a2,a3]).arraybuffer();
but slow then:
new Uint8Array(a1.length + a2.length + a3.length)
..set(a1,0)
..set(a2,a1.length)
..set(a3,a1.length+a2.length)
await new Blob([a1,a2,a3]).arrayBuffer()
set is many times faster.By extending @Domske approach, I did create a function that merge N Uint8Arrays into a single one:
function mergeUint8Arrays(...arrays) {
const totalSize = arrays.reduce((acc, e) => acc + e.length, 0);
const merged = new Uint8Array(totalSize);
arrays.forEach((array, i, arrays) => {
const offset = arrays.slice(0, i).reduce((acc, e) => acc + e.length, 0);
merged.set(array, offset);
});
return merged;
}
Typescript:
function mergeUint8Arrays(...arrays: Uint8Array[]): Uint8Array {
const totalSize = arrays.reduce((acc, e) => acc + e.length, 0);
const merged = new Uint8Array(totalSize);
arrays.forEach((array, i, arrays) => {
const offset = arrays.slice(0, i).reduce((acc, e) => acc + e.length, 0);
merged.set(array, offset);
});
return merged;
}
Usage:
const arrayOne = new Uint8Array([2,4,8]);
const arrayTwo = new Uint8Array([16,32,64]);
mergeUint8Arrays(arrayOne, arrayTwo); // Uint8Array(6) [2, 4, 8, 16, 32, 64]
You could have a helper function to concatenate an array of arrays of any length, in their provided order.
This looks nice and clean but it also makes copy of acc on each iteration because of the spread operator:
// arrays are Uint8Array[]
(arrays) => new Uint8Array(arrays.reduce((acc, curr) => [...acc, ...curr], []));
This looks a little longer, but is more performant since acc is not spread on each iteration, but pushed to and returned:
// arrays are Uint8Array[]
(arrays) => {
const flatNumberArray = arrays.reduce((acc, curr) => {
acc.push(...curr);
return acc;
}, []);
return new Uint8Array(flatNumberArray);
};
The above still makes a shallow copy of curr for every array.
The following is a bit better:
// arrays are Uint8Array[]
(arrays) => {
const flatNumberArray = arrays.reduce((acc, curr) => {
acc.push(curr);
return acc;
}, []).flat();
return new Uint8Array(flatNumberArray);
};
A shallow copy happens only once, when calling .flat().
This works for me: single_array is what you want
var arrays = [Uint8Array(16384), Uint8Array(16384), Uint8Array(16384), Uint8Array(16384), Uint8Array(16384), Uint8Array(8868)]
var single_array = concat(arrays)
function concat(arrays) {
// sum of individual array lengths
let totalLength = arrays.reduce((acc, value) => acc + value.length, 0);
if (!arrays.length) return null;
let result = new Uint8Array(totalLength);
// for each array - copy it over result
// next array is copied right after the previous one
let length = 0;
for(let array of arrays) {
result.set(array, length);
length += array.length;
}
return result;
}
Function based on Dominik's answer:
function mergeUInt8Arrays(a1: Uint8Array, a2: Uint8Array): Uint8Array {
// sum of individual array lengths
var mergedArray = new Uint8Array(a1.length + a2.length);
mergedArray.set(a1);
mergedArray.set(a2, a1.length);
return mergedArray;
}
