Javascript reading local file to UInt8Array fast

Question

How to read local binary file to UInt8Array fast. in below code

function readAllBytesAsUInt8Array(path) {
    var req = new XMLHttpRequest();
    req.open("GET", path, false);
    req.overrideMimeType("text/plain; charset=binary-data");
    req.send(null);
    if (req.status !== 200) {
        console.log("error");
        return null;
    }
    var text = req.responseText;
    var resultArray = new Uint8Array(text.length);
    for(var i = 0; i < text.length;i++){
    resultArray[i] = (text[i].charCodeAt() & 255) & 255;
    }
    return resultArray.buffer;
 }

var text = req.responseText; is executed less than a second,meanwhile this part

var resultArray = new Uint8Array(text.length);
for(var i = 0; i < text.length;i++){
    resultArray[i] = (text[i].charCodeAt() & 255) & 255;
}

takes around 10sec for 50MB of binary file, Is there a way to read binary file to UInt8Array faster ?

just wondering, do you really need to do var resultArray = new Uint8Array(text.length);. ? JavaScript doesn't ask for you to predefine the array size for it. just say resultArray = []; and fill it just like you fill yours — Dellirium
– Dellirium, Commented Sep 21, 2017 at 7:48
Why do you not set .responseType of XMLHttpRequest() to "arraybuffer"? — guest271314
– guest271314, Commented Sep 21, 2017 at 7:51
it will be more slower as i push each time to the array it will have to reallocate larger memory for every byte appending — user818117
– user818117, Commented Sep 21, 2017 at 7:51
@guest271314 then what is the right way to convert from arraybuffer to uint8array ? var z = new Uint8Array(buffer) ? — user818117
– user818117, Commented Sep 21, 2017 at 7:52
Have you tried using FileReaderSync() and transferring the object to avoid copying the object? — guest271314
– guest271314, Commented Sep 21, 2017 at 7:52

guest271314 · Accepted Answer · 2017-09-21 07:56:32Z

3

You can set .responseType of XMLHttpRequest() to "arraybuffer", then pass ArrayBuffer instance to new Uint8Array(). Alternatively you can use fetch() and Response.arrayBuffer() or FileReaderSync() within Worker to transfer the ArrayBuffer to main thread without copying the ArrayBuffer.

req.responseType = "arraybuffer";
let buffer = req.response;
let u = new Uint8Array(buffer);

answered Sep 21, 2017 at 7:56

guest271314

1

Sign up to request clarification or add additional context in comments.

4 Comments

user818117 Over a year ago

are req.response and (new Uint8Array(buffer)).buffer the same thing ? as the method returns return resultArray.buffer;

guest271314 Over a year ago

Yes, req.response is an ArrayBuffer

user818117 Over a year ago

let u = new Uint8Array(buffer); throws exceptions as Uint8Array() expects first argument as length

guest271314 Over a year ago

@user818117 No exception is thrown here at either Firefox or Chromium when passing an ArrayBuffer to Uint8Array(). At which browser was an exception thrown? Can you reproduce the exception at jsfiddle jsfiddle.net?

Rob Gates · Accepted Answer · 2017-09-21 07:48:24Z

0

You can use a TextEncoder! TextEncoder takes a stream of code points as input and emits a stream of bytes.

Here is your new code :)

function readAllBytesAsUInt8Array(path) {
    var req = new XMLHttpRequest();
    req.open("GET", path, false);
    req.overrideMimeType("text/plain; charset=binary-data");
    req.send(null);
    if (req.status !== 200) {
        console.log("error");
        return null;
    }
    var text = req.responseText;
    var encoder = new TextEncoder("utf-8");
    var resultArray = encoder.encode(text);
    return resultArray.buffer;
 }

See how that works, should be a lot faster.

answered Sep 21, 2017 at 7:48

Rob Gates

2756 silver badges14 bronze badges

1 Comment

user818117 Over a year ago

textencoder doesnt work on some browsers like IE,Safari, meanwhile i tested it has the similar performance with above code

Collectives™ on Stack Overflow

Javascript reading local file to UInt8Array fast

2 Answers 2

4 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

4 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related