Convert UTF8 to int16 in javascript

Question

I read a binary file (of 16bit integers) in my node.js code and send it to client using connection.sendUTF(data) of websocket. At the javascript code of the client side I receive an array of UTF8 and would like to convert back every pair to an int16 ( numbers are unsigned, if makes it simple).

I came up with this function to be used at the client side and was wondering if it is correct.

function char2BigEndian16(hiChar, lowChar){
    return  ((lowChar.charCodeAt(0)& 0x00FF) << 8) | (hiChar.charCodeAt(0) & 0xFF);
}

and here is how I call it to convert first to elements of array data to a 2-byte integer

 var  i16 = char2BigEndian16(data[1], data[0]);

If this help here is my node.js code snipped:

var fs = require('fs');
    filename = "test.txt";
    fs.readFile(filename, 'utf8', function(err, data) {
            if (err) {
                console.log('Error!!');
                return console.log(err);
            }
        connection.sendUTF(data);

        });

and here is how I deal with the received data on the client side

connection.onmessage = function(e) {
                console.log(e.data.length);
}

X Y problem detected - meta.stackexchange.com/questions/66377/what-is-the-xy-problem — Danack
– Danack, Commented Dec 4, 2013 at 2:12

Guffa · Accepted Answer · 2013-12-04 00:42:09Z

1

No, that won't work reliably. An UTF-8 file isn't a bunch of distinct 8-bit values, each character can use one or more bytes. When you read the file as UTF-8, some values will be interpreted as multi-byte groups and be converted into a single character. You can only reliably read a file as UTF-8 that actually was created as an UTF-8 text file.

You need to read the file without an encoding, so that you get back the data unchanged:

fs.readFile(filename, function(err, data) {

Now you can send the data to the client, and each byte will be a byte from the original file.

answered Dec 4, 2013 at 0:42

Guffa

703k111 gold badges760 silver badges1k bronze badges

Sign up to request clarification or add additional context in comments.

10 Comments

broofa Over a year ago

Hmm, good points. Could OP use String.fromCharCode() to create a UCS-2 encoded string and send that via sendUTF? ('Not sure if surrogate the surrogate ranges come into play here though.) Seems like he should either do that or use sendBytes() to send the raw data.

Guffa Over a year ago

@broofa: Well, sending bytes as UTF-16 would maybe work, but then the stream gets twice as big as the original file, and still needs decoding on the client side to patch 8-bit values (converted from 16-bit character codes) into 16-bit values.

broofa Over a year ago

I would assume that the server-side sendUTF API would convert the JS UCS-2 string to UTF8. And since this is sent as a UTF8 string, I assume it would manifest as a [JS UCS-2] String in the websocket message event - i.e. OP wouldn't have to do anything special to decode other than messageData.charCodeAt(someIndex) to get each int16 value. No?

Guffa Over a year ago

@broofa: You mean reading the file as UCS-2 although it's not a text file at all, and sending it as text... Sounds like there could be several ways that the data gets corrupted, like for example character code normalizing.

Guffa Over a year ago

@Cgraphics: I came to the same conclusion. If it doesn't have a length it has to be a Blob, and has a size instead. You can use a FileReader to read from the Blob: developer.mozilla.org/en-US/docs/Web/API/Blob

|

broofa · Accepted Answer · 2013-12-04 01:24:47Z

0

As noted by @Guffa, you're probably starting out with bad data. If your data file is a raw binary file of 16-bit values, then reading that as UTF8 encoded text is pretty much guaranteed to fail. Instead, omit the encoding argument so you get a Buffer of binary data on the server side, and use websocket's sendBytes method to transmit it as raw data.

On the client side your websocket's message event should then provide a data property that is a ArrayBuffer (or possibly a Blob) of binary data, which you could then convert to int16 values via something like ...

function arrayByteToInt16(byteArray) {
  var ints = [];
  for (var i = 0; i < byteArray.length; i += 2) {
    // Note: assuming Big-Endian here
    ints.push((byteArray[i] << 8) | (byteArray[i+1]));
  }
  return ints;
}

[Edit: complete rewrite of answer]

edited Dec 4, 2013 at 1:24

answered Dec 4, 2013 at 0:40

broofa

38.3k13 gold badges76 silver badges73 bronze badges

1 Comment

C graphics Over a year ago

For some reason giving "e.data" as input to this function would all return 0! as well e.data.length return error ( e.data.lengh is not a function)!

Collectives™ on Stack Overflow

Convert UTF8 to int16 in javascript

2 Answers 2

10 Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

10 Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related