10

I have Base64 encoded data that is in UTF-16 I am trying to decode the data but most libraries only support UTF-8. I believe I have to drop the null bites but I am unsure how.

Currently I am using David Chambbers Polyfill for Base64, but I have also tried other libraries such as phpjs.org, none of which support UTF-16.

One thing to point out is on Chrome the atob method works with out problem, Firefox I get results described here, and in IE I am only returned the first character.

Any help is greatly appreciated

14
  • 2
    I'm a bit confused as to what you are trying to do. You have Base64 data, but it's in UTF-16? I'm not sure, but I have a hunch that you can just chop off every other byte and be left with what you need. Commented Jan 29, 2013 at 21:12
  • 1
    Convertions can be easily done using PHP : php.net/manual/en/function.mb-convert-encoding.php Commented Jan 29, 2013 at 21:14
  • 1
    @Brad chopping off every other byte won't do if the data is non-ascii. Commented Jan 29, 2013 at 21:16
  • 2
    @sdespont, What does PHP have to do with JavaScript?... Commented Jan 29, 2013 at 21:23
  • 2
    @Brad I guess it's base64-encoded UTF-16 (which could contain any text), not UTF-16-encoded base64 (which would be kinda wasteful). Commented Jan 29, 2013 at 21:25

1 Answer 1

28

You want to decode UTF-16, not convert to UTF-8. Decoding means that the result is a string of abstract characters. Of course there is an internal encoding for strings as well, UTF-16 or UCS-2 in javascript, but that's an implementation detail.

With strings the goal is that you don't have to worry about encodings but just about manipulating characters "as they are". So you can write string methods that don't need to decode input at all. Of course there are many edge cases where this falls apart.

You cannot decode utf-16 just by removing nulls. I mean this will work fine for the first 256 code points of unicode, but you will get garbage when any of the other ~110000 characters in unicode are used. You cannot even get the most popular non-ASCII characters like em dash or any smart quotes working.

Also, looking at your example, it looks like UTF-16LE.

//Braindead decoder that assumes fully valid input
function decodeUTF16LE( binaryStr ) {
    var cp = [];
    for( var i = 0; i < binaryStr.length; i+=2) {
        cp.push( 
             binaryStr.charCodeAt(i) |
            ( binaryStr.charCodeAt(i+1) << 8 )
        );
    }

    return String.fromCharCode.apply( String, cp );
}

var base64decode = atob; //In chrome and firefox, atob is a native method available for base64 decoding

var base64 = "VABlAHMAdABpAG4AZwA";
var binaryStr = base64decode(base64);
var result = decodeUTF16LE(binaryStr);

Now you can even get smart quotes working:

var base64 = "HCBoAGUAbABsAG8AHSA="
var binaryStr = base64decode(base64);
var result = decodeUTF16LE(binaryStr);
//"“hello”"
Sign up to request clarification or add additional context in comments.

1 Comment

decodeUTF16LE feels like exacly what I need to get rid of \u001f, but using it I get unexpected results; "and\x1Fnegative" (mean to be "and negative") is converted to "湡ὤ敮慧楴敶".* What am I missing?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.