how to test if a string is a valid UTF16 string?

Question

I am using mongodb and javascript to do some string processing. Now I got some error like:

Sun May 23 07:42:20 Assertion failure JS_EncodeCharacters( _context , s , srclen , dst , &len) scripting/engine_spidermonkey.cpp 152 0x80f4f7e 0x80f8794 0x811525b 0x811a953 0x8119fc4 0x8111bc5 0x81b408e 0x81c4ee7 0x81b4a10 0x817a881 0x817a7d8 0x817a6e2 0x811e1bb 0x80a777b 0x80a8f8a 0xb7cb2455 0x80a37a1 mongodb-linux-i686-1.4.2/bin/mongo(_ZN5mongo12sayDbContextEPKc+0xfe) [0x80f4f7e]

After doing some google, I find that JS_EncodeCharacters return false if the input is not a valid UTF16 string. (if spidermonkey is build with UTF-8 enabled)

So I was wondering how to test if the input string if a proper UTF16 string? so I can skip such kind of string to avoid problem ...

Thanks

Although stackoverflow.com/questions/16868415/… is newer, maybe this question should be marked as duplicate, because that other question tells how to validate UTF-16 (encodeURIComponent throws if invalid) and answers how to sanitize. — T S
– T S, Commented Jul 5, 2018 at 19:35

Dean Harding · Accepted Answer · 2010-05-23 08:59:27Z

2

This part of the UTF-16 FAQ describes the sequences of invalid characters:

The two values FFFE₁₆ and FFFF₁₆ as well as the 32 values from FDD0₁₆ to FDEF₁₆ represent noncharacters. They are invalid in interchange, but may be freely used internal to an implementation. Unpaired surrogates are invalid as well, i.e. any value in the range D800₁₆ to DBFF₁₆ not followed by a value in the range DC00₁₆ to DFFF₁₆, or any value in the range DC00₁₆ to DFFF₁₆ not preceded by a value in the range D800₁₆ to DBFF₁₆.

If you're doing this in Javascript, I'm not sure it'll be all that easy to test for this, though...

answered May 23, 2010 at 8:59

Dean Harding

72.9k13 gold badges151 silver badges180 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

superb Over a year ago

Yeah .... That should be a way ... but seems a little hard :) hopefully there is simpler way to do that ...

Константин Ван · Accepted Answer · 2023-05-26 12:58:08Z

1

The String.prototype.isWellFormed() method returns a boolean indicating whether the string does not contain any lone (unmatched) surrogates.

answered May 26, 2023 at 12:58

Константин Ван

15.4k8 gold badges81 silver badges90 bronze badges

Collectives™ on Stack Overflow

how to test if a string is a valid UTF16 string?

2 Answers 2

1 Comment

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related