14

I'm happy to see the std::u16string and std::u32string in C++11, but I'm wondering why there is no std::u8string to handle the UTF-8 case. I'm under the impression that std::string is intended for UTF-8, but it doesn't seem to do it very well. What I mean is, doesn't std::string.length() still return the size of the string's buffer rather than the number of characters in the string?

So, how is the length() method of the standard strings defined for the new C++11 classes? Do they return the size of the string's buffer, the number of codepoints, or the number of characters (assuming a surrogate pair is 2 code points, but one character. Please correct me if I'm wrong)?

And what about size(); isn't it equal to length()? See http://en.cppreference.com/w/cpp/string/basic_string/length for the source of my confusion.

So, I guess, my fundamental question is how does one use std::string, std::u16string, and std::u32string and properly distinguish between buffer size, number of codepoints, and number of characters? If you use the standard iterators, are you iterating over bytes, codepoints, or characters?

Improve this question
2
  • 3
    std::string works as well for utf8 as u16string does for utf16: it handles elements of the corresponding type, and doesn't deal with characters that are represented by a sequence of more than one element. Commented Sep 3, 2012 at 16:30
  • Go here: utf8everywhere.org/#myth.strlen Commented Sep 4, 2012 at 6:28

3 Answers 3

Reset to default
18

u16string and u32string are not "new C++11 classes". They're just typedefs of std::basic_string for char16_t and cha32_t types.

length is always equal to size for any basic_string. It is the number of T's in the string, where T is the template type for the basic_string.

basic_string is not Unicode aware in any way, shape, or form. It has no concept of codepoints, graphemes, Unicode characters, Unicode normalization, or anything of the kind. It is simply a ordered sequence of Ts. The only thing that is Unicode-aware about u16string and u32string is that they use the type returned by u"" and U"" literals. Thus, they can store Unicode-encoded strings, but they do nothing that requires knowledge of said encoding.

Iterators iterate over elements of T, not "bytes, codepoints, or characters". If T is char16_t, then it will iterate over char16_ts. If the string is UTF-16-encoded, then it is iterating over UTF-16 code units, not Unicode codepoints or bytes.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

And code unit != code point. They are two different concepts. Just for later reference because I didn't know that...
1

All the string types do the same thing: they hold a sequence of elements, each of whose type is the character type for the string. length() and size() both return the number of elements. Iterators iterator over elements. Higher-level analysis, such as figuring out the number of characters, require much more complex calculations.

Improve this answer

Comments

0

Currently there is nothing built into the standard to distinguish between code units, codepoints or individual bytes. However, there do seem to be some things in the works to deal with this sort of thing. Depending on what the standards committee decides, it may be part of TR2 or the next standard.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.