How convert to utf8 string in c++

Question

I have a string output that ins not necessarily valid utf8. I have to pass it to a method only accepting valid utf8 strings.
Therefore I need to convert output to the closest valid utf8 string removing invalid bytes or parts. How can I do that in c++? I would like not to use a 3rd party library.

I think this is not safe. If your string is not UTF-8, the only safe thing is to abort entirely. Otherwise you're opening yourself up to attacks. — Kerrek SB
– Kerrek SB, Commented Oct 23, 2012 at 12:54
What does invalid bytes mean for you ? Do you want a valid utf-8 stream (with maybe invalid codepoints or non-sensical combinations) or a valid unicode utf-8 encoded stream ? — Matthieu M.
– Matthieu M., Commented Oct 23, 2012 at 12:55
i need a valid unicode utf-8 encoded stream... remove everything that is not valid. — Alex Schneider
– Alex Schneider, Commented Oct 23, 2012 at 13:09
The Wikipedia page for UTF-8 (en.wikipedia.org/wiki/Utf-8) contains a lot of information that should easily help you accomplish this. — Component 10
– Component 10, Commented Oct 23, 2012 at 13:18
@SteveJessop: No no, the attack isn't in the valid string, but in the way you attempt to recover from invalid data. This has happened before, and as a result, the Unicode standard now says that an application should give up immediately upon encountering an invalid byte. — Kerrek SB
– Kerrek SB, Commented Oct 23, 2012 at 13:50

dsgriffin · Accepted Answer · 2012-10-23 12:47:58Z

2

You should use the icu::UnicodeString methods fromUTF8(const StringPiece &utf8) or toUTF8String(StringClass &result).

answered Oct 23, 2012 at 12:47

dsgriffin

68.8k17 gold badges141 silver badges139 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

Tino Didriksen · Accepted Answer · 2013-01-06 16:57:51Z

0

If you're sure your string is valid UTF-8 with only a few corrupt bytes, http://utfcpp.sourceforge.net/ can fix that. From the page:

#include "utf8.h"
void fix_utf8_string(std::string& str) {
    std::string temp;
    utf8::replace_invalid(str.begin(), str.end(), back_inserter(temp));
    str = temp;
}

Your requirement for not using a 3rd party library is pretty much impossible when dealing with Unicode data, but the UTF8-CPP library is header-only which is as light as you can get.

answered Jan 6, 2013 at 16:57

Tino Didriksen

2,26518 silver badges22 bronze badges

Collectives™ on Stack Overflow

How convert to utf8 string in c++

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related