3

Well, I am reading Programing Windows with MFC, and I came across Unicode and ASCII code characters. I understood the point of using Unicode over ASCII, but what I do not get is how and why is it important to use 8bit/16bit/32bit character? What good does it do to the system? How does the processing of the operating system differ for different bits of character.

My question here is, what does it mean to a character when it is a x-bit character?

Improve this question
3
  • 9
    This may be a good read: joelonsoftware.com/articles/Unicode.html Commented Jul 23, 2012 at 12:09
  • If you use an eight bit sized character, how will you support multiple languages, like Russian and Greek within the same application? What about Chinese or Japanese? Commented Jul 23, 2012 at 21:43
  • I use 32-bit characters (UTF32) for random access simplicity which is the most memory hoggy, but least error prone and maps the most directly to the data structures and algorithms I've already been using for ages. In my case my application doesn't work with huge strings let alone numerous tiny strings. If I did and had more sequential access patterns as hotspots, I might be prone to use UTF8 since the common case scenarios are still extended ASCII characters (the strings are not always user-end strings). Commented Dec 8, 2017 at 15:45

3 Answers 3

Reset to default
9

It relates to the amount of possible letters/numbers/symbols a character set can have. An 8-Bit character can only have 256 possible characters. Whereas a 16-bit can have 65,536. A 32-bit character can have 4,294,967,296 possible characters. A character set that large should be able to store every possible character in the world. Whereas a 8-bit character set can only store enough characters for the English language. 8-bit character sets were the preferred standard in the early days of computing where memory was measured in bits and in some cases, KB. But with computers with multi-core processors and gigs of RAM it is not such a concern anymore (except in some rare cases)

Improve this answer
5
  • and it does not make a difference to the processing speed whatsoever? Commented Jul 23, 2012 at 11:55
  • It's not all that rare. Textual data is still pretty commonplace, and it takes a lot of storage space. Even on a 64-bit processor, you can only get 4 UTF-32 characters in a 128-bit cache line. That means more bus transfers to bring in data, which directly translates to slower processing speed. Plus, a lot of web applications are pushing the limits of even 32GB servers. It's still not a good idea to inflate your storage requirements. Commented Jul 23, 2012 at 12:44
  • Extended ascii (8bit) has enough characters for English, Esperanto, Spanish, French, German (half-true: German provides alternative ways of typing unrepresented characters). If you're willing to use different character sets depending on language, you can handle most languages (i.e., languages with 256 or fewer characters in their alphabet), though not simultaneously. Commented Jul 23, 2012 at 13:08
  • 1
    @Brian Not sure what you mean by “alternative ways of typing unrepresented characters” but be that as it may, Extended ASCII still seems to lack “ß” (although ISO-8859-1 includes it), and some French characters (“œ”, “Œ”, “Ÿ”). It also lacks core elements of punctuation in those languages (notably quotation marks). Commented Jul 23, 2012 at 13:24
  • @KonradRudolph: Touché. German has some characters that can be represented using either latin characters or using German characters. The German characters are not represented within ascii. To go with your example, ß is often written as "sz", or "ss". I'm sure a German speaker would happily point out that this is a bit wrong, but that's what old printers did to handle the lack of a ß character. Commented Jul 23, 2012 at 15:26
10

Joel Spolsky, co-founder of this site, actually has a wonderful and brief article on character sets titled The Absolute Minimum Every Software Developer Absolutely, Positively Must Know About Unicode and Character Sets (No Excuses!).

I found it to be a good read, and it answers many questions about different character sets that you may have, as well as briefly going over the historical reasons for the whole character set mess anyway.

I'm not sure why you're being voted down; it's pretty useful stuff to know!

Improve this answer
1
  • 1
    yeah the article is really great.. it answers almost everything, thanks for sharing! :) Commented Jul 23, 2012 at 19:19
2

Unicode is logically a 21-bit code. As modern computers don’t conveniently work with such units, there are various solutions: use 32 bits (4 bytes), wasting a lot of bits, especially if your data is dominantly in English; use a special scheme that uses one or two 16-bit units per character; and use a variable number of 8-bit bytes per character. These are known as UTF-32, UTF-16, and UTF-8 transfer encodings.

Windows uses internally UTF-16, whereas UTF-8 dominates e.g. on the Web, so you often need to convert between them. This is nontrivial but usually made with suitable library routines, maybe implicitly, depending on programming environment. UTF-32 is rarely used.

Technically, UTF-16 is very simple for all characters that fit into the 16-bit subspace of Unicode, Basic Multilingual Plane (BMP)—quite possibly all characters you ever heard of. UTF-8 is more complex but has been design with Western emphasis: all Ascii characters are represented as single bytes in UTF-8, so any file that contains dominantly Ascii is of almost the same size in UTF-8 as in Ascii. This is opposite to UTF-16, which always uses two bytes per Ascii character.

Improve this answer
6
  • Is Windows really UTF-16 aware? I'm afraid it mostly uses UCS-2 in its APIs. Commented Jul 23, 2012 at 12:09
  • 3
    That was only true for the original Windows NT, I believe, and possibly the compatibility layers for 98 and previous. Newer Windows variants support surrogates. I would also say that UTF-8 was designed with backwards compatibility in mind, rather than simply Western emphasis, and it is exactly the same size as ASCII for all information in that plane. Commented Jul 23, 2012 at 12:13
  • 1
    That UTF-16 is so simple for the BMP is probably also its biggest drawback -- it is very easy to forget about non-BMP characters, and have them come back and bite you later. UTF-32 doesn't have that problem at all, and you will encounter it much sooner when using UTF-8, since all it takes is a single character U+0080 or above (which is plenty of characters seen in more or less everyday texts even in English, these days). Commented Jul 23, 2012 at 14:05
  • In my mind, converting between UTF-8 and UTF-16 is pretty trivial; the canonical implementation is a simple algorithmic transformation in the Unicode reference does it in a pretty small number of lines of code. It's certainly trivial in comparison to converting between Unicode and legacy national encodings, say, JIS or EUC-JP to UTF-8. Commented Jul 23, 2012 at 20:02
  • 1
    @JasonTrue: The conversion between 8 and 16 is pretty simple when you have a well-formed UTF sequence. Once you take care for illegal encodings in both UTF-8 and UTF-16 and for endianness of UTF-16, it gets a bit more complicated. Commented Jul 23, 2012 at 21:58

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.