Same strings, different encoding, but not equal in Ruby

Question

Can anyone explain what's going on with this code?

s1 = "\x20".force_encoding 'UTF-8'
s2 = "\x20".force_encoding 'ASCII-8BIT'
puts "s1 == s2: #{s1 == s2}"

s3 = "\xAB".force_encoding 'UTF-8'
s4 = "\xAB".force_encoding 'ASCII-8BIT'
puts "s3 == s4: #{s3 == s4}"

In Ruby 2.0.0p353 it prints:

s1 == s2: true
s3 == s4: false

I don't understand why s3 and s4 are not equal when s1 and s2 is. 0xAB is the ASCII code for '½', which as far as I know is representable in both ASCII-8BIT and UTF8.

\0xAB is also not ½ as a UTF-8 character code. I found this: "\xAB".force_encoding('CP850').encode('UTF-8') - gives ½ . . . en.wikipedia.org/wiki/Code_page_850 - probably a few other MSDOS-based extensions have this mapping too. — Neil Slater
– Neil Slater, Commented Feb 8, 2014 at 16:31
I don't know where you got your info about that being the ASCII code for 1/2. It is actually the Left-pointing double angle quotation mark, left pointing guillemet. Did you mean \xBD? — roippi
– roippi, Commented Feb 8, 2014 at 16:45

Малъ Скрылевъ · Accepted Answer · 2016-09-22 20:40:38Z

3

\xAB in utf-8 isn't the same as \xAB in ascii-8bit codepage, because to encode the utf-8 is coded in multi-byte set, and chars from \x80 to \xff is used to encode symbols with codes over \x80.

But since the ASCII-8BIT isn't specific encoding, but can be treated as the encoding class based on ascii, and it is aliased to binary encoding in ruby. The codes from from \x80 to \xff can't also be converted any encoding. So it is like an abstraction for ASCII based-codepages.

So, if you try to convert from ASCII-8BIT to utf-8 you would get the conversion exception:

Encoding::UndefinedConversionError: "\xC9" from ASCII-8BIT to UTF-8

However, you are able to handle ½ symbol properly in 8-bit encoding using explicitly set iso-8859-1 or cp1252 codepages, and char \xBD as follows:

"\xBD".force_encoding('ISO-8859-1').encode('UTF-8')
# => "½"
"\xBD".force_encoding('CP1252').encode('UTF-8')
# => "½"

edited Sep 22, 2016 at 20:40

answered Feb 9, 2014 at 19:32

Малъ Скрылевъ

16.5k5 gold badges59 silver badges74 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

Same strings, different encoding, but not equal in Ruby

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related