I am reading about Perl's Encode and utf8.
The doc says:
$octets = encode_utf8($string);
Equivalent to
$octets = encode("utf8", $string) .
The characters in $string are encoded in Perl's internal format, and the result is returned as a sequence of octets.
I have no idea what this means. Isn't a string in Perl a sequence of octets (i.e. bytes) anyway?
So what is the difference between:
$string and $octets?
No, a string in Perl is a sequence of characters, not necessarily octets. The chr and ord functions (for transforming between integers and single characters), to name two, can deal with integer values larger than 255. For example
$string = "\x{0421}\x{041F}";
print ord($_)," " for split //, $string;
outputs
1057 1055
When a string is written to a terminal, file, or other output stream, the device receiving the string usually requires and expects bytes, however, so this is where encoding comes in. As you have seen, UTF-8 is a scheme for encoding single value in the range 0x7F-0x10FFFF into multiple bytes.
$octets = Encode::encode("utf-8", "\x{0421}\x{041F}");
print ord($_)," " for split //, $octets;
Now the output is
208 161 208 159
and suitable to be stored on a filesystem.
Internally, perl (in all lower case, this refers to the executable implementation of Perl, the programming language specification) often uses UTF-8 to represent strings with "wide" characters, but this is not something you would every normally have to worry about.
\x{0421} mean?
\x{0421} is the character whose encoding is the hex number 0x421. That's apparently 'CYRILLIC CAPITAL LETTER ES' (see fileformat.info/info/unicode/char/421)decode instead of encode here to decode into Perl characters?
substr, chop, split, regular expressions, or any other feature of Perl that acts on a string, you are dealing with characters.