Converting any encoding to utf8 in python?

Question

I'm crawling webpages from different websites and they have varied encodings. A sample of the encodings I get are -

Big5
TIS-620
utf-16le
shift_JIS
EUC-JP
MacCyrillic
koi8-r

apart from the more common encodings. I can get the unicode source of the web page by decoding using the above encodings.

My question is this: I would like to store all the files as utf8. If I encode the unicode source using utf8, will it work for all webpages? Does utf8 support all unicode code points?

The “UTF” part of the name stands for Unicode Transformation Format: any of the “UTF-...” encodings can indeed store all Unicode characters. — bobince
– bobince, Commented Aug 7, 2011 at 19:06

Ned Batchelder · Accepted Answer · 2011-08-07 11:40:04Z

4

Yes, UTF-8 is nothing more than a scheme for storing integers in bytes, in such a way that smaller integers take fewer bytes. The result is that values less than 128 are stored in one byte so that ASCII is still ASCII. UTF-8 can represent all Unicode codepoints.

answered Aug 7, 2011 at 11:40

Ned Batchelder

378k77 gold badges583 silver badges675 bronze badges

Sign up to request clarification or add additional context in comments.

Comments

john · Accepted Answer · 2011-08-07 11:25:06Z

1

Short and sweet, ........ yes!

answered Aug 7, 2011 at 11:25

john

88.3k5 gold badges61 silver badges95 bronze badges

Collectives™ on Stack Overflow

Converting any encoding to utf8 in python?

2 Answers 2

Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related