HttpRequestHeader Content encoding issue

Question

I am using below code snippet to download HTTP response to local file. Sometimes my content which is in url is multi-lingual (chinese, japanese, thai data etc.). I am using ContentEncoding header to specify my content is in UTF-8 encoding, but this has no effect in my local output file which is generating in ASCII. Due to this, multi-lingual data is corrupted. Any help?

using (var webClient = new WebClient())
        {
            webClient.Credentials = CredentialCache.DefaultCredentials;
            webClient.Headers.Add(HttpRequestHeader.UserAgent, "Mozilla/4.0");
            webClient.Headers.Add(HttpRequestHeader.ContentEncoding, "utf-8");

            webClient.DownloadFile(url, @"c:\temp\tempfile.htm");
        }

If you want to support wide characters you should use something like utf-16 — Peter Ritchie
– Peter Ritchie, Commented Jul 17, 2012 at 22:07
UTF8 is not ASCII/ANSI, but UTF8 supports ASCII character set. — Mahender
– Mahender, Commented Jul 17, 2012 at 23:00

Jim Mischel · Accepted Answer · 2012-07-18 02:28:39Z

7

The ContentEncoding header is not used to specify the character set. It's used by the client to say what kind of encoding (compression) it supports.

The client can't tell the server what character set to send. The server sends its data and some header fields that say what character set is being used. Typically it's in the ContentTypeheader and looks like: text/html; charset=UTF-8.

When you're using WebClient, you want to set the Encoding property as a fallback so that if the server doesn't identify the character set, your default will be used. For example:

WebClient client = new WebClient();
client.Encoding = Encoding.UTF8;
string s = client.DownloadString(DownloadUrl);

See http://www.informit.com/guides/content.aspx?g=dotnet&seqNum=800 for a bit more information.

answered Jul 18, 2012 at 2:28

Jim Mischel

135k25 gold badges197 silver badges377 bronze badges

Sign up to request clarification or add additional context in comments.

Collectives™ on Stack Overflow

HttpRequestHeader Content encoding issue

1 Answer 1

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Related