HttpClient not sending Accept-Encoding on Windows 2008 R2

Question

I have a .NET Core 2.0 console app that does a GET request.

It seems like the published version does not send Accept-Encoding headers for compression on a test machine, but works on my local machine.

I cannot find any other pre-requesites that would make compression fail. Both are running the .NET Core 2.1.4 SDK.

I have tested the console app by running dotnet Console.dll in both environments.

Publish in VS2017
Go to output folder and run dotnet Console.dll. Verify header present in Fiddler.
Copy entire output folder and deploy onto server
Run dotnet Console.dll again and verify header missing on the server with Fiddler.

I have tried both HttpClient and RestSharp and I'm pretty perplexed.

Proof of concept that goes to a page that echoes the request headers:

 var handler = new HttpClientHandler()
            {
                AutomaticDecompression = DecompressionMethods.Deflate | DecompressionMethods.GZip
            };

 using (var client = new HttpClient(handler))
 {
      response = client.GetStringAsync("http://scooterlabs.com/echo").Result;
 }

Local environment (Win10)

GET http://scooterlabs.com/echo HTTP/1.1
Connection: Keep-Alive
Accept-Encoding: gzip, deflate
Host: scooterlabs.com

Server (Win2008 R2 on AWS)

GET http://scooterlabs.com/echo HTTP/1.1
Connection: Keep-Alive
Host: scooterlabs.com

That's the request header though. I'm not sure what scooterlabs uses. Edit: I tried using WebClient and that seems to work fine on the server, so it's something in the HttpClient implementation that doesn't seem to work? — Jun Wei Lee
– Jun Wei Lee, Commented Feb 9, 2018 at 10:11
HttpClient works just fine. You are not using HttpClient in your example though, you are using RestClient. In any case, if compression didn't work, thousands of developers would have noticed 5 years ago. — Panagiotis Kanavos
– Panagiotis Kanavos, Commented Feb 9, 2018 at 10:17
Write a minimal, reproducible example and use Fiddler to check the HTTP request and response. If the request headers are there and the service doesn't compress the response, it's not a problem with HttpClient — Panagiotis Kanavos
– Panagiotis Kanavos, Commented Feb 9, 2018 at 10:20
And what if you add header yourself (and remove AutomaticDecompression)? — Evk
– Evk, Commented Feb 9, 2018 at 10:33
@JunWeiLee to put it another way - I'm using HttpClient on Core 2.0 and compression works. You assume it doesn't. The server isn't HttpClient. If the header doesn't arrive at the server, something (eg firewall) may have removed it. Does the server support compression in the first place? How did you configure the server? What is the server? — Panagiotis Kanavos
– Panagiotis Kanavos, Commented Feb 9, 2018 at 10:35

Evk · Accepted Answer · 2018-02-09 11:22:27Z

3

My best guess would be that's because WinHttp library, which is used by default by HttpClient on windows does not support gzip\deflate on windows versions before Windows 8.1+. When supported - WinHttp will also set Accept-Encoding header. So on Windows Server 2008 when .NET routes request through WinHttp - it either sets this option and it is ignored, or it checks if this option is supported and if not - just not sets it.

If you set this header manually (like client.DefaultRequestHeaders.AcceptEncoding.Add(new StringWithQualityHeaderValue("gzip"));) - option is still ignored but header passes through and server returns compressed response. If supported - WinHttp will decompress that response and remove Content-Encoding header, so response will arrive decompressed to .NET. If not supported - response will arrive compressed and .NET itself will decompress it if you set AutomaticDecompression.

So to conclude - on Windows versions before 8.1 it seems you need to set both AutomaticDecompression and related Accept-Encoding headers for this to work as expected.

edited Feb 9, 2018 at 11:22

answered Feb 9, 2018 at 11:14

Evk

102k8 gold badges156 silver badges204 bronze badges

Sign up to request clarification or add additional context in comments.

1 Comment

Jimi Over a year ago

In Windows 7, if you set manually the header (something like: httprequest.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip, deflate;q=0.8");, and don't specify AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;, the stream will arrive compressed with one of the preferred methods (gzip, if available, in the sample code). System.IO.Compression.GZipStream and System.IO.Compression.DeflateStream can be used to decompress the stream and copy it to a MemoryStream for the final StreamReader.ReadToEnd([MemoryStream]).

Jun Wei Lee · Accepted Answer · 2018-02-09 23:37:46Z

Building on Ivan and Evk's answers, it does appear that this is an issue specific to older versions of Windows (older than Win8.1). Here is how to work around and successfully handle compression on older versions of Windows.

   var handler = new HttpClientHandler()
                {
                    AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate
                };

   using (var client = new HttpClient(handler))
   {
        client.DefaultRequestHeaders.AcceptEncoding.Add(new StringWithQualityHeaderValue("gzip"));
        client.DefaultRequestHeaders.AcceptEncoding.Add(new StringWithQualityHeaderValue("deflate"));
        response = client.GetStringAsync("http://scooterlabs.com/echo").Result;
   }

AutomaticDecompression must be set in addition to the headers, otherwise you will get the compressed payload instead.

Ivan Zaruba · Accepted Answer · 2018-02-09 11:09:48Z

2

Not the answer, though I've tried to reproduce this on Azure's Win Server 2008 R2 with different Echo servers (.NET Core SDK 2.1.4)

// http://scooterlabs.com/echo output

[headers] => Array
(
    [Connection] => Keep-Alive
    [Host] => scooterlabs.com
)


// http://httpbin.org/headers output
{
  "headers": {
    "Connection": "close",
    "Host": "httpbin.org"
  }
}

// https://postman-echo.com/headers output
{
    "headers": {
        "host": "postman-echo.com",
        "x-forwarded-port": "443",
        "x-forwarded-proto": "https"
    }
}

On Win10 indeed I also have those headers present.

// http://scooterlabs.com/echo output
[headers] => Array
(
    [Connection] => Keep-Alive
    [Accept-Encoding] => gzip, deflate
    [Host] => scooterlabs.com
)

// http://httpbin.org/headers output
{
  "headers": {
    "Accept-Encoding": "gzip, deflate",
    "Connection": "close",
    "Host": "httpbin.org"
  }
}

// https://postman-echo.com/headers output
{
    "headers": {
        "host": "postman-echo.com",
        "accept-encoding": "gzip, deflate",
        "x-forwarded-port": "443",
        "x-forwarded-proto": "https"
    }
}

which makes me think that HttpClient's Accept-Encoding header is really not supported on WS 2008 R2.

answered Feb 9, 2018 at 11:09

Ivan Zaruba

4,5445 gold badges22 silver badges33 bronze badges

1 Comment

Jun Wei Lee Over a year ago

Thanks, just confirmed with a Windows 8.1 VM that it is indeed set. Combined with @Evk's answer we can conclude that it might be something the Core team might want to look at, as it really can handle the compression.

Collectives™ on Stack Overflow

HttpClient not sending Accept-Encoding on Windows 2008 R2

3 Answers 3

1 Comment

Comments

1 Comment

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

3 Answers 3

1 Comment

Comments

1 Comment

Your Answer

Sign up or log in

Post as a guest

Related