GRPC - The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR when upgrading from .netcore 3.1 to net5 or net6

Question

I have a .netcore3.1 test project which calls a GRPC service.
The service is only available via HTTP. I cannot change the service now.

When I update the project to .net5-windows or .net6-windows (literally only a csproj change), the calls fail with the following error:

Grpc.Core.RpcException : Status(StatusCode="Unavailable", Detail="Error starting gRPC call. HttpRequestException: An error occurred while sending the request. IOException: The request was aborted. Http2ConnectionException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1).", DebugException="System.Net.Http.HttpRequestException: An error occurred while sending the request. ---> System.IO.IOException: The request was aborted. ---> System.Net.Http.Http2ConnectionException: The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR' (0x1). at System.Net.Http.Http2Connection.ThrowProtocolError(Http2ProtocolErrorCode errorCode) at System.Net.Http.Http2Connection.ReadFrameAsync(Boolean initialFrame) at System.Net.Http.Http2Connection.ProcessIncomingFramesAsync() --- End of inner exception stack trace --- at System.Net.Http.Http2Connection.ThrowRequestAborted(Exception innerException) at System.Net.Http.Http2Connection.Http2Stream.CheckResponseBodyState() at System.Net.Http.Http2Connection.Http2Stream.TryEnsureHeaders() at System.Net.Http.Http2Connection.Http2Stream.ReadResponseHeadersAsync(CancellationToken cancellationToken) at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) --- End of inner exception stack trace --- at System.Net.Http.Http2Connection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken) at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken) at Grpc.Net.Client.Web.GrpcWebHandler.SendAsyncCore(HttpRequestMessage request, CancellationToken cancellationToken) at System.Net.Http.HttpClient.g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken) at Grpc.Net.Client.Internal.GrpcCall2.RunCall(HttpRequestMessage request, Nullable1 timeout)") at..

I had a look at few SO questions (e.g. .Net Core 3.1 gRPC client with unencrypted HTTP2 connection and Grpc.Core.RpcException: 'Status (StatusCode = "Unavailable", Detail = "Error starting gRPC call) as well as the 'call insecure service' troubleshooting guide https://learn.microsoft.com/en-us/aspnet/core/grpc/troubleshoot?view=aspnetcore-5.0#call-insecure-grpc-services-with-net-core-client however they are not helpful.

I have tried a few options of setting AppContext (even though they should apply for netcore3.1) like this before initializing a client, but again no help:

  AppContext.SetSwitch(“System.Net.Http.SocketsHttpHandler.Http2UnencryptedSupport”, true);
  AppContext.SetSwitch(“System.Net.Http.SocketsHttpHandler.Http2Support”, true);

The channel is initialized as follows:

 var baseUri = new Uri(baseUrl);
        HttpMessageHandler httpMessageHandler = new HttpClientHandler();
        if (baseUri.AbsolutePath != "/")
            httpMessageHandler = new SubDirectoryHandler(httpMessageHandler, baseUri.AbsolutePath);
        var handler = new GrpcWebHandler(GrpcWebMode.GrpcWebText, httpMessageHandler);
        _channel = GrpcChannel.ForAddress(baseUrl, new GrpcChannelOptions { HttpClient = new HttpClient(handler) });

Inspecting the GRPC channel before sending the request shows the request version is 1.1 (regardless of AppContext settings and TargetFramework):

The GRPC client libraries are:

Grpc.Net.Client v. 2.42.0
Grpc.Net.Client.Web v. 2.42.0

When the request is sent, the service side is logging the following in the output:

Exception thrown: 'Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException' in Microsoft.AspNetCore.Server.Kestrel.Core.dll
Exception thrown: 'Microsoft.AspNetCore.Server.Kestrel.Core.BadHttpRequestException' in System.Private.CoreLib.dll 'TradingOrchestrationService.exe' (CoreCLR: clrhost): Loaded 'C:\Program Files\dotnet\shared\Microsoft.AspNetCore.App\6.0.5\Microsoft.AspNetCore.WebUtilities.dll'.
Exception thrown: 'System.Net.Sockets.SocketException' in Microsoft.AspNetCore.Server.Kestrel.Transport.Sockets.dll
The I/O operation has been aborted because of either a thread exit or an application request.

The key thing I'd like to stress is the only difference between working and non-working behaviour is bumping up TargetFramework version in the entry assembly from 3.1 to 5 or 6.

Might be the server side, which maybe doesn't support HTTP2 properly? — Charlieface
– Charlieface, Commented Jul 10, 2022 at 10:16
@Charlieface, yeah, ok, but what causes the difference between net .3.1 and 5+ ? — Bartosz
– Bartosz, Commented Jul 10, 2022 at 10:27
Might be a default, or it might be Windows 7 vs 10, see stackoverflow.com/q/62245471/14868997 — Charlieface
– Charlieface, Commented Jul 10, 2022 at 10:32
Try capture good and bad request with a sniffer and compare the http headers. Use HTTP (not HTTPs) for the capture so request is not encrypted. Often default headers are different. if they are the same then issue is probably TLS. Maybe old was using TLS 1.1 while new is using TLS 1.2. Some encryption modes from 1.1 are now not valid in TLS 1.2 — jdweng
– jdweng, Commented Jul 10, 2022 at 13:56

Bartosz · Accepted Answer · 2022-07-11 12:34:31Z

For completeness and future reference I am posting the replies that I got from JamesNK on GitHub https://github.com/dotnet/runtime/issues/71906

When using HTTP then the client and server must use the same protocol. Without https then there is no negotiation.

What protocol is Kestrel configured to use? HTTP/1.1 or HTTP/2? If you haven't explicitly configured it to only HTTP/2 then it will probably be HTTP/1.1.

GrpcWebHandler has a HttpVersion property on it that you can set to explicitly specify what HTTP version you want. You could try setting it to new Version(1, 1).

Does setting this property have any other implications?
And when we migrate the server to use HTTP/2, I suppose it will have to be removed?

Yes

What happened to you: in .NET Core 3.x the gRPC client was configuring the gRPC request to be HTTP/2, but because you weren't using TLS the request was silently downgraded to HTTP/1.1

That was fixed in .NET 5+, but you were relying on broken behavior so the fix broke you.

The client is now sending HTTP/2 over http, but your server is configured for HTTP/1.1. That creates a protocol error. GrpcWebHandler.HttpVersion provides a way for you to override the HTTP version and to specify exactly what you want (because gRPC-Web works with both HTTP versions).

kjz99 · Accepted Answer · 2024-03-12 08:27:26Z

I've had this problem also. During development I have a local Kestrel server that was silently downgrading the connection to http/1.1 while the client still ran http/2. What I understood more or less from the grpc core team is that this could happen when SSL is not used, which skips the http protocol negotiations and defaults kestrel to http/1.1.

What you can do to resolve this issue in development/IoT edge scenario's like me is setting http/2 as the kestrel default by adding the following directives to the root of the appsetting.json of your application.

"Kestrel": {
  "EndpointDefaults": {
    "Protocols": "Http2"
  },
  "gRPC": {
    "Protocols": "Http2"
  }
}

So in this scenario, Kestrel controls the port, proto and url to bind on. So what also can be beneficial is changing the port Kestrel(and thus grpc) runs on during development. This can be done by setting an endpoint to kestrel. Eg, ListenLocalHost(port) in your program.cs

var host = Host.CreateDefaultBuilder(args)
    .UseWindowsService()
    .ConfigureWebHostDefaults(webBuilder =>
    {
        webBuilder.ConfigureKestrel(options =>
        {
            // Port to use by Kestrel and thus grpc
            options.ListenLocalhost(5001);
        });
        webBuilder.UseStartup<Startup>();
    })
    .Build();

await host.RunAsync();
Environment.Exit(0);

Collectives™ on Stack Overflow

GRPC - The HTTP/2 server sent invalid data on the connection. HTTP/2 error code 'PROTOCOL_ERROR when upgrading from .netcore 3.1 to net5 or net6

2 Answers 2

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related