0

Technical details

  • Client applications are using .NET 6
  • Client applications are using the OpenAIClient class from the official OpenAI nuget library
  • There is an inhouse cluster of LLM servers with OpenAI API support
  • A reverse proxy that intercepts the client requests

Goal

  • The client apps (using the OpenAiAPI SDK) to define some custom HTTP headers which will be used by the reverse proxy for balancing the load and sticky sessions

Note: the values of the headers are not static. Each request can potentially have a different value.

Problem

I believe that the official SDK:

  • Does not allow defining custom headers
  • Does not allow defining an HttpClient object nor a DelegatingHandler (which they could be used to define custom headers)

Questions

  • Is the problem statement correct?
  • Is there a workaround to this problem?
  • Is there a completely different solution on how to approach to the sticky session requirement (given the dynamic nature of the header values)?

1 Answer 1

1

(!) Disclaimer: This is merely a solution to the initial requirement of adding custom headers. I am not knowledgeable enough to assess whether the solution is affecting the OpenAiClient capabilities in any way (!)

The trick was to use a Pipeline policy (via the HttpClientPipelineTransport).

        services.AddTransient<CustomHeadersHandler>(); //this is a standard 'DelegatingHandler' implementation that adds HTTP headers
        services.AddHttpClient("OpenAiHttpClient", client =>
        {
            client.Timeout = TimeSpan.FromMilliseconds(apiTimeout);
        })
        .AddHttpMessageHandler<CustomHeadersHandler>();

        services.AddSingleton(services =>
        {
            var httpClientFactory = services.GetRequiredService<IHttpClientFactory>();
            var httpClient = httpClientFactory.CreateClient("OpenAiHttpClient");

            return new OpenAIClient(
                new ApiKeyCredential(apiKey),
                new OpenAIClientOptions
                {
                    Endpoint = new Uri(apiUri),
                    NetworkTimeout = TimeSpan.FromMilliseconds(apiTimeout),
                    Transport = new HttpClientPipelineTransport(httpClient)
                }
            );
        });
Sign up to request clarification or add additional context in comments.

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.