Technical details
- Client applications are using .NET 6
- Client applications are using the
OpenAIClientclass from the official OpenAI nuget library - There is an inhouse cluster of LLM servers with OpenAI API support
- A reverse proxy that intercepts the client requests
Goal
- The client apps (using the OpenAiAPI SDK) to define some custom HTTP headers which will be used by the reverse proxy for balancing the load and sticky sessions
Note: the values of the headers are not static. Each request can potentially have a different value.
Problem
I believe that the official SDK:
- Does not allow defining custom headers
- Does not allow defining an
HttpClientobject nor aDelegatingHandler(which they could be used to define custom headers)
Questions
- Is the problem statement correct?
- Is there a workaround to this problem?
- Is there a completely different solution on how to approach to the sticky session requirement (given the dynamic nature of the header values)?