How to reuse an http.Client instance with a dynamic proxy URL in Go

Question

I'm building an HTTP client in Go and I need to make calls to an endpoint via a proxy server. The proxy address is not fixed, so I'm currently creating a new http.Client instance every time I need to make a call. This isn't very efficient, as it involves creating a new transport and client object every time.

Is there a way to create the http.Client instance only once and then update the proxy address before making a call? I'd like to avoid creating a new client instance every time I need to make a request.

One more thing - There will be a fixed number of proxy servers (<5) and hence fixed number of URLs but I don't know the address in the beginning of those servers, and I will get those addresses dynamically.

Because of this condition, I am thinking if I could create multiple client - one for each server and reuse them based on the URL I get. But I would still prefer a solution where I could create only one client and update proxy URL somehow.

Any suggestions would be very helpful.

Here's my current code for creating the client:

proxyURL, err := url.Parse("http://proxy_url:proxy_port")
if err != nil {
    fmt.Println(err)
    return
}

transport := &http.Transport{
    Proxy: http.ProxyURL(proxyURL),
}
client := &http.Client{
    Transport: transport,
}

Thanks!

Peter · Accepted Answer · 2024-09-05 07:52:20Z

1

You are already using the Proxy field of the Transport, but there is no reason to use http.ProxyURL (which returns a static value). Write your own lookup function instead:

package main

import (
    "net/http"
    "net/url"
)

func main() {
    c := &http.Client{
        Transport: &http.Transport{
            Proxy: lookupProxy,
        },
    }

    // use (and re-use) c for any request
}

func lookupProxy(r *http.Request) (*url.URL, error) {
    // TODO: map r.URL to the appropriate proxy
    panic("unimplemented")
}

If the decision which proxy to use depends on things not available in http.Request, you make it before calling Client.Do and pass the proxy URL via the request context:

package main

import (
    "context"
    "net/http"
    "net/url"
)

func main() {
    c := &http.Client{
        Transport: &http.Transport{
            Proxy: ProxyFromContext,
        },
    }

    req, err := http.NewRequest("GET", "http://example.com", nil)

    req = WithProxy(req, &url.URL{
        // ...
    })

    res, err := c.Do(req)


    // use (and re-use) c for any request
}

type ctxKey int

const (
    ctxKeyProxyURL ctxKey = iota
)

func WithProxy(r *http.Request, u *url.URL) *http.Request {
    ctx := context.WithValue(r.Context(), ctxKeyProxyURL, u)

    return r.WithContext(ctx)
}

func ProxyFromContext(r *http.Request) (*url.URL, error) {
    u, _ := r.Context().Value(ctxKeyProxyURL).(*url.URL)
    return u, nil
}

edited Sep 5, 2024 at 7:52

answered Sep 4, 2024 at 13:55

Peter

32.1k6 gold badges57 silver badges68 bronze badges

Sign up to request clarification or add additional context in comments.

4 Comments

ray an Over a year ago

Hi, thanks for the reply. Unfortunately, there are no fixed mappings between the destination URL and the proxy to be used. I get the proxy for each request at runtime, so it is possible to request the same destination URL using a different prox Let me know if I am missing somethig.

ray an Over a year ago

But I think this is still useful. Maybe I could custom headers for proxy address to the request and access it in the lookupProxy and return it.

Peter Over a year ago

I don't follow. What information do you need that you have before sending the request but not within lookupProxy?

Peter Over a year ago

"Maybe I could custom headers". I suggest using the request context instead.

Collectives™ on Stack Overflow

How to reuse an http.Client instance with a dynamic proxy URL in Go

1 Answer 1

4 Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

4 Comments

Your Answer

Sign up or log in

Post as a guest

Related