How to get an HTTP Proxy response Header in Golang built-in HTTP client?

Question

I'm doing a simple HTTP GET request with the default net/http Golang lib via HTTP proxy and want to read the content of the first proxy reply(for the HTTP client req with the CONNECT method).

In plain text, it looks like this

HTTP/1.1 200 OK
Request-Uid: <some id>
<another header>: <another value>

Golang code:

...
proxyUrlParsed, errUrl := url.Parse(proxyUrl)
tr := &http.Transport{
   Proxy:   http.ProxyURL(proxyUrlParsed),
}
client := &http.Client{
   Transport: tr,
}
request, errReq := http.NewRequest("GET", targetUrl, nil)
response, errDo := client.Do(request)
// Response contains HTTP headers from the reply from the 
// target resource but not the intermediate proxy.

I partially solved it with DialContext, but I were needed to impl some parts of the protocol that I found not so handy and costly for later support. So is there an easy and clever way to do it?

surfmuggle · Accepted Answer · 2023-09-30 15:02:56Z

1

tunnel proxy

Take the use of the curl client as an example. When requesting https, use the CONNECT method to connect to the tunnel connection. The obtained stream content is TLS encrypted content, which cannot be decrypted by the proxy.

If have a tls certificate, can try to parse the response stream.

When wireshark captures https requests, a parameter needs to be configured in the browser. The certificate is saved in the specified file

http_proxy=127.0.0.1:8021 https_proxy=127.0.0.1:8021 curl -v https://qq.com
http_proxy=127.0.0.1:8021 https_proxy=127.0.0.1:8021 curl -v https://qq.com

package main

import (
    "fmt"
    "io"
    "log"
    "net"
    "net/http"
    "net/http/httputil"
    "os"
)

func main() {
    proxy := func(w http.ResponseWriter, req *http.Request) {
        log.Println("proxy", req.Method, req.RequestURI)
        if req.URL.Host != "" {
            if req.Method == http.MethodConnect {
                // tunnel
                conn, err := net.Dial("tcp", req.URL.Host)
                if err != nil {
                    w.WriteHeader(502)
                    fmt.Fprint(w, err)
                    return
                }

                client, _, err := w.(http.Hijacker).Hijack()
                if err != nil {
                    w.WriteHeader(502)
                    fmt.Fprint(w, err)
                    conn.Close()
                    return
                }
                client.Write([]byte("HTTP/1.0 200 OK\r\n\r\n"))

                hr, hw := io.Pipe()
                go func(){
                    io.Copy(os.Stdout, hr)
                    hr.Close()
                }()
                go func() {
                    // print response to stdout
                    io.Copy(io.MultiWriter(client, hw), conn)
                    client.Close()
                    conn.Close()
                    hw.Close()
                }()
                go func() {
                    io.Copy(conn, client)
                    client.Close()
                    conn.Close()
                }()
                return
            }

            httputil.NewSingleHostReverseProxy(req.URL).ServeHTTP(w, req)
        }
    }
    http.ListenAndServe(":8021", http.HandlerFunc(proxy))
}

reverse proxy

use /net/http/httputil.ReverseProxy proxy a request ,set ModifyResponse field is response hook.

package main

import (
    "log"
    "net/http"
    "net/http/httputil"
    "net/url"
)

func main() {
    proxy := httputil.NewSingleHostReverseProxy(&url.URL{
                                                Scheme: "http", 
                                                Host: "127.0.0.1:8020"})

    proxy.ModifyResponse = func(w *http.Response) error {
        w.Header.Add("Author", "eudore")
        log.Println(w.Request.Method, w.Request.RequestURI, w.Status)
        return nil
    }
    http.ListenAndServe(":8021", proxy)
}

Curl requests

With port 8020 like [root@node1 ~]# curl -I 127.0.0.1:8020?222

HTTP/1.1 401 Unauthorized
Www-Authenticate: Basic
Date: Thu, 17 Nov 2022 01:34:06 GMT

Or port 8021 like [root@node1 ~]# curl -I 127.0.0.1:8021?222

HTTP/1.1 401 Unauthorized
Author: eudore
Date: Thu, 17 Nov 2022 01:34:07 GMT
Www-Authenticate: Basic

edited Sep 30, 2023 at 15:02

surfmuggle

6,0408 gold badges58 silver badges96 bronze badges

answered Nov 17, 2022 at 1:23

eudore

8036 silver badges11 bronze badges

Sign up to request clarification or add additional context in comments.

3 Comments

greggyNapalm Over a year ago

I'm talking about forward proxies (or tunnel, or gateway), not reverse proxies. developer.mozilla.org/en-US/docs/Web/HTTP/… Thanks for your reply, but it's irrelevant to my case.

eudore Over a year ago

@greggyNapalm Add tunnel proxy content.

greggyNapalm Over a year ago

I've added the screenshot of the typical plain-text interaction with the generic HTTP proxy service. As you can see, the first reply is pliant-text and can be read by the client. After the successful proxy reply, the interaction with the target resource will take place, and it might be increased, like in my case b/c of the 443 port == TLS protocol.

greggyNapalm · Accepted Answer · 2024-03-03 20:19:54Z

0

In Golang >=1.20 this task can be solved using standard library https://pkg.go.dev/net/http@master#Transport.OnProxyConnectResponse

Code example https://github.com/greggyNapalm/proxychick/blob/v0.0.1/pkg/httpx/httpx.go#L116

answered Mar 3, 2024 at 20:19

greggyNapalm

5436 silver badges21 bronze badges

Collectives™ on Stack Overflow

How to get an HTTP Proxy response Header in Golang built-in HTTP client?

2 Answers 2

tunnel proxy

reverse proxy

Curl requests

3 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

tunnel proxy

reverse proxy

Curl requests

3 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related