golang http timeout and goroutines accumulation

Question

I use goroutines achieve http.Get timeout, and then I found that the number has been rising steadily goroutines, and when it reaches 1000 or so, the program will exit

Code:

package main

import (
        "errors"
        "io/ioutil"
        "log"
        "net"
        "net/http"
        "runtime"
        "time"
)

// timeout dialler
func timeoutDialler(timeout time.Duration) func(network, addr string) (net.Conn, error) {
        return func(network, addr string) (net.Conn, error) {
                return net.DialTimeout(network, addr, timeout)
        }
}

func timeoutHttpGet(url string) ([]byte, error) {
        // change dialler add timeout support && disable keep-alive
        tr := &http.Transport{
                Dial:              timeoutDialler(3 * time.Second),
                DisableKeepAlives: true,
        }

        client := &http.Client{Transport: tr}

        type Response struct {
                resp []byte
                err  error
        }

        ch := make(chan Response, 0)
        defer func() {
                close(ch)
                ch = nil
        }()

        go func() {
                resp, err := client.Get(url)
                if err != nil {
                        ch <- Response{[]byte{}, err}
                        return
                }
                defer resp.Body.Close()

                body, err := ioutil.ReadAll(resp.Body)
                if err != nil {
                        ch <- Response{[]byte{}, err}
                        return
                }

                tr.CloseIdleConnections()
                ch <- Response{body, err}
        }()

        select {
        case <-time.After(5 * time.Second):
                return []byte{}, errors.New("timeout")
        case response := <-ch:
                return response.resp, response.err
        }
}

func handler(w http.ResponseWriter, r *http.Request) {
        _, err := timeoutHttpGet("http://google.com")
        if err != nil {
                log.Println(err)
                return
        }
}

func main() {
        go func() {
                for {
                        log.Println(runtime.NumGoroutine())
                        time.Sleep(500 * time.Millisecond)
                }
        }()

        s := &http.Server{
                Addr:         ":8888",
                ReadTimeout:  15 * time.Second,
                WriteTimeout: 15 * time.Second,
        }

        http.HandleFunc("/", handler)
        log.Fatal(s.ListenAndServe())
}

http://play.golang.org/p/SzGTMMmZkI

Please always put the relevant code in your question. A link to a fiddle is fine but as a complement. — Denys Séguret
– Denys Séguret, Commented Jan 8, 2014 at 8:27

maddyblue · Accepted Answer · 2014-01-08 08:47:15Z

Init your chan with 1 instead of 0:

ch := make(chan Response, 1)

And remove the defer block that closes and nils ch.

See: http://blog.golang.org/go-concurrency-patterns-timing-out-and

Here is what I think is happening:

after the 5s timeout, timeoutHttpGet returns
the defer statement runs, closing ch and then setting it to nil
the go routine it started to do the actual fetch finishes and attempts to send its data to ch
but ch is nil, and so won't receive anything, preventing that statement from finishing, and thus preventing the go routine from finishing

I assume you are setting ch = nil because before you had that, you would get run-time panics because that's what happens when you attempt to write to a closed channel, as described by the spec.

Giving ch a buffer of 1 means that the fetch go routine can send to it without needing a receiver. If the handler has returned due to timeout, everything will just get garbage collected later on.

Collectives™ on Stack Overflow

golang http timeout and goroutines accumulation

1 Answer 1

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

1 Answer 1

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related