How to measure system overload when using GO

Question

I am rewriting an old system in GO, and in the old system I was measuring the system load average to know if I should increase the number of thread in my thread-pool.

In go people are not using threadpool or pool of goroutine because starting a goroutine is very cheap. But still running too many goroutine is less efficient then just enough to keep the cpu usage near 100%

Thus is there a way to know how many goroutine are ready to run (not blocked) but not currently running. Or is there a way to get the number of scheduled runnable goroutine "Run queue".

It should only barely be the case that 'running too many goroutine is less efficient then just enough to keep the cpu usage near 100%'. Go rarely switches away from a running goroutine unless it's blocked waiting for I/O or on a channel op or sync primitive, so starting lots of goroutines, with the OS thread count matching CPU count via runtime.GOMAXPROCS(runtime.NumCPU()), needn't create much additional context-switch overhead. We might be able to help more with additional info on the workload--are your goroutines mostly spinning the CPU, or waiting on a DB, or channel ops, or...? — twotwotwo
– twotwotwo, Commented Nov 4, 2013 at 23:51
Thanks user2714852 so you are saying if GOMAXPROCS is set to 2 and I start 4 goroutine that never stop and never block the runtime will only run the first 2 and never context-switch to the other one ? — skyde
– skyde, Commented Nov 5, 2013 at 17:39
In Go 1.1 that's exactly right: goroutine scheduling is purely cooperative and if there's an endless loop with no I/O, etc., it hogs the thread forever. That's talked about in Go bug 543. (You can always call runtime.Gosched() to explicitly yield.) In Go 1.2rc3, "The scheduler is invoked occasionally upon entry to a function."; the "occasionally" in that sentence led me to say Go "rarely" forces a switch. That's all I know; I couldn't glean much more from peeking at the Go source just now. — twotwotwo
– twotwotwo, Commented Nov 5, 2013 at 20:17
You could add some level of tracking of how many jobs a goroutine processes during runtime as a measure of liveness per goroutine. — Daniel Williams
– Daniel Williams, Commented Nov 5, 2013 at 23:11
to answer twotwotwo question, Please assume the goroutine are doing mostly CPU intensive work. — skyde
– skyde, Commented Nov 5, 2020 at 20:04

Intermernet · Accepted Answer · 2013-11-06 05:07:57Z

10

Check out the runtime/pprof package.

To print "stack traces of all current goroutines" use:

pprof.Lookup("goroutine").WriteTo(os.Stdout, 1)

To print "stack traces that led to blocking on synchronization primitives" use:

pprof.Lookup("block").WriteTo(os.Stdout, 1)

You can combine these with the functions in the runtime package such as runtime.NumGoroutine to get some basic reporting.

This example deliberately creates many blocked goroutines and waits for them to complete. Every 5 seconds it prints the output of the block pprof profile, as well as the number of goroutines still in existence:

package main

import (
    "fmt"
    "math/rand"
    "os"
    "runtime"
    "runtime/pprof"
    "strconv"
    "sync"
    "time"
)

var (
    wg sync.WaitGroup
    m  sync.Mutex
)

func randWait() {
    defer wg.Done()
    m.Lock()
    defer m.Unlock()
    interval, err := time.ParseDuration(strconv.Itoa(rand.Intn(499)+1) + "ms")
    if err != nil {
        fmt.Errorf("%s\n", err)
    }
    time.Sleep(interval)
    return
}

func blockStats() {
    for {
        pprof.Lookup("block").WriteTo(os.Stdout, 1)
        fmt.Println("# Goroutines:", runtime.NumGoroutine())
        time.Sleep(5 * time.Second)
    }
}

func main() {
    rand.Seed(time.Now().Unix())
    runtime.SetBlockProfileRate(1)
    fmt.Println("Running...")
    for i := 0; i < 100; i++ {
        wg.Add(1)
        go randWait()
    }
    go blockStats()
    wg.Wait()
    fmt.Println("Finished.")
}

I'm not sure if that's what you're after, but you may be able to modify it to suit your needs.

Playground

edited Nov 6, 2013 at 5:07

answered Nov 1, 2013 at 1:03

Intermernet

19.6k4 gold badges51 silver badges61 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Brenden Over a year ago

+1 this seems to answer the question. I found this SOA informative stackoverflow.com/a/10096686/143225

twotwotwo Over a year ago

The working sample program is awesome. It may also be worth testing the premise of the question, i.e., whether dynamically tuning goroutine count improves perf enough to be worth it (compared to just launching a ton of goroutines or some other simple strategy). The answer to that may depend on the app. But, either way, this seems to answer the question that was asked.

Intermernet Over a year ago

@user2714852 I agree, I've not yet come across a situation where manually tuning the number of goroutines provides better performance than just launching as many as required and letting the runtime handle the scheduling. I'm sure there are examples, just not very common ones, and from what I can tell this situation should only get better in Go 1.2 and beyond.

voidlogic Over a year ago

"I've not yet come across a situation where manually tuning the number of goroutines provides better performance than just launching as many as required" This is very bad idea if any of your goroutines could be doing blocking filesystem or syscall work. Say you have a 6 stage task. Its better to create a worker pool for each stage that has runtime.NumCPU * X (some scaling factor) for each stage. Each stage then reading from a toDo channel until it is closed and puts onto a nextStage channel until it closes it when it toDo channel is closed. This is not for performance but to avoid crashing.

Intermernet Over a year ago

@voidlogic Very good example. I've yet to do much with Syscalls in Go and I usually use buffered I/O for any FS work so haven't yet hit these problems.

|

VonC · Accepted Answer · 2014-10-29 20:16:31Z

is there a way to know how many goroutine are ready to run (not blocked) but not currently running.?

You will be able (Q4 2014/Q1 2015) to try and visualize those goroutines, with a new tracer being developed (Q4 2014): Go Execution Tracer

The trace contains:

events related to goroutine scheduling:

a goroutine starts executing on a processor,

a goroutine blocks on a synchronization primitive,

a goroutine creates or unblocks another goroutine;

network-related events:

a goroutine blocks on network IO,

a goroutine is unblocked on network IO;

syscalls-related events:

a goroutine enters into syscall,

a goroutine returns from syscall;

garbage-collector-related events:

GC start/stop,

concurrent sweep start/stop; and

user events.

By "processor" I mean a logical processor, unit of GOMAXPROCS.
Each event contains event id, a precise timestamp, OS thread id, processor id, goroutine id, stack trace and other relevant information (e.g. unblocked goroutine id).

Collectives™ on Stack Overflow

How to measure system overload when using GO

2 Answers 2

7 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

7 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related