4

I'm realizing my very first Golang application and I'm having some issues on using MAX CPU & Cores when using GoRoutines and I don't really know why.

When using a tool such as htop, CPU isn't used at its max power and only 1..4 threads are active at time. Also, all cores are active but they are around 25%-40% utilization.

I used:

func MaxParallelism() int {
    maxProcs := runtime.GOMAXPROCS(0)
    numCPU := runtime.NumCPU()
    if maxProcs < numCPU {
        return maxProcs
    }
    return numCPU
}

In order to get the number of goroutines to be instantiated.

Here's How I set up the application:

//Common Channel for the goroutines
tasks := make(chan *exec.Cmd, 64)

    //Spawning Max Cores Number goroutines (8)
    var wg sync.WaitGroup
    cores := MaxParallelism()
    for i := 0; i < cores; i++ {
        wg.Add(1)
        go func(num int, w *sync.WaitGroup) {
            defer w.Done()
            var (
                out []byte
                err error
            )
            for cmd := range tasks {
                out, err = cmd.Output()
                if err != nil {
                    fmt.Printf("Can't get stdout:", err)
                }
                . . .
            }
        }(i, &wg)
    }

    //Generate Tasks
    for i := 0; i < 100000; i++ {
      tasks <- exec.Command(cmd1, args...)
      tasks <- exec.Command(cmd2, args...)
    }

close(tasks)
// wait for the workers to finish
wg.Wait()

I share two screenshots of htop while executing the application

enter image description here enter image description here

I don't know If it May help but I'm launching it through Intellij Idea.

How do I use Max CPU properly and Cores?

Thanks in advance.

Improve this question
11
  • What makes you think it's not using enough CPU cores? What behavior do you expect, and what behavior do you observe? Commented Oct 28, 2016 at 16:13
  • I share a screen of htop while executing the application so you'll understand it. Basically, there's only 1 thread active at a time and all cores are used around 25%. Commented Oct 28, 2016 at 16:15
  • You say you're spawning 4 goroutines. How do you know you're getting 4? Commented Oct 28, 2016 at 16:51
  • That's an error, I spawn 8 of them using the for and giving it Max Cores. Commented Oct 28, 2016 at 16:56
  • Your goroutines are just spawning other processes. Your Go process isn't going any real work, so why would it use any CPU time? Commented Oct 28, 2016 at 17:05

3 Answers 3

Reset to default
4

Goroutines and threads are not the same. Ergo you should not expect any CPU affinity. See more for details here http://tleyden.github.io/blog/2014/10/30/goroutines-vs-threads/.

Here are some of the advantages of Goroutines over threads:

  • You can run more goroutines on a typical system than you can threads.
  • Goroutines have growable segmented stacks.
  • Goroutines have a faster startup time than threads.
  • Goroutines come with built-in primitives to communicate safely between themselves (channels).
  • Goroutines allow you to avoid having to resort to mutex locking when sharing data structures.
  • Goroutines are multiplexed onto a small number of OS threads, rather than a 1:1 mapping.
  • You can write massively concurrent servers without having to resort to evented programming.

EDIT: Answer to your question in the comments. There is no definitive answer. As others mentioned, it depends on what your code does. You may never end up using 100% CPU if you, for example, do I/O, which is slow. So no matter how many routines you start, I/O is slow. Contrarily, if your goroutine has a very tight loop doing just some computation then it's likely that 8 goroutines will consume your 8 CPUs completely.

Improve this answer
Sign up to request clarification or add additional context in comments.

1 Comment

Thank you for the answer. As I said I'm Pretty new to Go in general and I don't know many things since I'm using it from few days. So, basically, Hoe many goroutines should I instantiate with 4 cores and a solid I7? I would just like to use my resources at their maximum possibilities.
2

After reading around, getting some hints and trying some stuff, it came out that the code was written just fine and there were not real problems caused by it.

Essentially, Go scales pretty good and, the more tasks you generate, the more CPU resources are used.

For instance, setting an higher range on the for, e.g. 1.000.000, makes Go use all the cores available at ~55% of their possibilities.

I hope it will help someone.

Improve this answer

Comments

0

For the record, here's a code which will likely max out your CPU. I run concurrently 256 routines (#1 routine + those spawned within the for loop).

package main

func main() {
  for i := 0; i < 255; i++ {
    go func() {
      for {}
    }()
  }

  for {}
}

If you execute only one routine (the #1 routine), you'll see only one CPU core is used:

package main

func main() {
  for {}
}

If you execute 2 routines (the #1 routine + the spawned one), you'll see 2 CPU cores are used:

package main

func main() {
  go func(){
    for {}
  }()

  for {}
}

Go scales routines across CPU cores automatically.

Worth noting that go routines are not the same as functions.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.