Parallel processing in golang

Question

Given the following code:

package main

import (
    "fmt"
    "math/rand"
    "time"
)

func main() {
    for i := 0; i < 3; i++ {
        go f(i)
    }

    // prevent main from exiting immediately
    var input string
    fmt.Scanln(&input)
}

func f(n int) {
    for i := 0; i < 10; i++ {
        dowork(n, i)
        amt := time.Duration(rand.Intn(250))
        time.Sleep(time.Millisecond * amt)
    }
}

func dowork(goroutine, loopindex int) {
    // simulate work
    time.Sleep(time.Second * time.Duration(5))
    fmt.Printf("gr[%d]: i=%d\n", goroutine, loopindex)
}

Can i assume that the 'dowork' function will be executed in parallel?

Is this a correct way of achieving parallelism or is it better to use channels and separate 'dowork' workers for each goroutine?

Just in case anyone finds it interesting: This article explains how to do parallel for loops in Go. — rath
– rath, Commented Aug 3, 2014 at 16:09
@rath Many thanks - that website will prove a fantastic resource for me! — Simon Whitehead
– Simon Whitehead, Commented Aug 4, 2014 at 22:29

Kaushik Kakdey · Accepted Answer · 2023-08-28 04:38:05Z

33

Regarding GOMAXPROCS, you can find this in Go 1.5's release docs:

By default, Go programs run with GOMAXPROCS set to the number of cores available; in prior releases it defaulted to 1.

Regarding preventing the main function from exiting immediately, you could leverage WaitGroup's Wait function.

I wrote this utility function to help parallelize a group of functions:

import "sync"

// Parallelize parallelizes the function calls
func Parallelize(functions ...func()) {
    var waitGroup sync.WaitGroup
    waitGroup.Add(len(functions))

    defer waitGroup.Wait()

    for _, function := range functions {
        go func(f func()) {
            defer waitGroup.Done()
            f()
        }(function)
    }
}

So in your case, we could do this

func1 := func() {
    f(0)
}

func2 = func() {
    f(1)
}

func3 = func() {
    f(2)
}

Parallelize(func1, func2, func3)

If you wanted to use the Parallelize function, you can find it here https://github.com/shomali11/util

edited Aug 28, 2023 at 4:38

Kaushik Kakdey

6736 silver badges11 bronze badges

answered Jun 7, 2017 at 3:32

Raed Shomali

1,45516 silver badges8 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

Krishnadas PC Over a year ago

Is it possible to make parallel SQL queries with this?

Tyler Kropp Over a year ago

I would avoid using copy as a variable name, because it overrides a predeclared identifier: golang.org/ref/spec#Predeclared_identifiers.

Inanc Gumus · Accepted Answer · 2020-02-13 13:53:23Z

23

This answer is outdated. Please see this answer instead.

Your code will run concurrently, but not in parallel. You can make it run in parallel by setting GOMAXPROCS.

It's not clear exactly what you're trying to accomplish here, but it looks like a perfectly valid way of achieving concurrency to me.

edited Feb 13, 2020 at 13:53

Inanc Gumus

28.2k10 gold badges89 silver badges104 bronze badges

answered Aug 3, 2014 at 16:26

Evan

6,5731 gold badge31 silver badges30 bronze badges

1 Comment

Gnana Over a year ago

I followed the above mentioned article example. I am not getting same output. methods are getting executed sequentially not parallel.

Shantanu Bhadoria · Accepted Answer · 2017-06-04 13:23:18Z

0

f() will be executed concurrently but many dowork() will be executed sequentially within each f(). Waiting on stdin is also not the right way to ensure that your routines finished execution. You must spin up a channel that each f() pushes a true on when the f() finishes. At the end of the main() you must wait for n number of true's on the channel. n being the number of f() that you have spun up.

answered Jun 4, 2017 at 13:23

Shantanu Bhadoria

14.5k1 gold badge15 silver badges9 bronze badges

1 Comment

captncraig Over a year ago

sync.WaitGroup is better than a channel for that case.

Wolfie · Accepted Answer · 2019-01-03 20:01:28Z

This helped me when I was starting out.

    package main

    import "fmt"

    func put(number chan<- int, count int) {
        i := 0
        for ; i <= (5 * count); i++ {
            number <- i
        }
        number <- -1
    }

    func subs(number chan<- int) {
        i := 10
        for ; i <= 19; i++ {
            number <- i
        }
    }

    func main() {
        channel1 := make(chan int)
        channel2 := make(chan int)
        done := 0
        sum := 0

        go subs(channel2)
        go put(channel1, 1)
        go put(channel1, 2)
        go put(channel1, 3)
        go put(channel1, 4)
        go put(channel1, 5)

        for done != 5 {
            select {
            case elem := <-channel1:
                if elem < 0 {
                    done++
                } else {
                    sum += elem
                    fmt.Println(sum)
                }
            case sub := <-channel2:
                sum -= sub
                fmt.Printf("atimta : %d\n", sub)
                fmt.Println(sum)
            }
        }
        close(channel1)
        close(channel2)
    }

"Conventional cluster-based systems (such as supercomputers) employ parallel execution between processors using MPI. MPI is a communication interface between processes that execute in operating system instances on different processors; it doesn't support other process operations such as scheduling. (At the risk of complicating things further, because MPI processes are executed by operating systems, a single processor can run multiple MPI processes and/or a single MPI process can also execute multiple threads!)"

Zombo · Accepted Answer · 2021-04-19 01:12:23Z

0

You can add a loop at the end, to block until the jobs are done:

package main
import "time"

func f(n int, b chan bool) {
   println(n)
   time.Sleep(time.Second)
   b <- true
}

func main() {
   b := make(chan bool, 9)
   for n := cap(b); n > 0; n-- {
      go f(n, b)
   }
   for <-b {
      if len(b) == 0 { break }
   }
}

edited Apr 19, 2021 at 1:12

answered Apr 18, 2021 at 18:18

Zombo

1

Collectives™ on Stack Overflow

Parallel processing in golang

5 Answers 5

2 Comments

1 Comment

1 Comment

Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

5 Answers 5

2 Comments

1 Comment

1 Comment

Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related