looking for golang concurrency pattern

Question

Here is the problem I am trying to solve:

package main

import "fmt"

func workerA(work_in_chan <-chan int,work_out_chan chan<- int){
   for d := range work_in_chan {
    fmt.Println("A ",d)
    work_out_chan <- d
   }
}

func workerB(work_in_chan <-chan int,work_out_chan chan<- int){
   for d := range work_in_chan {
    fmt.Println("B ",d)
    work_out_chan <- d
   }
}

func account(account_chan <-chan int,final_chan chan<- int){

    wa_in := make(chan int)
    wa_out := make(chan int)
    wb_in := make(chan int)
    wb_out := make(chan int)

    go workerA(wa_in,wa_out)
    go workerB(wb_in,wb_out)

    for d := range account_chan {

        //TODO - dumb implementation starts here
        wa_in <- d
        <-wa_out

        wb_in <- d
        <-wb_out
        //TODO - dumb implementation ends here

        final_chan <- d
    }
}

func main() {

    account_chan := make(chan int, 100)
    final_chan := make(chan int, 100)

    go account(account_chan,final_chan)

    account_chan <- 1
    account_chan <- 2
    account_chan <- 3

    fmt.Println(<-final_chan)
    fmt.Println(<-final_chan)
    fmt.Println(<-final_chan)
}

The account goroutine receives incoming data on account_chan, executes some work on the data, and once complete sends the data to final_chan. The account work is done by workerA and workerB (order is not important),both must complete on the data before account sends it to final_data. There are a few requirements:

workerA and workerB are single goroutines
there should be a constant amount of goroutines at any one time (so no adding new goroutines for each new data item).

My pasted implementation is dumb since now workerA and workerB are never executing concurrently (as they could & should since they are completely independent of each other). So which concurrency pattern can I use to solve this problem?

1) If you seek help with large code samples: Please consider writing idiomatic Go code, e.g. account instead of account_chan (or at least accountChan. 2) Your use of channels is pretty weird: Passing ints through channels when a normal function or goroutine call would be enough looks strange. 3) You ask about concurrency; but do you mean parallelism? 4) Depending on the number of physical cores your worker goroutines might never run in parallel. 5) Buffer your wa_{in,out} channels? — Volker
– Volker, Commented Aug 29, 2013 at 15:40
@Volker 1)this is the naming convention used on out current project, so it came to me naturally, 2)this is a proof-of-concept sample, in my real example I am passing structs (not ints) 3) I benefit from concurrency (eg. workerA does something over a network, workerB writes to fs) 4)look at 3 5)doesn't solve my problem. — m2o
– m2o, Commented Aug 29, 2013 at 15:48
Are you looking for a solution where one worker can finish all of its items while the second is still working on the first. All with the requirement that nothing get sent on final_chan until both have finished the particular item? If workerA gets done first could it help workerB with its items? — deft_code
– deft_code, Commented Aug 30, 2013 at 16:59

deft_code · Accepted Answer · 2013-08-30 00:55:12Z

1

With the restrictions you've provided there isn't much that can be done. Simply reordering the channel operation to allows concurrency might be all you're looking for.

for d := range account_chan {
    wa_in <- d
    wb_in <- d

    <-wa_out
    <-wb_out

    final_chan <- d
}

play.golang.org/p/4d8hKyHTWq
The first time I saw this pattern, I worried "but what if B gets done first". It turns out the order doesn't really matter as both need to recv'd from.

An aside on style:
The provided snippet smells like it has too many channels and goroutines. But that may because this is a more complicated problem distilled down to a the essential parts. One thing that may actually be a problem is the out channel from the workers. Their output isn't used in the example and I can't see how it could be in a full listing. Either the values are copied in which case the out channel isn't needed (a sync.WaitGroup would be better) or they're not safe to share between the workers.

answered Aug 30, 2013 at 0:55

deft_code

59.8k31 gold badges145 silver badges225 bronze badges

Sign up to request clarification or add additional context in comments.

2 Comments

m2o Over a year ago

Yes, the worker out chan values are never used (they are only used to signal completion).I have changed there type to chan bool in this example: play.golang.org/p/Gua-raGaqb.

m2o Over a year ago

I have tried to implement this with a waitgroup, example here:play.golang.org/p/aFRfQ78Wu7. It looks cleaner to me somehow, but I prefer the first option (since I don't spawn goroutines for each data item. Yes there are only 2 at any one time but still...)

Kissaki · Accepted Answer · 2013-08-29 19:12:38Z

You pass the input for the workers and then block until you get their result separately.

// Give worker A work
wa_in <- d
// Wait until worker A finished
<-wa_out

// Give worker B work
wb_in <- d
// Wait until worker B finished
<-wb_out

Instead, use the select statement to wait for a result on one of two channels symultaneously:

func account(account_chan <-chan int,final_chan chan<- int){

    wa_in := make(chan int)
    wa_out := make(chan int)
    wb_in := make(chan int)
    wb_out := make(chan int)

    go workerA(wa_in,wa_out)
    go workerB(wb_in,wb_out)

    for d := range account_chan {
        wa_in <- d
        wb_in <- d
        for i := 0 ; i < 2; i++ {
            select {
                case <-wa_out:
                case <-wb_out:
            }
        }
        final_chan <- d
    }
}

http://play.golang.org/p/U0fk1yiqWL

Now, the two workers will run concurrently but the program is still guaranteed to wait for all the workers to finish.

Also see the concurrency patterns go doc.

Collectives™ on Stack Overflow

looking for golang concurrency pattern

2 Answers 2

2 Comments

Comments

Your Answer

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

2 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Related