2

I am diving into Golang and have a problem that I have been working on a few days and I just cant seem to grasp the concept of go routines and how they are used.

Basically I am, trying to generate millions of random records. I have functions that make the random data, and will create a giant .CSV file containing this data.

My questions is if it is possible to make this concurrent and speed things up?

My code is basically generate a random string, write string to file up to N times (where N is whatever you want).

My question is if its possible to even do this concurrently in order to reduce execution time. It seems that no matter how I approach this problem, I still get the same benchmark as if I did it without go routines.

This is a sample of what I have so far:

func worker(c chan string) {
        for {
                c <- /* Generate random data using other functions here */
        }
        close(c)
}

func writer(s string) {
        csvfile.WriteString(s)
}

func main(){    
    receive := make(chan string)

    for i := 0; i < 100; i++ {
        go worker(receive)
    }

    for i := 0; i < 10000; i++ {
        go writer(<-receive)
    }
}

Where I generate data, I am using tons and tons of function calls from: https://github.com/Pallinder/go-randomdata. Do you think that that could be where I am losing all this time?

Any help would be appreciated.

Improve this question
3
  • 1
    possible duplicate of Why doesn't the go statement execute in parallel? Commented Jul 6, 2015 at 16:19
  • Note that the default random number generator is safe for concurrent use, which it achieves by locking. If you're using lots of random numbers in goroutines you want to avoid them serializing access to random numbers by having an isolated *math.Rand per goroutine. You can do this via rand.New (and rand.NewSource, with different seeds for each of course). Commented Jul 6, 2015 at 18:18
  • Tangentially related: The fastest way to speed up file writing is write it all at once with one big fwrite, and if you need faster than that, then you can look into toggling off internal buffering in the processor caches & hard drive caches which is typically, I presume, system dependant. You may need a lower level language for this such as C if you get really detailed Commented Mar 24, 2019 at 4:53

2 Answers 2

Reset to default
1

I don't think you should be trying to use a go routine here. File writes are almost always atomic, you want to make the mechanism which writes to your file concurrent... That would require a complicated locking mechanism that ultimately probably won't improve application performance due to the write itself still being atomic.

If data generation were bottle necking your program then it would make sense to split that work off in go routines and write from that place where you get all the data. But 

for i := 0; i < 100; i++ {
    go worker(receive)
}

for {
    select {
    case item := <-receive:
         writer(item)
    case <-abort:
         cleanUp()
         return
    }
}

You can't just loop on some int while recieving from a channel and calling a function endlessly... You can receive from a channel in a select though. Or just by doing item := <-recieve which would block until one item gets read. In my example above I've provided some pseudo code to demonstrate more what your design should be in this case. You need an abort channel so you can get out of your go routines should you want to stop the application. It should probably finalize the write to your file and then close it before returning.

Improve this answer
Sign up to request clarification or add additional context in comments.

11 Comments

Ok that was something I always thought could hinder the performance when you are doing writes. Basically what you are saying is that no matter how fast you generate data, writing will always be the same unless we have some sort of "done" condition?
@art_vandelay I'm saying the OS and hard drive will dictate how fast you can write to a single file, not your apps level of concurrency. If you wrote to a bunch of files and aggregated them into one (assuming you're dealing with very large amounts of data now) then it could save you time perhaps. What I'm saying is that if generating data is slower than writing you can do the data generation concurrently to improve performance. Since your writes are inherently not concurrent you can't just do try writing from a bunch of different go routines to make performance better.
Wow, that makes perfect sense haha. Ok with this better understanding im going to try and re-write this badboy. I appreciate the input.
@art_vandelay glad I could help! In that case doing each write in it's own go routine makes perfect sense. You still may want to use a similar model to what I have above because it's somewhat necessary to make a responsive app but when you read you could call a method to do the write in a goroutine and pass the item into it so that execution continues and the next file begins writing as soon as data for it becomes available.
@art_vandelay I would recommend seeding in your for loop using the loop variable to ensure a different value each iteration and pass that into the worker method. ie func Seed(seed int64) with i as the arg, then add an arg to worker and call it with rand.Uint32() or 64 or whatever type you want your random value to be.
|
0

Try buffering your channel:

receive := make(chan string, 1000)

Write speed is limited by your disk, so there's only so much you can do to help by writing concurrently, and from what you're telling generating data concurrently doesn't help either.

Concurrency isn't the solution for anything slow, either accept you're at the limit or optimize.

Improve this answer

Comments

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.