I am trying to parallelize an operation in golang and save the results in a manner that I can iterate over to sum up afterwords.
I have managed to set up the parameters so that no deadlock occurs, and I have confirmed that the operations are working and being saved correctly within the function. When I iterate over the Slice of my struct and try and sum up the results of the operation, they all remain 0. I have tried passing by reference, with pointers, and with channels (causes deadlock).
I have only found this example for help: https://golang.org/doc/effective_go.html#parallel. But this seems outdated now, as Vector as been deprecated? I also have not found any references to the way this function (in the example) was constructed (with the func (u Vector) before the name). I tried replacing this with a Slice but got compile time errors.
Any help would be very appreciated. Here is the key parts of my code:
type job struct {
a int
b int
result *big.Int
}
func choose(jobs []Job, c chan int) {
temp := new(big.Int)
for _,job := range jobs {
job.result = //perform operation on job.a and job.b
//fmt.Println(job.result)
}
c <- 1
}
func main() {
num := 100 //can be very large (why we need big.Int)
n := num
k := 0
const numCPU = 6 //runtime.NumCPU
count := new(big.Int)
// create a 2d slice of jobs, one for each core
jobs := make([][]Job, numCPU)
for (float64(k) <= math.Ceil(float64(num / 2))) {
// add one job to each core, alternating so that
// job set is similar in difficulty
for i := 0; i < numCPU; i++ {
if !(float64(k) <= math.Ceil(float64(num / 2))) {
break
}
jobs[i] = append(jobs[i], Job{n, k, new(big.Int)})
n -= 1
k += 1
}
}
c := make(chan int, numCPU)
for i := 0; i < numCPU; i++ {
go choose(jobs[i], c)
}
// drain the channel
for i := 0; i < numCPU; i++ {
<-c
}
// computations are done
for i := range jobs {
for _,job := range jobs[i] {
//fmt.Println(job.result)
count.Add(count, job.result)
}
}
fmt.Println(count)
}
Here is the code running on the go playground https://play.golang.org/p/X5IYaG36U-
As long as the []Job slice is only modified by one goroutine at a time, there's no reason you can't modify the job in place.
for i, job := range jobs {
jobs[i].result = temp.Binomial(int64(job.a), int64(job.b))
}
https://play.golang.org/p/CcEGsa1fLh
You should also use a WaitGroup, rather than rely on counting tokens in a channel yourself.
Related
The code below starts a few workers. Each worker receives a value via a channel which is added to a map where the key is the worker ID and value is the number received. Finally, when I add all the values received, I should get an expected result (in this case 55 because that is what you get when you add from 1..10). In most cases, I am not seeing the expected output. What am I doing wrong here? I do not want to solve it by adding a sleep. I would like to identify the issue programmatically and fix it.
type counter struct {
value int
count int
}
var data map[string]counter
var lock sync.Mutex
func adder(wid string, n int) {
defer lock.Unlock()
lock.Lock()
d := data[wid]
d.count++
d.value += n
data[wid] = d
return
}
func main() {
fmt.Println(os.Getpid())
data = make(map[string]counter)
c := make(chan int)
for w := 1; w <= 3; w++ { //starting 3 workers here
go func(wid string) {
data[wid] = counter{}
for {
v, k := <-c
if !k {
continue
}
adder(wid, v)
}
}(strconv.Itoa(w)) // worker is given an ID
}
time.Sleep(1 * time.Second) // If this is not added, only one goroutine is recorded.
for i := 1; i <= 10; i++ {
c <- i
}
close(c)
total := 0
for i, v := range data {
fmt.Println(i, v)
total += v.value
}
fmt.Println(total)
}
Your code has two significant races:
The initialization of data[wid] = counter{} is not synchronized with other goroutines that may be reading and rewriting data.
The worker goroutines do not signal when they are done modifying data, which means your main goroutine may read data before they finish writing.
You also have a strange construct:
for {
v, k := <-c
if !k {
continue
}
adder(wid, v)
}
k will only be false when the channel c is closed, after which the goroutine spins as much as it can. This would be better written as for v := range c.
To fix the reading code in the main goroutine, we'll use the more normal for ... range c idiom and add a sync.WaitGroup, and have each worker invoke Done() on the wait-group. The main goroutine will then wait for them to finish. To fix the initialization, we'll lock the map (there are other ways to do this, e.g., to set up the map before starting any of the goroutines, or to rely on the fact that empty map slots read as zero, but this one is straightforward). I also took out the extra debug. The result is this code, also available on the Go Playground.
package main
import (
"fmt"
// "os"
"strconv"
"sync"
// "time"
)
type counter struct {
value int
count int
}
var data map[string]counter
var lock sync.Mutex
var wg sync.WaitGroup
func adder(wid string, n int) {
defer lock.Unlock()
lock.Lock()
d := data[wid]
d.count++
d.value += n
data[wid] = d
}
func main() {
// fmt.Println(os.Getpid())
data = make(map[string]counter)
c := make(chan int)
for w := 1; w <= 3; w++ { //starting 3 workers here
wg.Add(1)
go func(wid string) {
lock.Lock()
data[wid] = counter{}
lock.Unlock()
for v := range c {
adder(wid, v)
}
wg.Done()
}(strconv.Itoa(w)) // worker is given an ID
}
for i := 1; i <= 10; i++ {
c <- i
}
close(c)
wg.Wait()
total := 0
for i, v := range data {
fmt.Println(i, v)
total += v.value
}
fmt.Println(total)
}
(This can be improved easily, e.g., there's no reason for wg to be global.)
Well, I like #torek's answer but I wanted to post this answer as it contains a bunch of improvements:
Reduce the usage of locks (For such simple tasks, avoid locks. If you benchmark it, you'll notice a good difference because my code uses the lock only numworkers times).
Improve the naming of variables.
Remove usage of global vars (Use of global vars should always be as minimum as possible).
The following code adds a number from minWork to maxWork using numWorker spawned goroutines.
package main
import (
"fmt"
"sync"
)
const (
bufferSize = 1 // Buffer for numChan
numworkers = 3 // Number of workers doing addition
minWork = 1 // Sum from [minWork] (inclusive)
maxWork = 10000000 // Sum upto [maxWork] (inclusive)
)
// worker stats
type worker struct {
workCount int // Number of times, worker worked
workDone int // Amount of work done; numbers added
}
// workerMap holds a map for worker(s)
type workerMap struct {
mu sync.Mutex // Guards m for safe, concurrent r/w
m map[int]worker // Map to hold worker id to worker mapping
}
func main() {
var (
totalWorkDone int // Total Work Done
wm workerMap // WorkerMap
wg sync.WaitGroup // WaitGroup
numChan = make(chan int, bufferSize) // Channel for nums
)
wm.m = make(map[int]worker, numworkers)
for wid := 0; wid < numworkers; wid++ {
wg.Add(1)
go func(id int) {
var wk worker
// Wait for numbers
for n := range numChan {
wk.workCount++
wk.workDone += n
}
// Fill worker stats
wm.mu.Lock()
wm.m[id] = wk
wm.mu.Unlock()
wg.Done()
}(wid)
}
// Send numbers for addition by multiple workers
for i := minWork; i <= maxWork; i++ {
numChan <- i
}
// Close the channel
close(numChan)
// Wait for goroutines to finish
wg.Wait()
// Print stats
for k, v := range wm.m {
fmt.Printf("WorkerID: %d; Work: %+v\n", k, v)
totalWorkDone += v.workDone
}
// Print total work done by all workers
fmt.Printf("Work Done: %d\n", totalWorkDone)
}
I've been learning go recently and had a question about the behavior of slices when reallocation occurs. Assume I have a slice of pointers to a struct, such as:
var a []*A
If I were to pass this slice to another function, and my understanding is that internally this passes a slice header by value, that runs on a separate goroutine and just reads from the slice, while the function that launched the goroutine continues to append to the slice, is that a problem? For example:
package main
type A struct {
foo int
}
func main() {
a := make([]*A, 0, 100)
ch := make(chan int)
for i := 0; i < 100; i++ {
a = append(a, &A{i})
}
go read_slice(a, ch)
for i := 0; i < 100; i++ {
a = append(a, &A{i+100})
}
<-ch
}
func read_slice(a []*A, ch chan int) {
for i := range a {
fmt.Printf("%d ", a[i].foo)
}
ch <- 1
}
So from my understanding, as the read_slice() function is running on its own goroutine with a copy of the slice header, it has an underlying pointer to the current backing array and the size at the time it was called through which I can access the foo's.
However, when the other goroutine is appending to the slice it will trigger a reallocation when the capacity is exceeded. Does the go runtime not deallocate the memory to the old backing array being used in read_slice() since there is a reference to it in that function?
I tried running this with "go run -race slice.go" but that didn't report anything, but I feel like I might be doing something wrong here? Any pointers would be appreciated.
Thanks!
The GC does not collect the backing array until there are no references to the backing array. There are no races in the program.
Consider the scenario with no goroutines:
a := make([]*A, 0, 100)
for i := 0; i < 100; i++ {
a = append(a, &A{i})
}
b := a
for i := 0; i < 100; i++ {
b = append(b, &A{i+100})
}
The slice a will continue to reference the backing array with the first 100 pointers when append to b allocates a new backing array. The slice a is not left with a dangling reference to a backing array.
Now add the goroutine to the scenario:
a := make([]*A, 0, 100)
for i := 0; i < 100; i++ {
a = append(a, &A{i})
}
b := a
go read_slice(a, ch)
for i := 0; i < 100; i++ {
b = append(b, &A{i+100})
}
The goroutine can happily use slice a. There's no dangling reference.
Now consider the program in the question. It's functionaly identical to the last snippet here.
For purely educational purposes I created a base58 package. It will encode/decode a uint64 using the bitcoin base58 symbol chart, for example:
b58 := Encode(100) // return 2j
num := Decode("2j") // return 100
While creating the first tests I came with this:
func TestEncode(t *testing.T) {
var i uint64
for i = 0; i <= (1<<64 - 1); i++ {
b58 := Encode(i)
num := Decode(b58)
if num != i {
t.Fatalf("Expecting %d for %s", i, b58)
}
}
}
This "naive" implementation, tries to convert all the range from uint64 (From 0 to 18,446,744,073,709,551,615) to base58 and later back to uint64 but takes too much time.
To better understand how go handles concurrency I would like to know how to use channels or goroutines and perform the iteration across the full uint64 range in the most efficient way?
Could data be processed by chunks and in parallel, if yes how to accomplish this?
Thanks in advance.
UPDATE:
Like mention in the answer by #Adrien, one-way is to use t.Parallel() but that applies only when testing the package, In any case, by implementing it I found that is noticeably slower, it runs in parallel but there is no speed gain.
I understand that doing the full uint64 may take years but what I want to find/now is how could a channel or goroutine, may help to speed up the process (testing with small range 1<<16) probably by using something like this https://play.golang.org/p/9U22NfrXeq just as an example.
The question is not about how to test the package is about what algorithm, technic could be used to iterate faster by using concurrency.
This functionality is built into the Go testing package, in the form of T.Parallel:
func TestEncode(t *testing.T) {
var i uint64
for i = 0; i <= (1<<64 - 1); i++ {
t.Run(fmt.Sprintf("%d",i), func(t *testing.T) {
j := i // Copy to local var - important
t.Parallel() // Mark test as parallelizable
b58 := Encode(j)
num := Decode(b58)
if num != j {
t.Fatalf("Expecting %d for %s", j, b58)
}
})
}
}
I came up with this solutions:
package main
import (
"fmt"
"time"
"github.com/nbari/base58"
)
func encode(i uint64) {
x := base58.Encode(i)
fmt.Printf("%d = %s\n", i, x)
time.Sleep(time.Second)
}
func main() {
concurrency := 4
sem := make(chan struct{}, concurrency)
for i, val := uint64(0), uint64(1<<16); i <= val; i++ {
sem <- struct{}{}
go func(i uint64) {
defer func() { <-sem }()
encode(i)
}(i)
}
for i := 0; i < cap(sem); i++ {
sem <- struct{}{}
}
}
Basically, start 4 workers and calls the encode function, to notice/understand more this behavior a sleep is added so that the data can be printed in chunks of 4.
Also, these answers helped me to better understand concurrency understanding: https://stackoverflow.com/a/18405460/1135424
If there is a better way please let me know.
Consider the following application, designed to measure goroutine creation latency. Assume that we are running with GOMAXPROCS=2.
package main
import "fmt"
import "time"
const numRuns = 10000
type timeRecord struct{
Ts time.Time
Msg string
}
var timeStamps []timeRecord
func threadMain(done chan bool) {
timeStamps = append(timeStamps, timeRecord{time.Now(), "Inside thread"})
done <- true
}
func main() {
timeStamps = make([]timeRecord, 0, numRuns*2)
done := make(chan bool)
dummy := 0
for i := 0; i < numRuns; i++ {
timeStamps = append(timeStamps, timeRecord{time.Now(), "Before creation"})
go threadMain(done)
<-done
}
// Regularize
regularizedTime := make([]time.Duration, numRuns*2)
for i := 0; i < len(timeStamps) ; i++ {
regularizedTime[i] = timeStamps[i].Ts.Sub(timeStamps[0].Ts)
}
// Fake timetraced
fmt.Printf("%6d ns (+%6d ns): %s\n", 0, 0, timeStamps[0].Msg)
for i := 1; i < len(timeStamps) ; i++ {
fmt.Printf("%8d ns (+%6d ns): %s\n", regularizedTime[i], (regularizedTime[i] - regularizedTime[i-1]).Nanoseconds(), timeStamps[i].Msg)
}
}
On my server, this consistently outputs roughly a median 260 ns delta from Before creation to Inside thread. Now consider the following variation of the main method.
timeStamps = make([]timeRecord, 0, numRuns*2)
done := make(chan bool)
dummy := 0
for i := 0; i < numRuns; i++ {
timeStamps = append(timeStamps, timeRecord{time.Now(), "Before creation"})
go threadMain(done)
for j := 0; j < 1000; j++ {
dummy += j
}
<-done
}
Under this variation, the same time delta takes roughly 890 ns.
Obviously, the exact numbers are machine-specific, but the difference between the numbers is curious. Logically, if I am measuring between "Before creation" and "Inside thread", adding extra logic after the go statement seems like it should not increase that time, but it does.
Does anyone have a good idea for why the time increase is not occurring in the expected location?
Go scheduler is cooperative. It can switch the current goroutine to another one only at certain points of the program such as function calls, reading/writing to a channel, etc. I expect that the difference you observe is due to the fact that the goroutines are started later (after the added for loop).
func worker(id int, jobs <-chan int, results chan<- int) {
for j := range jobs {
fmt.Println("worker", id, "processing job", j)
time.Sleep(time.Second)
results <- j * 2
}
}
func main() {
t := time.Now()
fmt.Println(t)
jobs := make(chan int, 100)
results := make(chan int, 100)
for w := 1; w <= 4; w++ {
go worker(w, jobs, results)
}
for j := 1; j <= 20; j++ {
jobs <- j
}
close(jobs)
for a := 1; a <= 20; a++ {
<-results
}
t = time.Now()
fmt.Println(t)
}
I am confused of the "<-" and I can not find any related documents about "<-". So what is the difference between <- and =?? why I can not use = here?
The = operator deals with variable assignment as in most languages. It expresses the idea of wanting to update the value that an identifier references. The <- operator represents the idea of passing a value from a channel to a reference. If you think of the channel as a queue using an assignment operator = would assign the reference to the queue to the target variable. The receive operator <- is equivalent to dequeuing from the queue and assigning the value of the item to the target variable.
You cannot use the operators interchangeably because of a type mismatch. Please note the links to the Go specification which speak at greater length to the operators.
This is related to channels in Go. You are thinking it's related to assignment as in other languages. In your code, a value "j" is being sent to the channel "jobs".
https://gobyexample.com/channels
"=" is assignment,just like other language.
<- is a operator only work with channel,it means put or get a message from a channel.
channel is an important concept in go,especially in concurrent programming.you can try this Channel TourPage to see its using scene.
This example is meant to illustrate the usage of channels and of the <- notation, so if it's still confusing, providing annotation/explanation should help:
func worker(id int, jobs <-chan int, results chan<- int) {
// each worker will be a goroutine
// it has an integer id,
// the notation `jobs <-chan int` means `jobs` is a channel
// of ints, and that `worker` can only read from the channel
// the notation `results chan<- int` means results is also
// a channel of ints, and that `worker` can only write to
// the channel
for j := range jobs {
fmt.Println("worker", id, "processing job", j)
time.Sleep(time.Second)
results <- j * 2 // This notation means `j * 2` is
// written to the channel `results`
}
// so the job the worker is doing is multiplication by 2
}
func main() {
t := time.Now()
fmt.Println(t)
// jobs and results channels of capacity 100
jobs := make(chan int, 100)
results := make(chan int, 100)
// We start running 4 worker goroutines
for w := 1; w <= 4; w++ {
go worker(w, jobs, results)
}
// We load up the jobs queue with 20 jobs
for j := 1; j <= 20; j++ {
jobs <- j // This syntax means `j` is written
// to the channel `jobs`
}
close(jobs) // Signals that we won't be adding any more jobs
// We wait until we've pulled 20 expected
// results from the results queue
for a := 1; a <= 20; a++ {
<-results // This syntax means we pull an element out of
// the results queue and discard it (since we
// aren't assigning to any variable)
}
// Count how much time all that work took
t = time.Now()
fmt.Println(t)
}