I'm trying to generate combinations (e.g. every 6 combo out of 10 numbers) in parallel using Golang.
I have a solution that runs serially: Serial Code
For the case where the number of items(n) = 3 and the sample size (r) = 2, the output is:
Got [1 2]
Got [1 3]
Got [2 3]
Now I have tried parallelizing this and here is that code: Parallel Code. It doesn't work and I don't know why. For the same problem the output is:
Put [3 3] into the channel.
Got [3 3] out of the channel.
Put [3 3] into the channel.
Got [3 3] out of the channel.
Put [3 3] into the channel.
Got [3 3] out of the channel.
Any help much appreciated.
First, there is a data race.
$ go run -race main.go
==================
WARNING: DATA RACE
Write at 0x00c0000b0018 by goroutine 11:
main.combinationUtil()
/home/leaf/spike/stackoverflow/concomb/main.go:33 +0x11e
Previous write at 0x00c0000b0018 by goroutine 9:
main.combinationUtil()
/home/leaf/spike/stackoverflow/concomb/main.go:33 +0x11e
Goroutine 11 (running) created at:
main.combinationUtil()
/home/leaf/spike/stackoverflow/concomb/main.go:35 +0x211
Goroutine 9 (running) created at:
main.combinationUtil()
/home/leaf/spike/stackoverflow/concomb/main.go:35 +0x211
==================
==================
WARNING: DATA RACE
Read at 0x00c0000b0010 by goroutine 12:
runtime.slicecopy()
/usr/lib/go-1.13/src/runtime/slice.go:197 +0x0
main.combinationUtil()
/home/leaf/spike/stackoverflow/concomb/main.go:21 +0x39c
Previous write at 0x00c0000b0010 by goroutine 10:
main.combinationUtil()
/home/leaf/spike/stackoverflow/concomb/main.go:33 +0x11e
Goroutine 12 (running) created at:
main.combinationUtil()
/home/leaf/spike/stackoverflow/concomb/main.go:35 +0x211
Goroutine 10 (running) created at:
main.combinationUtil()
/home/leaf/spike/stackoverflow/concomb/main.go:40 +0x2dc
==================
This data race is caused by writing to the underlying array of data concurently - which means, underlying array of data (and thus the content of data) is shared on all goroutine. That is undesired.
In Go, what is under the hood of slice is a slice header (see reflect.SliceHeader), it keeps in three bytes the len, cap and ptr to the underlying array. When you copy it, it does not copy the underlying array, but rather only the header, so the underlying array is shared and raced - neither is desired.
To avoid that, just make a new copy of it. (using sync techniques may avoid race, but the content is still shared). However, that alone is a costing operation, both in terms of time and space. Whether that will worth the benefit of having parellel (if there is any benefit), is another topic and out of scope of this answer.
For example,
newdata := make([]int, r)
copy(newdata, data)
// current is included, put next at next location
newdata[index] = arr[i]
wg.Add(1)
go combinationUtil(ch, wg, arr, n, r, index+1, newdata, i+1)
// current is excluded, replace it with next (Note that
// i+1 is passed, but index is not changed)
wg.Add(1)
go combinationUtil(ch, wg, arr, n, r, index, newdata, i+1)
playground: https://play.golang.org/p/YebyCGapSMs
Note 1: This simple addition of a copy works only because that the recursion does not relies on change of part of data (data[index:]). Otherwise, there needs to be a back-copy for newdata to data.
Note 2: As I implied before, I doubt how efficient is this kind of parellel. There can be other ways of calculating combinations in parellel, which may utilize power of parellel calculation better, but requires quite different algorithms. However, I am not certain of that, so you should do your own research if interested.
Related
I am learning Go by going through A Tour of Go. One of the exercises there asks me to create a 2D slice of dy rows and dx columns containing uint8. My current approach, which works, is this:
a:= make([][]uint8, dy) // initialize a slice of dy slices
for i:=0;i<dy;i++ {
a[i] = make([]uint8, dx) // initialize a slice of dx unit8 in each of dy slices
}
I think that iterating through each slice to initialize it is too verbose. And if the slice had more dimensions, the code would become unwieldy. Is there a concise way to initialize 2D (or n-dimensional) slices in Go?
There isn't a more concise way, what you did is the "right" way; because slices are always one-dimensional but may be composed to construct higher-dimensional objects. See this question for more details: Go: How is two dimensional array's memory representation.
One thing you can simplify on it is to use the for range construct:
a := make([][]uint8, dy)
for i := range a {
a[i] = make([]uint8, dx)
}
Also note that if you initialize your slice with a composite literal, you get this for "free", for example:
a := [][]uint8{
{0, 1, 2, 3},
{4, 5, 6, 7},
}
fmt.Println(a) // Output is [[0 1 2 3] [4 5 6 7]]
Yes, this has its limits as seemingly you have to enumerate all the elements; but there are some tricks, namely you don't have to enumerate all values, only the ones that are not the zero values of the element type of the slice. For more details about this, see Keyed items in golang array initialization.
For example if you want a slice where the first 10 elements are zeros, and then follows 1 and 2, it can be created like this:
b := []uint{10: 1, 2}
fmt.Println(b) // Prints [0 0 0 0 0 0 0 0 0 0 1 2]
Also note that if you'd use arrays instead of slices, it can be created very easily:
c := [5][5]uint8{}
fmt.Println(c)
Output is:
[[0 0 0 0 0] [0 0 0 0 0] [0 0 0 0 0] [0 0 0 0 0] [0 0 0 0 0]]
In case of arrays you don't have to iterate over the "outer" array and initialize "inner" arrays, as arrays are not descriptors but values. See blog post Arrays, slices (and strings): The mechanics of 'append' for more details.
Try the examples on the Go Playground.
There are two ways to use slices to create a matrix. Let's take a look at the differences between them.
First method:
matrix := make([][]int, n)
for i := 0; i < n; i++ {
matrix[i] = make([]int, m)
}
Second method:
matrix := make([][]int, n)
rows := make([]int, n*m)
for i := 0; i < n; i++ {
matrix[i] = rows[i*m : (i+1)*m]
}
In regards to the first method, making successive make calls doesn't ensure that you will end up with a contiguous matrix, so you may have the matrix divided in memory. Let's think of an example with two Go routines that could cause this:
The routine #0 runs make([][]int, n) to get allocated memory for matrix, getting a piece of memory from 0x000 to 0x07F.
Then, it starts the loop and does the first row make([]int, m), getting from 0x080 to 0x0FF.
In the second iteration it gets preempted by the scheduler.
The scheduler gives the processor to routine #1 and it starts running. This one also uses make (for its own purposes) and gets from 0x100 to 0x17F (right next to the first row of routine #0).
After a while, it gets preempted and routine #0 starts running again.
It does the make([]int, m) corresponding to the second loop iteration and gets from 0x180 to 0x1FF for the second row. At this point, we already got two divided rows.
With the second method, the routine does make([]int, n*m) to get all the matrix allocated in a single slice, ensuring contiguity. After that, a loop is needed to update the matrix pointers to the subslices corresponding to each row.
You can play with the code shown above in the Go Playground to see the difference in the memory assigned by using both methods. Note that I used runtime.Gosched() only with the purpose of yielding the processor and forcing the scheduler to switch to another routine.
Which one to use? Imagine the worst case with the first method, i.e. each row is not next in memory to another row. Then, if your program iterates through the matrix elements (to read or write them), there will probably be more cache misses (hence higher latency) compared to the second method because of worse data locality. On the other hand, with the second method it may not be possible to get a single piece of memory allocated for the matrix, because of memory fragmentation (chunks spread all over the memory), even though theoretically there may be enough free memory for it.
Therefore, unless there's a lot of memory fragmentation and the matrix to be allocated is huge enough, you would always want to use the second method to get advantage of data locality.
With Go 1.18 you get generics.
Here is a function that uses generics to allow to create a 2D slice for any cell type.
func Make2D[T any](n, m int) [][]T {
matrix := make([][]T, n)
rows := make([]T, n*m)
for i, startRow := 0, 0; i < n; i, startRow = i+1, startRow+m {
endRow := startRow + m
matrix[i] = rows[startRow:endRow:endRow]
}
return matrix
}
With that function in your toolbox, your code becomes:
a := Make2D[uint8](dy, dx)
You can play with the code on the Go Playground.
Here a consive way to do it:
value := [][]string{}{[]string{}{"A1","A2"}, []string{}{"B1", "B2"}}
PS.: you can change "string" to the type of element you're using in your slice.
I understand it's handy in reusing a buffer rather than allocating every time hen using io.Copy. However, having printed its value several times, I get all zeros and the size of my buffer never changes. I tried to set the size to 8 and 1.
On a related note, to what value should I set my buffer size?
io.CopyBuffer() documents that:
func CopyBuffer(dst Writer, src Reader, buf []byte) (written int64, err error)
CopyBuffer is identical to Copy except that it stages through the provided buffer (if one is required) rather than allocating a temporary one. If buf is nil, one is allocated; otherwise if it has zero length, CopyBuffer panics.
If either src implements WriterTo or dst implements ReaderFrom, buf will not be used to perform the copy.
So io.CopyBuffer() copies data (bytes) from src to dst. The source is an io.Reader and the destination is an io.Writer. These interfaces allow you to read and write slices of bytes ([]byte).
In the general case to do the copying, we need a slice to read into from the source, which we can write into the destination. So io.CopyBuffer() needs a buffer. The buf param allows you to pass a byte slice if you already have one, and if you do so, that buffer will be used to do the job, so no new slice have to be allocated (which would be thrown away at the end of the operation).
What size should it be? The bigger the better, but no bigger is needed than the data you want to copy. Obviously bigger requires more memory, so there's a trade-off. Typically a few KB is a good compromise.
Note that as documented, if the source implements io.WriterTo or the destination implements io.ReaderFrom, those interfaces allow to read /
write without having to pass a slice, so in that case the buffer you pass will not be used. Like in this example:
srcData := []byte{1, 2, 3, 4, 5, 6, 7}
src := bytes.NewBuffer(srcData)
dst := &bytes.Buffer{}
buf := make([]byte, 10)
io.CopyBuffer(dst, src, buf)
fmt.Println(srcData)
fmt.Println(dst.Bytes())
fmt.Println(buf)
Which outputs (try it on the Go Playground):
[1 2 3 4 5 6 7]
[1 2 3 4 5 6 7]
[0 0 0 0 0 0 0 0 0 0]
Since we used bytes.Buffer as the source and destination (and since it implements both io.ReaderFrom and io.WriterTo), the buffer is not used.
Let's construct a source and destination that does not implement these interfaces, so we can test if / how our passed buffer is used.
For this, I will embed *bytes.Buffer in a struct, but specify a WriteTo and ReadFrom fields, so those methods will not get promoted from the embedded bytes.Buffer:
srcData := []byte{1, 2, 3, 4, 5, 6, 7}
src := struct {
WriteTo int // "disable" WriteTo method
*bytes.Buffer
}{0, bytes.NewBuffer(srcData)}
dst := struct {
ReadFrom int // "disable" ReadFrom method
*bytes.Buffer
}{0, &bytes.Buffer{}}
buf := make([]byte, 10)
io.CopyBuffer(dst, src, buf)
fmt.Println(srcData)
fmt.Println(dst.Bytes())
fmt.Println(buf)
This will output (try it on the Go Playground):
[1 2 3 4 5 6 7]
[1 2 3 4 5 6 7]
[1 2 3 4 5 6 7 0 0 0]
As you can see, the data from the source was read into the buffer, which then was written to the destination.
Note that you may pass a buffer smaller than the data to be copied, in which case reading / writing will be done in several iterations. In such cases, the data in the buffer may hold only the last iteration, and may only hold partial data (if the copied size is not an integer multiplication of the buffer size). It also depends on how the Read() method is implemented on the source, as Read() is not required to read the full slice passed to it.
Also note that io.CopyBuffer() does not document that the data written to the passed buffer is retained, it may get cleared / zeroed. Although this clearing is not implemented for performance reasons, but you should not count on it holding valid data after io.CopyBuffer() returns.
I am absolute newbie to concurrency in Go. I was trying to produce race condition with two gouroutines and wrote the following code:
var x int = 2
func main() {
go f1(&x)
go f2(&x)
time.Sleep(time.Second)
fmt.Println("Final value of x:", x)
}
func f1(px *int) {
for i := 0; i < 10; i++ {
*px = *px * 2
fmt.Println("f1:", *px)
}
}
func f2(px *int) {
for i := 0; i < 10; i++ {
*px = *px + 1
fmt.Println("f2:", *px)
}
}
And in each variant of output there are all the f2's output lines in console and only after that there are f1's outputs. Here's an example:
f2: 3
f2: 4
f2: 5
f2: 6
f2: 7
f2: 8
f2: 9
f2: 10
f2: 21
f2: 22
f1: 20
f1: 44
f1: 88
f1: 176
f1: 352
f1: 704
f1: 1408
f1: 2816
f1: 5632
f1: 11264
Final value of x: 11264
But you can see that indeed some of f1's executions were made in between of f2's executions:
f2: 10
f2: 21
So that i have two questions:
Why all the Printl() of f1 executes strictly after execution of f2's Println() (I thought they must be mixed somehow)
Why when I change the order of goroutines in the code
go f2(&x)
go f1(&x)
instead of
go f1(&x)
go f2(&x)
the order of output lines changes vice versa, f1's first, f2'2 second. I mean how the order of gouroutines in code affects their execution?
Firstly, the behavior you are seeing is due to a tight-loop. The Go scheduler cannot reasonable know how to share the workload, since your loops are short and don't take significant amounts of time (below the 10ms threshold for instance)
How the Go scheduler works is a very board topic and has changed across Go versions, but to quote this artcile:
If loops don’t contain any preemption points (like function calls, or
allocate memory), they will prevent other goroutines from running
with preemption typically not occurring until 10ms later.
In the real world a processing loop will typically invoke some blocking call (DB operation, REST/gRPC call etc) - this will give a cue to the Go scheduler to set other goroutines as "Runnable". You can simulate this in your code by inserting a time.Sleep into your loops: https://play.golang.org/p/_C3QOUMNOaU
There are other methods to relinquish (runtime.Gosched) but these techniques should generally be avoided. Avoid tight-loops and let the schedule do its thing.
Execution Ordering
When multiple goroutines are involved - and as #Marc commented - without coordination between the goroutines, the order of execution is non-deterministic.
Go has many tools at it's disposal to coordinate go-routine activities:
channels
sync package's WaitGroup, Mutex etc.
that block the current goroutine and allow other goroutines to be scheduled. Using these techniques guarantee the precise ordering of larger tasks.
Predicting the execution order, however, of individual instructions that run between these coordination checkpoints is impossible.
This question already has answers here:
Why does append modify passed slice
(5 answers)
Closed 10 months ago.
When running the following program:
package main
import "fmt"
func main() {
edges := [][]int{{1,2}, {2,3}, {3,4}, {1,4}, {1,5}}
printSlice2d(edges)
_ = append(edges[:0], edges[1:]...)
printSlice2d(edges)
}
func printSlice2d(s [][]int) {
fmt.Printf("len=%d cap=%d %v\n", len(s), cap(s), s)
}
I get the following output:
len=5 cap=5 [[1 2] [2 3] [3 4] [1 4] [1 5]]
len=5 cap=5 [[2 3] [3 4] [1 4] [1 5] [1 5]]
I don't understand why edges is being changed by calling append. I would expect the first and second lines to be the same. I've checked the specification, but I can't seem to find anything that would explain this behaviour this kind of input.
Slice uses array internally to store the data.
append function does not create a new array if it is not necessary to increase array size beyond its current capacity. Instead it copies new data elements into existing array. After that function returns reference to a new slice that internally uses the same array.
You can read more in this article - https://blog.golang.org/go-slices-usage-and-internals
As mentioned earlier, re-slicing a slice doesn't make a copy of the
underlying array. The full array will be kept in memory until it is no
longer referenced. Occasionally this can cause the program to hold all
the data in memory when only a small piece of it is needed.
This is what is going on in this line:
_ = append(edges[:0], edges[1:]...)
second append function argument (edges[1:]...) copies 4 items of edges into temp var. Its value - [{2,3}, {3,4}, {1,4}, {1,5}]
these values are copied into array that edges uses internally to store data. That overrides all items except of last one. This is where edges is mutated.
append returns reference to a new slice that internally uses the same array to store the data as edges
returned slice is ignored and will be garbage collected, but that does not matter for edges.
That is why you see changed values when you check edges after performing append on it.
edges[:0] is a slice of length 0 starting at index 0 of the underlying array of edges.
To this slice, you append another slice, the slice of length 4 starting at index of the underlying array ofedges`. That gives you the first 4 elements of the result you see in the second line.
Then you print edges, which is a slice with an underlying array of 5 elements, whose last 4 you just shifted one element lower. The last element is duplicated. in the edges array.
If you look at the result of the append, then you'd see a slice with length 4, cap 5, the first 4 elements of the underlying array of edges.
If you expected the two lines to be the same, maybe you tried to do:
append(edges[:1],edges[1:]...)
I'm having trouble understanding the use of goroutines and channels in the tour of go. Referencing the code below from:
"https://tour.golang.org/concurrency/2"
package main
import "fmt"
func sum(s []int, c chan int) {
sum := 0
for _, v := range s {
sum += v
}
c <- sum // send sum to c
}
func main() {
s := []int{7, 2, 8, -9, 4, 0}
c := make(chan int)
go sum(s[:len(s)/2], c)
go sum(s[len(s)/2:], c)
x, y := <-c, <-c // receive from c
fmt.Println(x, y, x+y)
}
It runs the sum functions using goroutines with the 'go' keyword in front of them, but all they do is send in values to a channel. They shouldn't have to be run with go routines. However, when removing the go keyword to run the functions as normal I get this error:
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [chan send]:
main.sum(0xc420059f10, 0x3, 0x6, 0xc420088060)
/tmp/compile33.go:10 +0x5a
main.main()
/tmp/compile33.go:17 +0x99
I can't understand why goroutines are needed here. I might be misunderstanding the concept and would appreciate if anyone more familiar with go could shed some light.
Thanks,
Others have already pointed out in the comments that in terms of being an example, you obviously don't need to write this program with channels.
From your question, though, it sounds like you're curious about why separate goroutines are needed in order for the program to run.
To answer that, it might be helpful to think about how this might work in a world where you were only thinking about threads. You've got your main thread, and that thread invokes sum(s[:len(s)/2], c). So now the main thread gets to the c <- sum line in sum, and it blocks, because the channel is unbuffered - meaning there must be another listening thread to "take" from that channel in order for our main thread to put something into it. In other words, the threads are passing messages directly to each other, but there's no second thread to pass to. Deadlock!
In this context, goroutines and threads are functionally equivalent. So without a second goroutine, you've got your main goroutine calling...but nobody's picking up the telephone on the other end.