Why am I getting deadlock and how to fix it (synch)? - go

Getting deadlock somewhere. What is causing it and how should it be fixed?
Trying to create a molecule creation. Function Make takes input as a string:
func testMolecule() {
water := New()
water.Make("HOH") // water.Make("OOHHHH")
fmt.Println(water.Molecules())
// Output: 1
}
Function should return number of molecules created. Molecule should have 1 O-molecule and 2 H-molecule.
Produce function shouldn't be done till all molecules are created.
import (
"sync"
)
// Water structure holds the synchronization primitives and
// data required to solve the water molecule problem.
// moleculeCount holds the number of molecules formed so far.
// result string contains the sequence of "H" and "O".
// wg WaitGroup is used to wait for goroutine completion.
type Water struct {
sync.Mutex
wg sync.WaitGroup
cond *sync.Cond
moleculeCount int
result string
assignedO int
assignedH int
finished bool
}
// New initializes the water structure.
func New() *Water {
water := &Water{
moleculeCount: 0,
result: "",
assignedO: 0,
assignedH: 0,
finished: false,
}
water.cond = sync.NewCond(water)
return water
}
// releaseOxygen produces one oxygen atom if no oxygen atom is already present.
// If an oxygen atom is already present, it will block until enough hydrogen
// atoms have been produced to consume the atoms necessary to produce water.
//
// The w.wg.Done() must be called to indicate the completion of the goroutine.
func (w *Water) releaseOxygen() {
defer w.wg.Done()
for {
w.Lock()
if w.assignedO == 0 {
w.assignedO = 1
}
for !(w.assignedH == 2) {
w.cond.Wait()
}
w.result = w.result + "O"
w.cond.Broadcast()
w.Unlock()
}
}
// releaseHydrogen produces one hydrogen atom unless two hydrogen atoms are already present.
// If two hydrogen atoms are already present, it will block until another oxygen
// atom has been produced to consume the atoms necessary to produce water.
//
// The w.wg.Done() must be called to indicate the completion of the goroutine.
func (w *Water) releaseHydrogen() {
defer w.wg.Done()
for {
w.Lock()
if w.assignedH < 2 {
w.assignedH = w.assignedH + 1
}
for !(w.assignedO == 1) && w.assignedH == 2 {
w.cond.Wait()
}
w.result = w.result + "HH"
w.cond.Broadcast()
w.Unlock()
}
}
// produceMolecule forms the water molecules.
func (w *Water) produceMolecule(done chan bool) {
for {
w.Lock()
for w.assignedO == 1 && w.assignedH == 2 && !w.finished {
w.cond.Wait()
}
if w.assignedO == 0 && w.assignedH == 0 && w.finished {
w.Unlock()
break
}
w.moleculeCount = w.moleculeCount + 1
w.assignedH = 0
w.assignedO = 0
w.cond.Broadcast()
w.Unlock()
}
done <- true
}
func (w *Water) finish() {
w.Lock()
w.finished = true
w.cond.Broadcast()
w.Unlock()
}
func (w *Water) Molecules() int {
return w.moleculeCount
}
// Make returns a sequence of water molecules derived from the input of hydrogen and oxygen atoms.
func (w *Water) Make(input string) string {
done := make(chan bool)
go w.produceMolecule(done)
for _, ch := range input {
w.wg.Add(1)
switch ch {
case 'O':
go w.releaseOxygen()
case 'H':
go w.releaseHydrogen()
}
}
w.wg.Wait()
w.finish()
<-done
return w.result
}

Function should return number of molecules created. Molecule should have 1 O-molecule and 2 H-molecule.
You don't need any synchronization or concurrency for that. Just count the Os, the Hs, and return the lesser of number of O or half number of H.
https://play.golang.org/p/3FaJDMi5ER7
I know from the previous question that you probably are trying to build some sort of simulation or model, but I don't understand your goals for that.

Related

How does goroutine schedule work with GOMAXPROCS?

I am so confused by goroutines.
Here is the code
func main() {
// runtime.GOMAXPROCS(1)
go spinner(100 * time.Millisecond)
const n = 45
fibN := fib(n) // slow
fmt.Printf("\rFibonacci(%d) = %d\n", n, fibN)
}
func spinner(delay time.Duration) {
for {
for _, r := range `-\|/` {
fmt.Printf("\r%c", r)
time.Sleep(delay)
}
}
}
func fib(x int) int {
if x < 2 {
return x
}
return fib(x-1) + fib(x-2)
}
That is a simple goroutine tutorial code which using goroutine to show an ASCII animation when calculating Fibonacci.
When I set GOMAXPROCS to 1, I think that there will be only one thread to execute goroutine and the Fibonacci function doesn't have any point to yield to animation goroutine. But this demo still works. It shows animation while doing calculating.
How does Go do this without goroutine switching?
Among others : the compiler inserts potential switch points at each function call, so each recursive call to fib(...) can yield to the "spinner" goroutine.
If you try to implement fib without any function call for example :
// note : this is a truly horrific way to compute the Fibonacci sequence,
// don't do this at home
// simulate the "compute Fibonacci recursively" algorithm,
// but without any function call
func fib(n int) int {
var res = 0
var stack []int
stack = append(stack, n)
for len(stack) > 0 {
// pop :
n = stack[len(stack)-1]
stack = stack[0 : len(stack)-1]
if n < 2 {
res += n
continue
}
// else : push 'n-1' and 'n-2' on the stack
stack = append(stack, n-1, n-2)
}
return res
}
https://play.golang.org/p/pdoAaBwyscr
you should see your spinner 'stuck'

How to get "sum" of true bools

In this scenario I have 3x boolean vars (already set to either true or false)
The goal here is to identify if more than one of the set of bool vars are set to true
Right now I have the following written, which does work:
boolval := 0
if *firstbool {
boolval++
}
if *secondbool {
boolval++
}
if *thirdbool {
boolval++
}
if boolval > 1 {
// More than 1 of the bool vars are true
}
I always flag my logic if I am writing successive if{} statements, so I figured I would ask you geniuses how you would accomplish this.
identify if more than one of the set of bool vars are set to true
For example, write a set function:
package main
import "fmt"
func nTrue(b ...bool) int {
n := 0
for _, v := range b {
if v {
n++
}
}
return n
}
func main() {
v1, v2, v3 := true, false, true
fmt.Println(nTrue(v1, v2, v3))
if nTrue(v1, v2, v3) > 1 {
// . . .
}
// Or
if n := nTrue(v1, v2, v3); n > 1 {
// . . .
}
// Or
n := nTrue(v1, v2, v3)
if n > 1 {
// . . .
}
}
Playground: https://play.golang.org/p/g3WCN6BgGly
Output:
2
For example, range over the set,
package main
import "fmt"
func main() {
v1, v2, v3 := true, false, true
boolval := 0
for _, v := range []bool{v1, v2, v3} {
if v {
boolval++
}
}
if boolval > 1 {
// . . .
}
fmt.Println(boolval > 1, boolval)
}
Playground: https://play.golang.org/p/R6UGb8YYEFw
Output:
true 2
Depending on how you're getting the values of firstbool et. al, there might be a more idiomatic approach you should take here. Consider that channels and goroutines can handle doing the accounting in the background. If your booleans are the result of some heavier operations, it might make sense to do something like:
package main
import "fmt"
func firstFunc(ch chan bool) {
// assume this works, true!
ch <- true
}
func secondFunc(ch chan bool) {
// assume this fails, false!
ch <- false
}
func thirdFunc(ch chan bool) {
// assume this works, true!
ch <- true
}
func listener(ch chan bool, numCallers int) <-chan bool {
outch := make(chan bool)
THRESHOLD := 2
go func(outch chan<- bool) {
count := 0
trues := 0
for count < numCallers && trues < THRESHOLD {
val := <-ch
if val {
trues++
}
count++
}
outch <- trues >= THRESHOLD
}(outch)
return outch
}
func main() {
ch := make(chan bool)
resultch := listener(ch, 3)
go firstFunc(ch)
go secondFunc(ch)
go thirdFunc(ch)
if <-resultch {
fmt.Println("Two or more processes succeeded")
}
}
However this is way over-engineered for a simple need, so consider this only if this pattern fits the larger design of your application.

How to group by then merge slices with duplicated values in Go

Excuse me, this is my first Stackoverflow question, so, any tips/advice on what I can do to improve it would be wonderful, in addition to some help.
The problem:
I have a slice that I am trying to group into smaller slices by certain criteria. I then need to merge the newly created slices with each other if they contain any of the same values in the slice. (Essentially, appending slices together that have "overlapping" values).
Some additional notes about the problem:
The number of items in the original slice will likely be between 1-50, in most cases, with outliers rarely exceeding 100.
Once gropued, the size of the 'inside' slices will be between 1-10 values.
Performance is a factor, as this operation will be run as part of a webservice where a single request will perform this operation 20+ times, and there can be many (hundreds - thousands) of requests per minute at peak times. However, clarity of code is also important.
My implementation is using ints, the final implementation would have more complex structs, though I was considering making a map and then use the implementation shown below based upon the keys. Is this a good idea?
I have broken the problem down into a few steps:
Create a 2D slice with groupings of values, based up criteria (the initial grouping phase)
Attempt to merge slices in place if they include a duplicated value.
I am running into two problems:
First, I think my implementation might not scale super well, as it tends to have some nested loops (however, these loops will be iterating on small slices, so that might be ok)
Second, my implementation is requiring an extra step at the end to remove duplicated values, ideally we should remove it.
Input: [ 100, 150, 300, 350, 600, 700 ]
Expected Output: [[100 150 300 350] [600 700]]
This is with the 'selection criteria' of grouping values that are within 150 units of at least one other value in the slice.
And the code (Go Playground link) :
package main
import (
"fmt"
"sort"
)
func filter(vs []int, f func(int) bool) []int {
vsf := make([]int, 0)
for _, v := range vs {
if f(v) {
vsf = append(vsf, v)
}
}
return vsf
}
func unique(intSlice []int) []int {
keys := make(map[int]bool)
list := []int{}
for _, entry := range intSlice {
if _, value := keys[entry]; !value {
keys[entry] = true
list = append(list, entry)
}
}
return list
}
func contains(intSlice []int, searchInt int) bool {
for _, value := range intSlice {
if value == searchInt {
return true
}
}
return false
}
func compare(a, b []int) bool {
if len(a) != len(b) {
return false
}
if (a == nil) != (b == nil) {
return false
}
b = b[:len(a)]
for i, v := range a {
if v != b[i] {
return false
}
}
return true
}
func main() {
fmt.Println("phase 1 - initial grouping")
s := []int{100, 150, 300, 350, 600, 700}
g := make([][]int, 0)
// phase 1
for _, v := range s {
t := filter(s, func(i int) bool { return i - v >= -150 && i - v <= 150 })
for _, v1 := range t {
t1 := filter(s, func(i int) bool { return i - v1 >= -150 && i - v1 <= 150})
t = unique(append(t, t1...))
sort.Ints(t)
}
g = append(g, t)
fmt.Println(g)
}
// phase 2
fmt.Println("phase 2 - merge in place")
for i, tf := range g {
for _, death := range tf {
if i < len(g) - 1 && contains(g[i+1], death) {
g[i+1] = unique(append(g[i], g[i+1]...))
g = g[i+1:]
} else if i == len(g) - 1 {
fmt.Println(g[i], g[i-1])
// do some cleanup to make sure the last two items of the array don't include duplicates
if compare(g[i-1], g[i]) {
g = g[:i]
}
}
}
fmt.Println(i, g)
}
}
Not sure what you are actually asking, and the problem isn't fully defined.
So here's a version that is more efficient
If input is not sorted and output order matters, then this is a bad solution.
Here it is (on Play)
package main
import (
"fmt"
)
// Input: [ 100, 150, 300, 350, 600, 700 ] Expected Output: [[100 150 300 350] [600 700]]
func main() {
input := []int{100, 150, 300, 350, 600, 700}
fmt.Println("Input:", input)
fmt.Println("Output:", groupWithin150(input))
}
func groupWithin150(ints []int) [][]int {
var ret [][]int
// Your example input was sorted, if the inputs aren't actually sorted, then uncomment this
// sort.Ints(ints)
var group []int
for idx, i := range ints {
if idx > 0 && i-150 > group[len(group)-1] {
ret = append(ret, group)
group = make([]int, 0)
}
group = append(group, i)
}
if len(group) > 0 {
ret = append(ret, group)
}
return ret
}

communicating with n goroutines

I am trying to implement fibonacci recursion in golang using n goroutines with communicating via channels.
I am returning an integer from the function, but i am actually just sending the sum of f(n-1) +f(n-2) over channel c but this is not working correctly. It prints the first two values correct, and every value after is just 1.
package main
import "fmt"
// Fibonacci in a recursive version
func fiboR(n int, c chan int ) int {
if(n == 0){
c <- 0
return 0
} else if n == 1 {
c <- 1
return 1
} else{
c <- fiboR(n-1,c) + fiboR(n-2,c)
return fiboR(n-1,c) + fiboR(n-2,c)
}
}
func main() {
for i := 0; i < 10; i++ {
procchan := make(chan int)
go fiboR(i,procchan)
fmt.Println(i,<-procchan )
}
}
Also is it possible to use channels for receiving the two recursive calls?
Your solution will try to output more than the one value you extract from the channel as you increase the value of i.
What your code will try to send to the channel for each i:
0: 0
1: 1
2: 1,0,1
3: 1,0,1,1,2
4: 1,0,1,1,2,1,0,1,3
...
Since you create a new channel for each i and then only extract one value you will always get the first value in the line above.
If you try to run it with these modifications it will output what you wanted (https://play.golang.org/p/_mn3l5x8iZ).
package main
import "fmt"
// Fibonacci in a recursive version
func fiboRchan(n int, c chan int) {
c <- fiboR(n)
}
func fiboR(n int) int {
if n == 0 {
return 0
} else if n == 1 {
return 1
} else {
return fiboR(n-1) + fiboR(n-2)
}
}
func main() {
for i := 0; i < 10; i++ {
procchan := make(chan int)
go fiboRchan(i, procchan)
fmt.Println(i, <-procchan)
}
}
Adding to #nissefors answer, the main process there is most likely a sequential one because in the for loop you would be waiting on the channel to return and then proceed to the next iteration.
A minor modification in the main function could fire all the fibonaccis at once and then in a separate for loop the channels that are corresponding to each go routine can be accessed
Playground URL: https://play.golang.org/p/7e3JnWeSp6
package main
import "fmt"
// Fibonacci in a recursive version
func fiboRchan(n int, c chan int) {
fmt.Println("PROCESSING FOR %d", n)
c <- fiboR(n)
}
func fiboR(n int) int {
if n == 0 {
return 0
} else if n == 1 {
return 1
} else {
return fiboR(n-1) + fiboR(n-2)
}
}
func main() {
var arr[10]chan int
for i := 0; i < 10; i++ {
procchan := make(chan int)
arr[i] = procchan
go fiboRchan(i, procchan)
}
// By now all the go routines are fired
// Now iterate through the channel array and read from the
// respective channel
for i:=0; i< 10; i++ {
fmt.Println(i, <-arr[i])
}
}

Golang share big chunk of data between goroutines

I have a need to read structure fields set from another goroutine, afaik doing so directly even when knowing for sure there will be no concurrent access(write finished before read occurred, signaled via chan struct{}) may result in stale data
Will sending a pointer to the structure(created in the 1st goroutine, modified in the 2nd, read by the 3rd) resolve the possible staleness issue, considering I can guarantee no concurrent access?
I would like to avoid copying as structure is big and contains huge Bytes.Buffer filled in the 2nd goroutine, I need to read from the 3rd
There is an option for locking, but seems like an overkill considering I know that there will be no concurrent access
There are many answers to this, and it depends to your data structure and program logic.
see: How to lock/synchronize access to a variable in Go during concurrent goroutines?
and: How to use RWMutex in Golang?
1- using Stateful Goroutines and channels
2- using sync.Mutex
3- using sync/atomic
4- using WaitGroup
5- using program logic(Semaphore)
...
1: Stateful Goroutines and channels:
I simulated very similar sample(imagine you want to read from one SSD and write to another SSD with different speed):
In this sample code one goroutine (named write) does some job prepares data and fills the big struct, and another goroutine (named read) reads data from big struct then do some job, And the manger goroutine, guarantee no concurrent access to same data.
And communication between three goroutines done with channels. And in your case you can use pointers for channel data, or global struct like this sample.
output will be like this:
mean= 36.6920166015625 stdev= 6.068973186592054
I hope this helps you to get the idea.
Working sample code:
package main
import (
"fmt"
"math"
"math/rand"
"runtime"
"sync"
"time"
)
type BigStruct struct {
big []uint16
rpos int
wpos int
full bool
empty bool
stopped bool
}
func main() {
wg.Add(1)
go write()
go read()
go manage()
runtime.Gosched()
stopCh <- <-time.After(5 * time.Second)
wg.Wait()
mean := Mean(hist)
stdev := stdDev(hist, mean)
fmt.Println("mean=", mean, "stdev=", stdev)
}
const N = 1024 * 1024 * 1024
var wg sync.WaitGroup
var stopCh chan time.Time = make(chan time.Time)
var hist []int = make([]int, 65536)
var s *BigStruct = &BigStruct{empty: true,
big: make([]uint16, N), //2GB
}
var rc chan uint16 = make(chan uint16)
var wc chan uint16 = make(chan uint16)
func next(pos int) int {
pos++
if pos >= N {
pos = 0
}
return pos
}
func manage() {
dataReady := false
var data uint16
for {
if !dataReady && !s.empty {
dataReady = true
data = s.big[s.rpos]
s.rpos++
if s.rpos >= N {
s.rpos = 0
}
s.empty = s.rpos == s.wpos
s.full = next(s.wpos) == s.rpos
}
if dataReady {
select {
case rc <- data:
dataReady = false
default:
runtime.Gosched()
}
}
if !s.full {
select {
case d := <-wc:
s.big[s.wpos] = d
s.wpos++
if s.wpos >= N {
s.wpos = 0
}
s.empty = s.rpos == s.wpos
s.full = next(s.wpos) == s.rpos
default:
runtime.Gosched()
}
}
if s.stopped {
if s.empty {
wg.Done()
return
}
}
}
}
func read() {
for {
d := <-rc
hist[d]++
}
}
func write() {
for {
wc <- uint16(rand.Intn(65536))
select {
case <-stopCh:
s.stopped = true
return
default:
runtime.Gosched()
}
}
}
func stdDev(data []int, mean float64) float64 {
sum := 0.0
for _, d := range data {
sum += math.Pow(float64(d)-mean, 2)
}
variance := sum / float64(len(data)-1)
return math.Sqrt(variance)
}
func Mean(data []int) float64 {
sum := 0.0
for _, d := range data {
sum += float64(d)
}
return sum / float64(len(data))
}
5: another way(faster) for some use cases:
here another way to use shared data structure for read job/write job/ processing job which it was separated in first post, now here doing same 3 jobs without channels and without mutex.
working sample:
package main
import (
"fmt"
"math"
"math/rand"
"time"
)
type BigStruct struct {
big []uint16
rpos int
wpos int
full bool
empty bool
stopped bool
}
func manage() {
for {
if !s.empty {
hist[s.big[s.rpos]]++ //sample read job with any time len
nextPtr(&s.rpos)
}
if !s.full && !s.stopped {
s.big[s.wpos] = uint16(rand.Intn(65536)) //sample wrire job with any time len
nextPtr(&s.wpos)
}
if s.stopped {
if s.empty {
return
}
} else {
s.stopped = time.Since(t0) >= 5*time.Second
}
}
}
func main() {
t0 = time.Now()
manage()
mean := Mean(hist)
stdev := StdDev(hist, mean)
fmt.Println("mean=", mean, "stdev=", stdev)
d0 := time.Since(t0)
fmt.Println(d0) //5.8523347s
}
var t0 time.Time
const N = 100 * 1024 * 1024
var hist []int = make([]int, 65536)
var s *BigStruct = &BigStruct{empty: true,
big: make([]uint16, N), //2GB
}
func next(pos int) int {
pos++
if pos >= N {
pos = 0
}
return pos
}
func nextPtr(pos *int) {
*pos++
if *pos >= N {
*pos = 0
}
s.empty = s.rpos == s.wpos
s.full = next(s.wpos) == s.rpos
}
func StdDev(data []int, mean float64) float64 {
sum := 0.0
for _, d := range data {
sum += math.Pow(float64(d)-mean, 2)
}
variance := sum / float64(len(data)-1)
return math.Sqrt(variance)
}
func Mean(data []int) float64 {
sum := 0.0
for _, d := range data {
sum += float64(d)
}
return sum / float64(len(data))
}
To prevent concurrent modifications to a struct while retaining the ability to read, you'd typically embed a sync.RWMutex. This is no exemption. You can simply lock your struct for writes while it is in transit and unlock it at a point in time of your convenience.
package main
import (
"fmt"
"sync"
"time"
)
// Big simulates your big struct
type Big struct {
sync.RWMutex
value string
}
// pump uses a groutine to take the slice of pointers to Big,
// locks the underlying structs and sends the pointers to
// the locked instances of Big downstream
func pump(bigs []*Big) chan *Big {
// We make the channel buffered for this example
// for illustration purposes
c := make(chan *Big, 3)
go func() {
for _, big := range bigs {
// We lock the struct before sending it to the channel
// so it can not be changed via pointer while in transit
big.Lock()
c <- big
}
close(c)
}()
return c
}
// sink reads pointers to the locked instances of Big
// reads them and unlocks them
func sink(c chan *Big) {
for big := range c {
fmt.Println(big.value)
time.Sleep(1 * time.Second)
big.Unlock()
}
}
// modify tries to achieve locks to the instances and modify them
func modify(bigs []*Big) {
for _, big := range bigs {
big.Lock()
big.value = "modified"
big.Unlock()
}
}
func main() {
bigs := []*Big{&Big{value: "Foo"}, &Big{value: "Bar"}, &Big{value: "Baz"}}
c := pump(bigs)
// For the sake of this example, we wait until all entries are
// send into the channel and hence are locked
time.Sleep(1 * time.Second)
// Now we try to modify concurrently before we even start to read
// the struct of which the pointers were sent into the channel
go modify(bigs)
sink(c)
// We use sleep here to keep waiting for modify() to finish simple.
// Usually, you'd use a sync.waitGroup
time.Sleep(1 * time.Second)
for _, big := range bigs {
fmt.Println(big.value)
}
}
Run on playground

Resources