i want to implement add below in pure atomic operation
var num int
func add(max int) int {
if num < max {
num++
}
return num
}
i have try one edition
func add(max int64) int64 {
for {
old := atomic.LoadInt64(&x)
if old + 1 < max {
if atomic.CompareAndSwapInt64(&x, old, old+1) {
return old+1
}
} else {
return old
}
}
}
however, i guess there might be a better solution with less fail chance and avoid dead loop
Here's a revised version of your algorithm with a multiple goroutine test. It passes the test with and without the Go race detector.
add.go:
package main
import (
"fmt"
"runtime"
"sync"
"sync/atomic"
)
var x int64 = -42
func add(max int64) int64 {
for {
old := atomic.LoadInt64(&x)
if old >= max {
return old
}
new := old + 1
if atomic.CompareAndSwapInt64(&x, old, new) {
return new
}
}
}
func main() {
const max = 123456
fmt.Println("max:", max)
fmt.Println("x: ", x)
var wg sync.WaitGroup
procs := runtime.GOMAXPROCS(0)
for i := 0; i < procs; i++ {
wg.Add(1)
go func(max int64) {
defer wg.Done()
for {
if add(max) >= max {
return
}
}
}(max)
}
wg.Wait()
fmt.Println("x: ", x)
}
https://go.dev/play/p/r-qsnyI7tqv
$ go build -race add.go && ./add
max: 123456
x: -42
x: 123456
$ go build add.go && ./add
max: 123456
x: -42
x: 123456
The easiest (but not the most efficient) solution would be to use a mutex:
var (
mu sync.Mutex
num int64
)
func add(max int64) int64 {
mu.Lock()
defer mu.Unlock()
if num < max {
num++
}
return num
}
Related
I'm creating global counter, which can be shared between goroutines.
Referring to this question, following code may satisfy my needs.
However if there ware lots of concurrent requests, could it happen that the same number is assigned to more than two goroutines?
If so how can I avoid this?
This question is different from the link I pasted, as what I want to know about is how I can avoid duplication using channel counter. if the only possible solution is other implementation like sync.Mutex or atomic, I'll use it. however, according to the link (again), channel seems to be the best option. Any comment or answer really helpful. thanks in advance.
I'm new to multithread coding and also go, might be silly question. sorry for that.
package main
import (
"fmt"
"time"
)
var counter int
var counter_chan chan int
func main() {
counter_chan = make(chan int, 100)
counter = 0
go func() {
for {
select {
case chanc := <-counter_chan:
counter += chanc
fmt.Printf("%d \n", counter)
}
}
}()
for i := 0; i < 10; i++ {
go AddCounter(counter_chan)
}
time.Sleep(time.Second)
fmt.Printf("Total Count is ... %d \n", GetCount())
}
func AddCounter(ch chan int) {
ch <- 1
}
func GetCount() int {
return counter
}
func ResetCount() {
if counter > 8190 {
counter = 0
}
}
-- Edit 05/14 2018
Assume following code is thread-safe for getting and resetting value. Am I right?
package main
import (
"fmt"
"time"
)
var counter int
var addCounterChan chan int
var readCounterChan chan int
func main() {
addCounterChan = make(chan int, 100)
readCounterChan = make(chan int, 100)
counter = 0
go func() {
for {
select {
case val := <-addCounterChan:
counter += val
if counter > 5 {
counter = 0
}
readCounterChan <- counter
fmt.Printf("%d \n", counter)
}
}
}()
for i := 0; i < 10; i++ {
go AddCounter(addCounterChan)
}
time.Sleep(time.Second)
for i := 0; i < 10; i++ {
fmt.Printf("Total Count #%d is ... %d \n", (i + 1), GetCount(readCounterChan))
}
}
// Following two functions will be implemented in another package in real case.
func AddCounter(ch chan int) {
ch <- 1
}
func GetCount(ch chan int) int {
r := <-ch
return r
}
The direct answer to your question is: The code you've pasted updates the counter safely, but doesn't read or reset it safely.
Contrary to the accepted answer in the question you linked to, however, the easiest, most efficient way to implement a shared counter is with the atomic package. It can be used to atomically increment several common types. Example:
var globalCounter *int32 = new(int32)
// .. later in your code
currentCount := atomic.AddInt32(globalCounter, 1)
Use a sync.Mutex to create a counter with add, get and reset operations as shown in the question.
type counter struct {
mu sync.Mutex
n int
}
func (c *counter) Add() {
c.mu.Lock()
c.n++
c.mu.Unlock()
}
func (c *counter) Get() int {
c.mu.Lock()
n := c.n
c.mu.Unlock()
return n
}
func (c *counter) Reset() {
c.mu.Lock()
if c.n > 8190 {
c.n = 0
}
c.mu.Unlock()
}
If the reset function is not needed, then use the sync/atomic.
type counter struct {
n int32
}
func (c *counter) Add() {
atomic.AddInt32(&c.n, 1)
}
func (c *counter) Get() int {
return int(atomic.LoadInt32(&c.n))
}
Go 1.19
The sync/atomic package now includes atomic types, such as atomic.Int32, which you can use to manage a value that can only be accessed atomically.
This basically accomplishes the same thing as having a custom struct with a mutex, or using top-level atomic functions to read and write a "naked" numerical type. Instead of rolling your own, you can simply rely on the standard library.
A simple example:
package main
import (
"fmt"
"sync"
"sync/atomic"
)
// zero value is 0
var counter = atomic.Int32{}
func main() {
wg := &sync.WaitGroup{}
wg.Add(100)
for i := 0; i < 100; i++ {
go func() {
counter.Add(1)
wg.Done()
}()
}
wg.Wait()
fmt.Println(counter.Load())
}
Playground: https://go.dev/play/p/76xM3xXTAM5?v=gotip
I'm trying launch goroutines in an infinite loop until I get the response I'm looking for but the select is unreachable if I change for i := 0; i < 10; i++ {} to for {}. What's a pattern for solving this?
package main
import (
"fmt"
"math/rand"
"time"
)
func myFunc() float64 {
c := make(chan float64)
for i := 0; i < 10; i++ {
go func() {
var value float64
value = someOp()
if value > .9 {
c <- value
}
}()
}
// unreachable code the if the for loop above is infinite
for {
select {
case x := <-c:
return x
default:
}
}
}
func someOp() float64 {
rand.Seed(time.Now().UnixNano())
return rand.Float64()
}
func main() {
fmt.Println(myFunc())
}
Starting an unlimited number of goroutines is usually not a good idea. A better approach is to start a fixed number of goroutines that loop looking for an answer. Return from these goroutines when an answer is found.
func myFunc() float64 {
c := make(chan float64, 1) // Size 1 prevents race between main goroutine and workers
done := make(chan struct{})
defer close(done)
// Start a fixed number of goroutines
for i := 0; i < 10; i++ {
go func() {
for {
select {
case <-done:
// myfunc exited with result, return from this goroutine
return
default:
var value float64
value = someOp()
if value > .9 {
select {
case c <- value:
// This is first goroutine to send a value
default:
// Another goroutine sent a value
}
return
}
}
}
}()
}
return <-c
}
https://play.golang.org/p/SRpeT8k34eA
I am trying to implement fibonacci recursion in golang using n goroutines with communicating via channels.
I am returning an integer from the function, but i am actually just sending the sum of f(n-1) +f(n-2) over channel c but this is not working correctly. It prints the first two values correct, and every value after is just 1.
package main
import "fmt"
// Fibonacci in a recursive version
func fiboR(n int, c chan int ) int {
if(n == 0){
c <- 0
return 0
} else if n == 1 {
c <- 1
return 1
} else{
c <- fiboR(n-1,c) + fiboR(n-2,c)
return fiboR(n-1,c) + fiboR(n-2,c)
}
}
func main() {
for i := 0; i < 10; i++ {
procchan := make(chan int)
go fiboR(i,procchan)
fmt.Println(i,<-procchan )
}
}
Also is it possible to use channels for receiving the two recursive calls?
Your solution will try to output more than the one value you extract from the channel as you increase the value of i.
What your code will try to send to the channel for each i:
0: 0
1: 1
2: 1,0,1
3: 1,0,1,1,2
4: 1,0,1,1,2,1,0,1,3
...
Since you create a new channel for each i and then only extract one value you will always get the first value in the line above.
If you try to run it with these modifications it will output what you wanted (https://play.golang.org/p/_mn3l5x8iZ).
package main
import "fmt"
// Fibonacci in a recursive version
func fiboRchan(n int, c chan int) {
c <- fiboR(n)
}
func fiboR(n int) int {
if n == 0 {
return 0
} else if n == 1 {
return 1
} else {
return fiboR(n-1) + fiboR(n-2)
}
}
func main() {
for i := 0; i < 10; i++ {
procchan := make(chan int)
go fiboRchan(i, procchan)
fmt.Println(i, <-procchan)
}
}
Adding to #nissefors answer, the main process there is most likely a sequential one because in the for loop you would be waiting on the channel to return and then proceed to the next iteration.
A minor modification in the main function could fire all the fibonaccis at once and then in a separate for loop the channels that are corresponding to each go routine can be accessed
Playground URL: https://play.golang.org/p/7e3JnWeSp6
package main
import "fmt"
// Fibonacci in a recursive version
func fiboRchan(n int, c chan int) {
fmt.Println("PROCESSING FOR %d", n)
c <- fiboR(n)
}
func fiboR(n int) int {
if n == 0 {
return 0
} else if n == 1 {
return 1
} else {
return fiboR(n-1) + fiboR(n-2)
}
}
func main() {
var arr[10]chan int
for i := 0; i < 10; i++ {
procchan := make(chan int)
arr[i] = procchan
go fiboRchan(i, procchan)
}
// By now all the go routines are fired
// Now iterate through the channel array and read from the
// respective channel
for i:=0; i< 10; i++ {
fmt.Println(i, <-arr[i])
}
}
I have a need to read structure fields set from another goroutine, afaik doing so directly even when knowing for sure there will be no concurrent access(write finished before read occurred, signaled via chan struct{}) may result in stale data
Will sending a pointer to the structure(created in the 1st goroutine, modified in the 2nd, read by the 3rd) resolve the possible staleness issue, considering I can guarantee no concurrent access?
I would like to avoid copying as structure is big and contains huge Bytes.Buffer filled in the 2nd goroutine, I need to read from the 3rd
There is an option for locking, but seems like an overkill considering I know that there will be no concurrent access
There are many answers to this, and it depends to your data structure and program logic.
see: How to lock/synchronize access to a variable in Go during concurrent goroutines?
and: How to use RWMutex in Golang?
1- using Stateful Goroutines and channels
2- using sync.Mutex
3- using sync/atomic
4- using WaitGroup
5- using program logic(Semaphore)
...
1: Stateful Goroutines and channels:
I simulated very similar sample(imagine you want to read from one SSD and write to another SSD with different speed):
In this sample code one goroutine (named write) does some job prepares data and fills the big struct, and another goroutine (named read) reads data from big struct then do some job, And the manger goroutine, guarantee no concurrent access to same data.
And communication between three goroutines done with channels. And in your case you can use pointers for channel data, or global struct like this sample.
output will be like this:
mean= 36.6920166015625 stdev= 6.068973186592054
I hope this helps you to get the idea.
Working sample code:
package main
import (
"fmt"
"math"
"math/rand"
"runtime"
"sync"
"time"
)
type BigStruct struct {
big []uint16
rpos int
wpos int
full bool
empty bool
stopped bool
}
func main() {
wg.Add(1)
go write()
go read()
go manage()
runtime.Gosched()
stopCh <- <-time.After(5 * time.Second)
wg.Wait()
mean := Mean(hist)
stdev := stdDev(hist, mean)
fmt.Println("mean=", mean, "stdev=", stdev)
}
const N = 1024 * 1024 * 1024
var wg sync.WaitGroup
var stopCh chan time.Time = make(chan time.Time)
var hist []int = make([]int, 65536)
var s *BigStruct = &BigStruct{empty: true,
big: make([]uint16, N), //2GB
}
var rc chan uint16 = make(chan uint16)
var wc chan uint16 = make(chan uint16)
func next(pos int) int {
pos++
if pos >= N {
pos = 0
}
return pos
}
func manage() {
dataReady := false
var data uint16
for {
if !dataReady && !s.empty {
dataReady = true
data = s.big[s.rpos]
s.rpos++
if s.rpos >= N {
s.rpos = 0
}
s.empty = s.rpos == s.wpos
s.full = next(s.wpos) == s.rpos
}
if dataReady {
select {
case rc <- data:
dataReady = false
default:
runtime.Gosched()
}
}
if !s.full {
select {
case d := <-wc:
s.big[s.wpos] = d
s.wpos++
if s.wpos >= N {
s.wpos = 0
}
s.empty = s.rpos == s.wpos
s.full = next(s.wpos) == s.rpos
default:
runtime.Gosched()
}
}
if s.stopped {
if s.empty {
wg.Done()
return
}
}
}
}
func read() {
for {
d := <-rc
hist[d]++
}
}
func write() {
for {
wc <- uint16(rand.Intn(65536))
select {
case <-stopCh:
s.stopped = true
return
default:
runtime.Gosched()
}
}
}
func stdDev(data []int, mean float64) float64 {
sum := 0.0
for _, d := range data {
sum += math.Pow(float64(d)-mean, 2)
}
variance := sum / float64(len(data)-1)
return math.Sqrt(variance)
}
func Mean(data []int) float64 {
sum := 0.0
for _, d := range data {
sum += float64(d)
}
return sum / float64(len(data))
}
5: another way(faster) for some use cases:
here another way to use shared data structure for read job/write job/ processing job which it was separated in first post, now here doing same 3 jobs without channels and without mutex.
working sample:
package main
import (
"fmt"
"math"
"math/rand"
"time"
)
type BigStruct struct {
big []uint16
rpos int
wpos int
full bool
empty bool
stopped bool
}
func manage() {
for {
if !s.empty {
hist[s.big[s.rpos]]++ //sample read job with any time len
nextPtr(&s.rpos)
}
if !s.full && !s.stopped {
s.big[s.wpos] = uint16(rand.Intn(65536)) //sample wrire job with any time len
nextPtr(&s.wpos)
}
if s.stopped {
if s.empty {
return
}
} else {
s.stopped = time.Since(t0) >= 5*time.Second
}
}
}
func main() {
t0 = time.Now()
manage()
mean := Mean(hist)
stdev := StdDev(hist, mean)
fmt.Println("mean=", mean, "stdev=", stdev)
d0 := time.Since(t0)
fmt.Println(d0) //5.8523347s
}
var t0 time.Time
const N = 100 * 1024 * 1024
var hist []int = make([]int, 65536)
var s *BigStruct = &BigStruct{empty: true,
big: make([]uint16, N), //2GB
}
func next(pos int) int {
pos++
if pos >= N {
pos = 0
}
return pos
}
func nextPtr(pos *int) {
*pos++
if *pos >= N {
*pos = 0
}
s.empty = s.rpos == s.wpos
s.full = next(s.wpos) == s.rpos
}
func StdDev(data []int, mean float64) float64 {
sum := 0.0
for _, d := range data {
sum += math.Pow(float64(d)-mean, 2)
}
variance := sum / float64(len(data)-1)
return math.Sqrt(variance)
}
func Mean(data []int) float64 {
sum := 0.0
for _, d := range data {
sum += float64(d)
}
return sum / float64(len(data))
}
To prevent concurrent modifications to a struct while retaining the ability to read, you'd typically embed a sync.RWMutex. This is no exemption. You can simply lock your struct for writes while it is in transit and unlock it at a point in time of your convenience.
package main
import (
"fmt"
"sync"
"time"
)
// Big simulates your big struct
type Big struct {
sync.RWMutex
value string
}
// pump uses a groutine to take the slice of pointers to Big,
// locks the underlying structs and sends the pointers to
// the locked instances of Big downstream
func pump(bigs []*Big) chan *Big {
// We make the channel buffered for this example
// for illustration purposes
c := make(chan *Big, 3)
go func() {
for _, big := range bigs {
// We lock the struct before sending it to the channel
// so it can not be changed via pointer while in transit
big.Lock()
c <- big
}
close(c)
}()
return c
}
// sink reads pointers to the locked instances of Big
// reads them and unlocks them
func sink(c chan *Big) {
for big := range c {
fmt.Println(big.value)
time.Sleep(1 * time.Second)
big.Unlock()
}
}
// modify tries to achieve locks to the instances and modify them
func modify(bigs []*Big) {
for _, big := range bigs {
big.Lock()
big.value = "modified"
big.Unlock()
}
}
func main() {
bigs := []*Big{&Big{value: "Foo"}, &Big{value: "Bar"}, &Big{value: "Baz"}}
c := pump(bigs)
// For the sake of this example, we wait until all entries are
// send into the channel and hence are locked
time.Sleep(1 * time.Second)
// Now we try to modify concurrently before we even start to read
// the struct of which the pointers were sent into the channel
go modify(bigs)
sink(c)
// We use sleep here to keep waiting for modify() to finish simple.
// Usually, you'd use a sync.waitGroup
time.Sleep(1 * time.Second)
for _, big := range bigs {
fmt.Println(big.value)
}
}
Run on playground
How to realize a nested iterator that takes a depth argument. A simple iterator would be when depth = 1. it is a simple iterator which runs like a simple for loop.
func Iter () chan int {
ch := make(chan int);
go func () {
for i := 1; i < 60; i++ {
ch <- i
}
close(ch)
} ();
return ch
}
Output is 1,2,3...59
For depth = 2 Output would be "1,1" "1,2" ... "1,59" "2,1" ... "59,59"
For depth = 3 Output would be "1,1,1" ... "59,59,59"
I want to avoid a nested for loop. What is the solution here ?
I don't know if it is possible to avoid nested loops, but one solution is to use a pipeline of channels. For example:
const ITER_N = 60
// ----------------
func _goFunc1(out chan string) {
for i := 1; i < ITER_N; i++ {
out <- fmt.Sprintf("%d", i)
}
close(out)
}
func _goFuncN(in chan string, out chan string) {
for j := range in {
for i := 1; i < ITER_N; i++ {
out <- fmt.Sprintf("%s,%d", j, i)
}
}
close(out)
}
// ----------------
// create the pipeline
func IterDepth(d int) chan string {
c1 := make(chan string)
go _goFunc1(c1)
var c2 chan string
for ; d > 1; d-- {
c2 = make(chan string)
go _goFuncN(c1, c2)
c1 = c2
}
return c1
}
You can test it with:
func main() {
c := IterDepth(2)
for i := range c {
fmt.Println(i)
}
}
I usually implement iterators using closures. Multiple dimensions don't make the problem much harder. Here's one example of how to do this:
package main
import "fmt"
func iter(min, max, depth int) func() ([]int, bool) {
s := make([]int, depth)
for i := range s {
s[i] = min
}
s[0] = min - 1
return func() ([]int, bool) {
s[0]++
for i := 0; i < depth-1; i++ {
if s[i] >= max {
s[i] = min
s[i+1]++
}
}
if s[depth-1] >= max {
return nil, false
}
return s, true
}
}
func main() {
// Three dimensions, ranging between [1,4)
i := iter(1, 4, 3)
for s, ok := i(); ok; s, ok = i() {
fmt.Println(s)
}
}
Try it out on the Playground.
It'd be a simple change for example to give arguments as a single int slice instead, so that you could have per-dimension limits, if such a thing were necessary.