How to create global counter in highly concurrent system - go

I'm creating global counter, which can be shared between goroutines.
Referring to this question, following code may satisfy my needs.
However if there ware lots of concurrent requests, could it happen that the same number is assigned to more than two goroutines?
If so how can I avoid this?
This question is different from the link I pasted, as what I want to know about is how I can avoid duplication using channel counter. if the only possible solution is other implementation like sync.Mutex or atomic, I'll use it. however, according to the link (again), channel seems to be the best option. Any comment or answer really helpful. thanks in advance.
I'm new to multithread coding and also go, might be silly question. sorry for that.
package main
import (
"fmt"
"time"
)
var counter int
var counter_chan chan int
func main() {
counter_chan = make(chan int, 100)
counter = 0
go func() {
for {
select {
case chanc := <-counter_chan:
counter += chanc
fmt.Printf("%d \n", counter)
}
}
}()
for i := 0; i < 10; i++ {
go AddCounter(counter_chan)
}
time.Sleep(time.Second)
fmt.Printf("Total Count is ... %d \n", GetCount())
}
func AddCounter(ch chan int) {
ch <- 1
}
func GetCount() int {
return counter
}
func ResetCount() {
if counter > 8190 {
counter = 0
}
}
-- Edit 05/14 2018
Assume following code is thread-safe for getting and resetting value. Am I right?
package main
import (
"fmt"
"time"
)
var counter int
var addCounterChan chan int
var readCounterChan chan int
func main() {
addCounterChan = make(chan int, 100)
readCounterChan = make(chan int, 100)
counter = 0
go func() {
for {
select {
case val := <-addCounterChan:
counter += val
if counter > 5 {
counter = 0
}
readCounterChan <- counter
fmt.Printf("%d \n", counter)
}
}
}()
for i := 0; i < 10; i++ {
go AddCounter(addCounterChan)
}
time.Sleep(time.Second)
for i := 0; i < 10; i++ {
fmt.Printf("Total Count #%d is ... %d \n", (i + 1), GetCount(readCounterChan))
}
}
// Following two functions will be implemented in another package in real case.
func AddCounter(ch chan int) {
ch <- 1
}
func GetCount(ch chan int) int {
r := <-ch
return r
}

The direct answer to your question is: The code you've pasted updates the counter safely, but doesn't read or reset it safely.
Contrary to the accepted answer in the question you linked to, however, the easiest, most efficient way to implement a shared counter is with the atomic package. It can be used to atomically increment several common types. Example:
var globalCounter *int32 = new(int32)
// .. later in your code
currentCount := atomic.AddInt32(globalCounter, 1)

Use a sync.Mutex to create a counter with add, get and reset operations as shown in the question.
type counter struct {
mu sync.Mutex
n int
}
func (c *counter) Add() {
c.mu.Lock()
c.n++
c.mu.Unlock()
}
func (c *counter) Get() int {
c.mu.Lock()
n := c.n
c.mu.Unlock()
return n
}
func (c *counter) Reset() {
c.mu.Lock()
if c.n > 8190 {
c.n = 0
}
c.mu.Unlock()
}
If the reset function is not needed, then use the sync/atomic.
type counter struct {
n int32
}
func (c *counter) Add() {
atomic.AddInt32(&c.n, 1)
}
func (c *counter) Get() int {
return int(atomic.LoadInt32(&c.n))
}

Go 1.19
The sync/atomic package now includes atomic types, such as atomic.Int32, which you can use to manage a value that can only be accessed atomically.
This basically accomplishes the same thing as having a custom struct with a mutex, or using top-level atomic functions to read and write a "naked" numerical type. Instead of rolling your own, you can simply rely on the standard library.
A simple example:
package main
import (
"fmt"
"sync"
"sync/atomic"
)
// zero value is 0
var counter = atomic.Int32{}
func main() {
wg := &sync.WaitGroup{}
wg.Add(100)
for i := 0; i < 100; i++ {
go func() {
counter.Add(1)
wg.Done()
}()
}
wg.Wait()
fmt.Println(counter.Load())
}
Playground: https://go.dev/play/p/76xM3xXTAM5?v=gotip

Related

Swap Concurrent Function

I am trying to figure out how to make concurrent the forward Swap function in Go for learning concepts purpose:
package main
import "fmt"
func Swap(a, b int) (int, int) {
return b, a
}
func main() {
for i := 0; i <10; i++ {
fmt.Println(Swap(i+1, i))
}
}
So far I have came up with this solution:
package main
import "fmt"
// Only Send Channel
func Swap(a, b chan<- int) {
go func() {
for i := 0; i < 10; i++ {
a <- i + 1
b <- i
}
}()
}
func main() {
a := make(chan int)
b := make(chan int)
Swap(a, b)
for i := 0; i < 10; i++ {
fmt.Printf("chan a received: %d | chan b received: %d\n", <-a, <-b)
}
}
It seems to work but I cannot be sure about it.
How can I measure this and does anyone knows how to really make a simple Swap function concurrent in Go?
I am sorry for the question but I figured out how to solve this.
It took days for me to be able to do this and on the day I posted at this forum some hours later I discovered how to solve it.
I am going to post the code and the walkthrough this was given to me in a Senior Golang Job Interview and I could not answer, I hope this can help someone.
// Pass as a parameter a int slice of type channel
// Send the swapped information throw the channel
func Swap(a, b int, ch chan []int) {
ch <- []int{b, a}
}
func main() {
ch := make(chan []int)
for i := 0; i < 100; i++ {
// Call Worker with the created channel
go Swap(i+1, i, ch)
}
for i := 0; i < 100; i++ {
// Receive Channel Value
fmt.Println(<-ch)
}
}
I really appreciate any comments, improvements and conceptual references on this.

Getting deadlock as I try to emulate fan in - fan out with factorial calculations

I am trying the fan in - fan out pattern with a factorial problem. But I am getting:
fatal error: all goroutines are asleep - deadlock!
and unable to identify the reason for deadlock.
I am trying to concurrently calculate factorial for 100 numbers using the fan-in fan-out pattern.
package main
import (
"fmt"
)
func main() {
_inChannel := _inListener(generator())
for val := range _inChannel {
fmt.Print(val, " -- ")
}
}
func generator() chan int { // NEED TO CALCULATE FACTORIAL FOR 100 NUMBERS
ch := make(chan int) // CREATE CHANNEL TO INPUT NUMBERS
go func() {
for i := 1; i <= 100; i++ {
ch <- i
}
close(ch) // CLOSE CHANNEL WHEN ALL NUMBERS HAVE BEEN WRITTEM
}()
return ch
}
func _inListener(ch chan int) chan int {
rec := make(chan int) // CHANNEL RECEIVED FROM GENERATOR
go func() {
for num := range ch { // RECEIVE THE INPUT NUMBERS FROM GENERATOR
result := factorial(num) // RESULT IS A NEW CHANNEL CREATED
rec <- <-result // MERGE INTO A SINGLE CHANNEL; rec
close(result)
}
close(rec)
}()
return rec // RETURN THE DEDICATED CHANNEL TO RECEIVE ALL OUTPUTS
}
func factorial(n int) chan int {
ch := make(chan int) // MAKE A NEW CHANNEL TO OUTPUT THE RESULT
// OF FACTORIAL
total := 1
for i := n; i > 0; i-- {
total *= i
}
ch <- total
return ch // RETURN THE CHANNEL HAVING THE FACTORIAL CALCULATED
}
I have put in comments, so that it becomes easier to follow the code.
I'm no expert in channels. I've taking on this to try and get more familiar with go.
Another issue is the int isn't large enough to take all factorials over 20 or so.
As you can see, I added a defer close as well as a logical channel called done in the generator func. The rest of the changes probably aren't needed. With channels you need to make sure something is ready to take off a value on the channel when you put something on a channel. Otherwise deadlock. Also, using
go run -race main.go
helps at least see which line(s) are causing problems.
I hope this helps and isn't removed for being off topic.
I was able to remove the deadlock by doing this:
package main
import (
"fmt"
)
func main() {
_gen := generator()
_inChannel := _inListener(_gen)
for val := range _inChannel {
fmt.Print(val, " -- \n")
}
}
func generator() chan int { // NEED TO CALCULATE FACTORIAL FOR 100 NUMBERS
ch := make(chan int) // CREATE CHANNEL TO INPUT NUMBERS
done := make(chan bool)
go func() {
defer close(ch)
for i := 1; i <= 100; i++ {
ch <- i
}
//close(ch) // CLOSE CHANNEL WHEN ALL NUMBERS HAVE BEEN WRITTEM
done <- true
}()
// this function will pull off the done for each function call above.
go func() {
for i := 1; i < 100; i++ {
<-done
}
}()
return ch
}
func _inListener(ch chan int) chan int {
rec := make(chan int) // CHANNEL RECEIVED FROM GENERATOR
go func() {
for num := range ch { // RECEIVE THE INPUT NUMBERS FROM GENERATOR
result := factorial(num) // RESULT IS A NEW CHANNEL CREATED
rec <- result // MERGE INTO A SINGLE CHANNEL; rec
}
close(rec)
}()
return rec // RETURN THE DEDICATED CHANNEL TO RECEIVE ALL OUTPUTS
}
func factorial(n int) int {
// OF FACTORIAL
total := 1
for i := n; i > 0; i-- {
total *= i
}
return total // RETURN THE CHANNEL HAVING THE FACTORIAL CALCULATED
}

Why does this golang program create a memory leak?

I am trying to understand concurrency and goroutines, and had a couple questions about the following experimental code:
Why does it create a memory leak? I thought that a return at the end of the goroutine would allow memory associated with it to get cleaned up.
Why do my loops almost never reach 999? In fact, when I output to a file and study the output, I notice that it rarely prints integers in double digits; the first time it prints "99" is line 2461, and for "999" line 6120. This behavior is unexpected to me, which clearly means I don't really understand what is going on with goroutine scheduling.
Disclaimer:
Be careful running the code below, it can crash your system if you don't stop it after a few seconds!
CODE
package main
import (
"fmt"
"sync"
)
func main() {
var wg sync.WaitGroup
for {
// spawn four worker goroutines
spawnWorkers(4, wg)
// wait for the workers to finish
wg.Wait()
}
}
func spawnWorkers(max int, wg sync.WaitGroup) {
for n := 0; n < max; n++ {
wg.Add(1)
go func() {
defer wg.Done()
f(n)
return
}()
}
}
func f(n int) {
for i := 0; i < 1000; i++ {
fmt.Println(n, ":", i)
}
}
Thanks to Tim Cooper, JimB, and Greg for their helpful comments. The corrected version of the code is posted below for reference.
The two fixes were to pass in the WaitGroup by reference, which fixed the memory leak, and to pass n correctly into the anonymous goroutine, and
package main
import (
"fmt"
"sync"
)
func main() {
var wg sync.WaitGroup
for {
// spawn four worker goroutines
spawnWorkers(4,&wg)
// wait for the workers to finish
wg.Wait()
}
}
func spawnWorkers(max int, wg *sync.WaitGroup) {
for n := 0; n < max; n++ {
wg.Add(1)
go func(n int) {
defer wg.Done()
f(n)
return
}(n)
}
}
func f(n int) {
for i := 0; i < 1000; i++ {
fmt.Println(n, ":", i)
}
}

Golang share big chunk of data between goroutines

I have a need to read structure fields set from another goroutine, afaik doing so directly even when knowing for sure there will be no concurrent access(write finished before read occurred, signaled via chan struct{}) may result in stale data
Will sending a pointer to the structure(created in the 1st goroutine, modified in the 2nd, read by the 3rd) resolve the possible staleness issue, considering I can guarantee no concurrent access?
I would like to avoid copying as structure is big and contains huge Bytes.Buffer filled in the 2nd goroutine, I need to read from the 3rd
There is an option for locking, but seems like an overkill considering I know that there will be no concurrent access
There are many answers to this, and it depends to your data structure and program logic.
see: How to lock/synchronize access to a variable in Go during concurrent goroutines?
and: How to use RWMutex in Golang?
1- using Stateful Goroutines and channels
2- using sync.Mutex
3- using sync/atomic
4- using WaitGroup
5- using program logic(Semaphore)
...
1: Stateful Goroutines and channels:
I simulated very similar sample(imagine you want to read from one SSD and write to another SSD with different speed):
In this sample code one goroutine (named write) does some job prepares data and fills the big struct, and another goroutine (named read) reads data from big struct then do some job, And the manger goroutine, guarantee no concurrent access to same data.
And communication between three goroutines done with channels. And in your case you can use pointers for channel data, or global struct like this sample.
output will be like this:
mean= 36.6920166015625 stdev= 6.068973186592054
I hope this helps you to get the idea.
Working sample code:
package main
import (
"fmt"
"math"
"math/rand"
"runtime"
"sync"
"time"
)
type BigStruct struct {
big []uint16
rpos int
wpos int
full bool
empty bool
stopped bool
}
func main() {
wg.Add(1)
go write()
go read()
go manage()
runtime.Gosched()
stopCh <- <-time.After(5 * time.Second)
wg.Wait()
mean := Mean(hist)
stdev := stdDev(hist, mean)
fmt.Println("mean=", mean, "stdev=", stdev)
}
const N = 1024 * 1024 * 1024
var wg sync.WaitGroup
var stopCh chan time.Time = make(chan time.Time)
var hist []int = make([]int, 65536)
var s *BigStruct = &BigStruct{empty: true,
big: make([]uint16, N), //2GB
}
var rc chan uint16 = make(chan uint16)
var wc chan uint16 = make(chan uint16)
func next(pos int) int {
pos++
if pos >= N {
pos = 0
}
return pos
}
func manage() {
dataReady := false
var data uint16
for {
if !dataReady && !s.empty {
dataReady = true
data = s.big[s.rpos]
s.rpos++
if s.rpos >= N {
s.rpos = 0
}
s.empty = s.rpos == s.wpos
s.full = next(s.wpos) == s.rpos
}
if dataReady {
select {
case rc <- data:
dataReady = false
default:
runtime.Gosched()
}
}
if !s.full {
select {
case d := <-wc:
s.big[s.wpos] = d
s.wpos++
if s.wpos >= N {
s.wpos = 0
}
s.empty = s.rpos == s.wpos
s.full = next(s.wpos) == s.rpos
default:
runtime.Gosched()
}
}
if s.stopped {
if s.empty {
wg.Done()
return
}
}
}
}
func read() {
for {
d := <-rc
hist[d]++
}
}
func write() {
for {
wc <- uint16(rand.Intn(65536))
select {
case <-stopCh:
s.stopped = true
return
default:
runtime.Gosched()
}
}
}
func stdDev(data []int, mean float64) float64 {
sum := 0.0
for _, d := range data {
sum += math.Pow(float64(d)-mean, 2)
}
variance := sum / float64(len(data)-1)
return math.Sqrt(variance)
}
func Mean(data []int) float64 {
sum := 0.0
for _, d := range data {
sum += float64(d)
}
return sum / float64(len(data))
}
5: another way(faster) for some use cases:
here another way to use shared data structure for read job/write job/ processing job which it was separated in first post, now here doing same 3 jobs without channels and without mutex.
working sample:
package main
import (
"fmt"
"math"
"math/rand"
"time"
)
type BigStruct struct {
big []uint16
rpos int
wpos int
full bool
empty bool
stopped bool
}
func manage() {
for {
if !s.empty {
hist[s.big[s.rpos]]++ //sample read job with any time len
nextPtr(&s.rpos)
}
if !s.full && !s.stopped {
s.big[s.wpos] = uint16(rand.Intn(65536)) //sample wrire job with any time len
nextPtr(&s.wpos)
}
if s.stopped {
if s.empty {
return
}
} else {
s.stopped = time.Since(t0) >= 5*time.Second
}
}
}
func main() {
t0 = time.Now()
manage()
mean := Mean(hist)
stdev := StdDev(hist, mean)
fmt.Println("mean=", mean, "stdev=", stdev)
d0 := time.Since(t0)
fmt.Println(d0) //5.8523347s
}
var t0 time.Time
const N = 100 * 1024 * 1024
var hist []int = make([]int, 65536)
var s *BigStruct = &BigStruct{empty: true,
big: make([]uint16, N), //2GB
}
func next(pos int) int {
pos++
if pos >= N {
pos = 0
}
return pos
}
func nextPtr(pos *int) {
*pos++
if *pos >= N {
*pos = 0
}
s.empty = s.rpos == s.wpos
s.full = next(s.wpos) == s.rpos
}
func StdDev(data []int, mean float64) float64 {
sum := 0.0
for _, d := range data {
sum += math.Pow(float64(d)-mean, 2)
}
variance := sum / float64(len(data)-1)
return math.Sqrt(variance)
}
func Mean(data []int) float64 {
sum := 0.0
for _, d := range data {
sum += float64(d)
}
return sum / float64(len(data))
}
To prevent concurrent modifications to a struct while retaining the ability to read, you'd typically embed a sync.RWMutex. This is no exemption. You can simply lock your struct for writes while it is in transit and unlock it at a point in time of your convenience.
package main
import (
"fmt"
"sync"
"time"
)
// Big simulates your big struct
type Big struct {
sync.RWMutex
value string
}
// pump uses a groutine to take the slice of pointers to Big,
// locks the underlying structs and sends the pointers to
// the locked instances of Big downstream
func pump(bigs []*Big) chan *Big {
// We make the channel buffered for this example
// for illustration purposes
c := make(chan *Big, 3)
go func() {
for _, big := range bigs {
// We lock the struct before sending it to the channel
// so it can not be changed via pointer while in transit
big.Lock()
c <- big
}
close(c)
}()
return c
}
// sink reads pointers to the locked instances of Big
// reads them and unlocks them
func sink(c chan *Big) {
for big := range c {
fmt.Println(big.value)
time.Sleep(1 * time.Second)
big.Unlock()
}
}
// modify tries to achieve locks to the instances and modify them
func modify(bigs []*Big) {
for _, big := range bigs {
big.Lock()
big.value = "modified"
big.Unlock()
}
}
func main() {
bigs := []*Big{&Big{value: "Foo"}, &Big{value: "Bar"}, &Big{value: "Baz"}}
c := pump(bigs)
// For the sake of this example, we wait until all entries are
// send into the channel and hence are locked
time.Sleep(1 * time.Second)
// Now we try to modify concurrently before we even start to read
// the struct of which the pointers were sent into the channel
go modify(bigs)
sink(c)
// We use sleep here to keep waiting for modify() to finish simple.
// Usually, you'd use a sync.waitGroup
time.Sleep(1 * time.Second)
for _, big := range bigs {
fmt.Println(big.value)
}
}
Run on playground

How to use <-chan and chan<- for one directional communication?

I'm working on understanding Go's channels. I think I understand a basic bidirectional chan but I'm falling short at understanding <-chan and chan<-.
I expected them to be useful for communicating one way to a thread but I'm having issues with the thread actually reading and receiving the value.
package main
import (
"fmt"
"time"
)
func Thread(c chan<- int) {
for {
num := <-c
fmt.Println("Thread : ", num)
time.Sleep(time.Second)
}
}
func main() {
c := make(chan<- int, 3)
go Thread(c)
for i := 1; i <= 10; i++ {
c <- i
}
for len(c) > 0 {
time.Sleep(100)
}
}
I've tried using <-chan instead of chan<- in the make() but the same kind of thing happens:
C:\>go run chan.go
# command-line-arguments
.\chan.go:10: invalid operation: <-c (receive from send-only type chan<- int)
If I can't read from the channel, why bother writing to it? With that thought in mind, I figure I must be doing something wrong. I had the expectation that a send only chan would mean that one thread can only send while the other thread can only receive. This does not seem to be the case.
If I remove the <- entirely, it works, but that would make it bidirectional allowing the go routine to respond (even though it never does) and I'm looking to avoid that. It seems like I can banish numbers to a chan that I'll never be able to read from, or that I could read from a chan that's impossible to write to.
What I'm hoping to do is send integers from the main thread to the go routine for it to print using a one way channel. What am I doing wrong?
This is with go 1.3.3 on Windows if it matters. Updating to 1.4 didn't help. I might want to mention this is all x64 as well.
The Go Programming Language Specification
Channel types
A channel provides a mechanism for concurrently executing functions to
communicate by sending and receiving values of a specified element
type. The value of an uninitialized channel is nil.
ChannelType = ( "chan" | "chan" "<-" | "<-" "chan" ) ElementType .
The optional <- operator specifies the channel direction, send or
receive. If no direction is given, the channel is bidirectional. A
channel may be constrained only to send or only to receive by
conversion or assignment.
You can be explicit about channel direction by conversion or assignment. For example, by conversion,
package main
import (
"fmt"
"time"
)
func Thread(r <-chan int) {
for {
num := <-r
fmt.Println("Thread : ", num)
time.Sleep(time.Second)
}
}
func main() {
c := make(chan int, 3)
s, r := (chan<- int)(c), (<-chan int)(c)
go Thread(r)
for i := 1; i <= 10; i++ {
s <- i
}
for len(c) > 0 {
time.Sleep(100)
}
}
Output:
Thread : 1
Thread : 2
Thread : 3
. . .
Or, equivalently, by assignment,
package main
import (
"fmt"
"time"
)
func Thread(r <-chan int) {
for {
num := <-r
fmt.Println("Thread : ", num)
time.Sleep(time.Second)
}
}
func main() {
c := make(chan int, 3)
var s chan<- int = c
var r <-chan int = c
go Thread(r)
for i := 1; i <= 10; i++ {
s <- i
}
for len(c) > 0 {
time.Sleep(100)
}
}
Go allows to create send- or receive-only channels like c := make(<-chan int), however, I haven't came across a use case. There is a discussion here in github.
About the error in the code;
func Thread(c chan<- int) {
for {
num := <-c
fmt.Println("Thread : ", num)
time.Sleep(time.Second)
}
}
Function parameter c can only be sent to. Attempting to receive from c will result in a compiler error but the line num := <-c tries to receive.
At the end, fixed version;
package main
import (
"fmt"
"time"
)
func Thread(c <-chan int) {
for {
num := <-c
fmt.Println("Thread : ", num)
time.Sleep(time.Second)
}
}
func main() {
c := make(chan int, 3)
go Thread(c)
for i := 1; i <= 10; i++ {
c <- i
}
for len(c) > 0 {
time.Sleep(100)
}
}
Your Thread function is passing a send-only channel, instead of a receive-only channel. But you don't need to make() it receive-only, you can pass a bidirectional channel to a goroutine as receive-only.
package main
import (
"fmt"
"time"
)
func Thread(c <-chan int) {
for {
num := <-c
fmt.Println("Thread : ", num)
time.Sleep(time.Second)
}
}
func main() {
c := make(chan int, 3)
go Thread(c)
for i := 1; i <= 10; i++ {
c <- i
}
for len(c) > 0 {
time.Sleep(100)
}
}
You can verify this by adding the line
c <- 3
in your Thread function, throwing the error prog.go:11: invalid operation: c <- 3 (send to receive-only type <-chan int)
See https://play.golang.org/p/5KNPNLqA_k

Resources