Monitor fullness of any channel in Go - go

There are several channels to monitor, their type are different and irrelevant(since we only care about len and cap), but golang compiler does not accept following code, whatever T is:
func monitorChan(ch chan T) {
for {
if len(ch) == cap(ch) {
log.Warn("log")
}
time.Sleep(chanMonitorInterval)
}
}
it shows error:
cannot use ch (type chan []byte) as type chan interface {} in argument
to monitorChan.
How can this function be modified to write once monitor every channel?
Here is my code:
package main
import (
"fmt"
"time"
)
func monitorChan(ch chan interface{}) {
for {
if len(ch) == cap(ch) {
fmt.Println("log")
}
time.Sleep(1 * time.Second)
}
}
func main() {
ch := make(chan []byte, 100)
go monitorChan(ch)
// actual things below ...
}
Playground: https://play.golang.org/p/t7T28IpLNAs

Use reflection. For example,
package main
import (
"log"
"reflect"
"time"
)
func monitorChan(ch interface{}, intvl time.Duration) {
v := reflect.ValueOf(ch)
if v.Kind() != reflect.Chan {
return
}
c := v.Cap()
if c == 0 {
return
}
for {
if l := v.Len(); l == c {
log.Printf("log: len(%d) cap(%d)", l, c)
}
time.Sleep(intvl)
}
}
func main() {
log.Print("main")
c := make(chan []byte, 10)
var chanMonitorInterval = 1 * time.Second
go monitorChan(c, chanMonitorInterval)
log.Print("monitor")
time.Sleep(5 * chanMonitorInterval)
for len(c) != cap(c) {
c <- []byte{}
}
log.Print("len(c) == cap(c)")
time.Sleep(3 * chanMonitorInterval)
<-c
log.Print("len(c) < cap(c)")
time.Sleep(5 * chanMonitorInterval)
log.Print("main")
}
Playground: https://play.golang.org/p/c5VhIIO0pik
Output:
2009/11/10 23:00:00 main
2009/11/10 23:00:00 monitor
2009/11/10 23:00:05 len(c) == cap(c)
2009/11/10 23:00:06 log: len(10) cap(10)
2009/11/10 23:00:07 log: len(10) cap(10)
2009/11/10 23:00:08 log: len(10) cap(10)
2009/11/10 23:00:08 len(c) < cap(c)
2009/11/10 23:00:13 main
References:
Package reflect
The Go Blog: The Laws of Reflection

Go 1.18
This is now trivial to accomplish, using type parameters:
func monitorChan[T any](ch chan T) {
for {
if len(ch) == cap(ch) {
log.Warn("log")
}
time.Sleep(chanMonitorInterval)
}
}
Type inference also allows to infer the type parameter T from the concrete type of the function argument, so the code in main doesn't even need to be rewritten:
func main() {
ch := make(chan []byte, 100)
go monitorChan(ch) // T instantiated as []byte
// actual things below ...
}

Create an interface{} type channel and pass any type wrapping around interface{}, then fetch the use type assert on receiving end.
package main
import (
"fmt"
"sync"
)
var wg sync.WaitGroup
func monitorChan(ch chan interface{}) {
val := <-ch
fmt.Println(string(val.(interface{}).([]uint8)))
wg.Done()
}
func main() {
ch := make(chan interface{}, 100)
wg.Add(1)
ch <- []byte("hello")
go monitorChan(ch)
wg.Wait()
// actual things below ...
}
Working code on Go Playground
Edited :- you can also go for reflect package to get the values of channels after wrapping the channels inside interface{}
package main
import (
"fmt"
"sync"
"reflect"
)
var wg sync.WaitGroup
func monitorChan(i interface{}) {
defer wg.Done()
v := reflect.ValueOf(i)
fmt.Printf("%s size: %d/%d\n", v.Kind(), v.Len(), v.Cap())
}
func main() {
ch := make(chan []byte, 100)
wg.Add(1)
go monitorChan(ch)
wg.Wait()
// actual things below ...
}
Playground example

Related

Sync WaitGroup not waiting for goroutine to assign pointer

Say I have the following code (playground):
package main
import (
"fmt"
"sync"
)
func createStr() *string {
tmp := "foo"
return &tmp
}
func main() {
var (
s *string
wg sync.WaitGroup
)
go func() {
wg.Add(1)
defer wg.Done()
s = createStr()
}()
wg.Wait()
fmt.Printf("s after: %v", s)
}
I would have expected s to not equal nil.
However, if I add a small wait, I get s != nil (playground):
package main
import (
"fmt"
"sync"
"time"
)
func createStr() *string {
tmp := "foo"
return &tmp
}
func main() {
var (
s *string
wg sync.WaitGroup
)
go func() {
wg.Add(1)
defer wg.Done()
s = createStr()
}()
wg.Wait()
time.Sleep(time.Second)
fmt.Printf("s after: %v", s)
}
This just caused a bug in a program I wrote. What is happening with sync.WaitGroup that's not causing my program to wait for s to be assigned a string pointer in my go func?
Your placement of Add method for sync.WaitGroup is wrong. Do not use Add inside a goroutine (anonymous goroutine here) but use it in the goroutine (main goroutine here) that's going to wait for it.
A probable situation that was happening in your code was that wg.Wait() didn't wait as the goroutine's wg.Add(1) wasn't called yet and hence s == nil. The following code fixes the issue:
Go Playground
package main
import (
"fmt"
"sync"
)
func createStr() *string {
tmp := "foo"
return &tmp
}
func main() {
var (
s *string
wg sync.WaitGroup
)
// Use wg.Add() here
wg.Add(1)
go func() {
defer wg.Done()
s = createStr()
}()
wg.Wait()
fmt.Printf("s after: %v", *s)
}

Deadlock in channels communication between two packages - Golang

Deadlock in channels communication between two packages - Golang.
I have two packages that are communicated by two channels. One is main and the other has a function. When I run it, I obtain a deadlock.
package main
import (
functionspackage "GoEjemplos/subFolder"
"fmt"
"sync"
)
func main() {
var wg sync.WaitGroup
ChannelSendData := make(chan functionspackage.FunctionStruct, 1)
defer close(ChannelSendData)
data := functionspackage.FunctionStruct{
FieldOne: 3.56,
FieldTwo: 23,
}
ChannelSendData <- data
wg.Add(1)
go functionspackage.FunctionExt(ChannelSendData, &wg)
recibe := <-functionspackage.ChannelOutFunct
fmt.Println("channelOut: ", recibe)
close(functionspackage.ChannelOutFunct)
wg.Wait()
}
The other package is
package functionspackage
import "sync"
type FunctionStruct struct {
FieldOne float64
FieldTwo int
}
var ChannelOutFunct chan float64
func FunctionExt(RecibeChan chan FunctionStruct, wg *sync.WaitGroup) (ChannelOutFunct chan float64) {
reciveData := <-RecibeChan
result := reciveData.FieldOne * float64(reciveData.FieldTwo)
ChannelOutFunct <- result
wg.Done()
return ChannelOutFunct
}
This is the deadlock.
PS C:\Go-Project\src\GoEjemplos> go run main.go
fatal error: all goroutines are asleep - deadlock!
goroutine 1 [chan receive (nil chan)]:
main.main()
C:/Go-Project/src/GoEjemplos/main.go:32 +0x13d
goroutine 19 [chan send (nil chan)]:
GoEjemplos/subFolder.FunctionExt(0xc0000d4000, 0xc0000a2070, 0xc0000c9f18)
C:/Go-Project/src/GoEjemplos/subFolder/functionsPackage.go:19 +0x85
created by main.main
C:/Go-Project/src/GoEjemplos/main.go:30 +0x11a
exit status 2
PS C:\Go-Project\src\GoEjemplos>
Could you explain it to me where is the problem?
Thanks!
The channel functionspackage.ChannelOutFunct is not initialized, thus it is a nil channel. Writing to a nil-channel or reading from a nil-channel will always block.
https://dave.cheney.net/2014/03/19/channel-axioms
I got it to work. This is the code
package main
import (
packagefunctions "GoEjemplos/subFolder"
"fmt"
"sync"
)
func main() {
var wg sync.WaitGroup
var ChannelSendData = make(chan packagefunctions.FunctionStruct, 0)
defer close(ChannelSendData)
var ChanReturn = make(chan float64)
defer close(ChanReturn)
data := packagefunctions.FunctionStruct{
FieldOne: 3.56,
FieldTwo: 2,
}
wg.Add(1)
go func() { ChannelSendData <- data }()
wg.Add(1)
go func() {
ChanReturn = packagefunctions.FunctionExt(ChannelSendData, &wg)
recibeChanReturn := <-ChanReturn
fmt.Println("channelOut: ", recibeChanReturn)
wg.Done()
}()
wg.Wait()
}
the other package is
package packagefunctions
import (
"fmt"
"sync"
)
type FunctionStruct struct {
FieldOne float64
FieldTwo int
}
func FunctionExt(ChanIn chan FunctionStruct, wg *sync.WaitGroup) chan float64 {
reciveData, ok := <-ChanIn
if ok == false {
fmt.Println("channel closed")
}
var result float64
result = reciveData.FieldOne * float64(reciveData.FieldTwo)
var ChannelReturn = make(chan float64, 1)
defer close(ChannelReturn)
ChannelReturn <- result
wg.Done()
return ChannelReturn
}

Get responses from multiple go routines into an array

I need to fetch responses from multiple go routines and put them into an array. I know that channels could be used for this, however I am not sure how I can make sure that all go routines have finished processing the results. Thus I am using a waitgroup.
Code
func main() {
log.Info("Collecting ints")
var results []int32
for _, broker := range e.BrokersByBrokerID {
wg.Add(1)
go getInt32(&wg)
}
wg.Wait()
log.info("Collected")
}
func getInt32(wg *sync.WaitGroup) (int32, error) {
defer wg.Done()
// Just to show that this method may just return an error and no int32
err := broker.Open(config)
if err != nil && err != sarama.ErrAlreadyConnected {
return 0, fmt.Errorf("Cannot connect to broker '%v': %s", broker.ID(), err)
}
defer broker.Close()
return 1003, nil
}
My question
How can I put all the response int32 (which may return an error) into my int32 array, making sure that all go routines have finished their processing work and returned either the error or the int?
If you don't process the return values of the function launched as a goroutine, they are discarded. See What happens to return value from goroutine.
You may use a slice to collect the results, where each goroutine could receive the index to put the results to, or alternatively the address of the element. See Can I concurrently write different slice elements. Note that if you use this, the slice must be pre-allocated and only the element belonging to the goroutine may be written, you can't "touch" other elements and you can't append to the slice.
Or you may use a channel, on which the goroutines send values that include the index or ID of the item they processed, so the collecting goroutine can identify or order them. See How to collect values from N goroutines executed in a specific order?
If processing should stop on the first error encountered, see Close multiple goroutine if an error occurs in one in go
Here's an example how it could look like when using a channel. Note that no waitgroup is needed here, because we know that we expect as many values on the channel as many goroutines we launch.
type result struct {
task int32
data int32
err error
}
func main() {
tasks := []int32{1, 2, 3, 4}
ch := make(chan result)
for _, task := range tasks {
go calcTask(task, ch)
}
// Collect results:
results := make([]result, len(tasks))
for i := range results {
results[i] = <-ch
}
fmt.Printf("Results: %+v\n", results)
}
func calcTask(task int32, ch chan<- result) {
if task > 2 {
// Simulate failure
ch <- result{task: task, err: fmt.Errorf("task %v failed", task)}
return
}
// Simulate success
ch <- result{task: task, data: task * 2, err: nil}
}
Output (try ot on the Go Playground):
Results: [{task:4 data:0 err:0x40e130} {task:1 data:2 err:<nil>} {task:2 data:4 err:<nil>} {task:3 data:0 err:0x40e138}]
I also believe you have to use channel, it must be something like this:
package main
import (
"fmt"
"log"
"sync"
)
var (
BrokersByBrokerID = []int32{1, 2, 3}
)
type result struct {
data string
err string // you must use error type here
}
func main() {
var wg sync.WaitGroup
var results []result
ch := make(chan result)
for _, broker := range BrokersByBrokerID {
wg.Add(1)
go getInt32(ch, &wg, broker)
}
go func() {
for v := range ch {
results = append(results, v)
}
}()
wg.Wait()
close(ch)
log.Printf("collected %v", results)
}
func getInt32(ch chan result, wg *sync.WaitGroup, broker int32) {
defer wg.Done()
if broker == 1 {
ch <- result{err: fmt.Sprintf("error: gor broker 1")}
return
}
ch <- result{data: fmt.Sprintf("broker %d - ok", broker)}
}
Result will look like this:
2019/02/05 15:26:28 collected [{broker 3 - ok } {broker 2 - ok } { error: gor broker 1}]
package main
import (
"fmt"
"log"
"sync"
)
var (
BrokersByBrokerID = []int{1, 2, 3, 4}
)
type result struct {
data string
err string // you must use error type here
}
func main() {
var wg sync.WaitGroup
var results []int
ch := make(chan int)
done := make(chan bool)
for _, broker := range BrokersByBrokerID {
wg.Add(1)
go func(i int) {
defer wg.Done()
ch <- i
if i == 4 {
done <- true
}
}(broker)
}
L:
for {
select {
case v := <-ch:
results = append(results, v)
if len(results) == 4 {
//<-done
close(ch)
break L
}
case _ = <-done:
break
}
}
fmt.Println("STOPPED")
//<-done
wg.Wait()
log.Printf("collected %v", results)
}
Thank cn007b and Edenshaw. My answer is based on their answers.
As Edenshaw commented, need another sync.Waitgroup for goroutine which getting results from channel, or you may get an incomplete array.
package main
import (
"fmt"
"sync"
"encoding/json"
)
type Resp struct {
id int
}
func main() {
var wg sync.WaitGroup
chanRes := make(chan interface{}, 3)
for i := 0; i < 3; i++ {
wg.Add(1)
resp := &Resp{}
go func(i int, resp *Resp) {
defer wg.Done()
resp.id = i
chanRes <- resp
}(i, resp)
}
res := make([]interface{}, 0)
var wg2 sync.WaitGroup
wg2.Add(1)
go func() {
defer wg2.Done()
for v := range chanRes {
res = append(res, v.(*Resp).id)
}
}()
wg.Wait()
close(chanRes)
wg2.Wait()
resStr, _ := json.Marshal(res)
fmt.Println(string(resStr))
}
package main
import (
"fmt"
"log"
"sync"
"time"
)
var (
BrokersByBrokerID = []int{1, 2, 3, 4}
)
type result struct {
data string
err string // you must use error type here
}
func main() {
var wg sync.WaitGroup.
var results []int
ch := make(chan int)
done := make(chan bool)
for _, broker := range BrokersByBrokerID {
wg.Add(1)
go func(i int) {
defer wg.Done()
ch <- i
if i == 4 {
done <- true
}
}(broker)
}
for v := range ch {
results = append(results, v)
if len(results) == 4 {
close(ch)
}
}
fmt.Println("STOPPED")
<-done
wg.Wait()
log.Printf("collected %v", results)
}
</pre>

Goroutines, Callbacks and sync.WaitGroup

With the following code:
package main
import (
"github.com/davecgh/go-spew/spew"
"sync"
"time"
)
func callbackWithTimeout(cbFunc func() ([]byte, error), timeout time.Duration) {
defer wg.Done() //I don't want this function to know about sync.WaitGroup
time.Sleep(timeout)
d, e := cbFunc()
spew.Dump(d)
spew.Dump(e)
}
var wg sync.WaitGroup
func main() {
wg.Add(1)
go func() {
cbFunc := func() ([]byte, error) {
//I feel like I should be able to defer here instead
return nil, nil
}
callbackWithTimeout(cbFunc, time.Duration(4*time.Second))
}()
println("some line")
wg.Wait()
}
In function callbackWithTimeout, I don't want to use defer wg.Done() because it's not callbackWithTimeout()'s concern to wg.Done(). How do I go about implementing such a thing? i.e., remove any sync.WaitGroup in callbackWithTimeout? I have a bit of problem understanding the separation of concerns here as a callback'er function should not have to know about waitgroups but in this case it seems, I have no other choice?
I feel like it should be a caller's responsibility to wg.Done() (which in this case is the cbFunc) but lack any concise reference to documentation or ideas on how to implement it in Go because by definition, all a callback function does is call the function back. So, where I am doing it wrong?
Silly assumptions were made by yours truly during refactoring. Working code below. Many thanks.
package main
import (
"errors"
"github.com/davecgh/go-spew/spew"
"sync"
"time"
)
func callbackWithTimeout(cbFunc func() ([]byte, error), timeout time.Duration) {
time.Sleep(timeout)
d, e := cbFunc()
spew.Dump(d)
spew.Dump(e)
}
func main() {
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
callbackWithTimeout(func() ([]byte, error) {
b := []byte{1, 2, 3, 4}
e := errors.New("error123")
return b, e
}, time.Duration(2*time.Second))
}()
println("some line")
wg.Wait()
}
May be like this?
package main
import (
"sync"
"time"
"github.com/davecgh/go-spew/spew"
)
func callbackWithTimeout(cbFunc func() ([]byte, error), timeout time.Duration) {
time.Sleep(timeout)
d, e := cbFunc()
spew.Dump(d)
spew.Dump(e)
}
func main() {
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done() // move it here
cbFunc := func() ([]byte, error) {
//I feel like I should be able to defer here instead
return nil, nil
}
callbackWithTimeout(cbFunc, time.Duration(4*time.Second))
}()
println("some line")
wg.Wait()
}

Go: One producer many consumers

So I have seen a lot of ways of implementing one consumer and many producers in Go - the classic fanIn function from the Concurrency in Go talk.
What I want is a fanOut function. It takes as a parameter a channel it reads a value from and returns a slice of channels that it writes copies of this value to.
Is there a correct/recommended way of implementing this?
You pretty much described the best way to do it but here is a small sample of code that does it.
Go playground: https://play.golang.org/p/jwdtDXVHJk
package main
import (
"fmt"
"time"
)
func producer(iters int) <-chan int {
c := make(chan int)
go func() {
for i := 0; i < iters; i++ {
c <- i
time.Sleep(1 * time.Second)
}
close(c)
}()
return c
}
func consumer(cin <-chan int) {
for i := range cin {
fmt.Println(i)
}
}
func fanOut(ch <-chan int, size, lag int) []chan int {
cs := make([]chan int, size)
for i, _ := range cs {
// The size of the channels buffer controls how far behind the recievers
// of the fanOut channels can lag the other channels.
cs[i] = make(chan int, lag)
}
go func() {
for i := range ch {
for _, c := range cs {
c <- i
}
}
for _, c := range cs {
// close all our fanOut channels when the input channel is exhausted.
close(c)
}
}()
return cs
}
func fanOutUnbuffered(ch <-chan int, size int) []chan int {
cs := make([]chan int, size)
for i, _ := range cs {
// The size of the channels buffer controls how far behind the recievers
// of the fanOut channels can lag the other channels.
cs[i] = make(chan int)
}
go func() {
for i := range ch {
for _, c := range cs {
c <- i
}
}
for _, c := range cs {
// close all our fanOut channels when the input channel is exhausted.
close(c)
}
}()
return cs
}
func main() {
c := producer(10)
chans := fanOutUnbuffered(c, 3)
go consumer(chans[0])
go consumer(chans[1])
consumer(chans[2])
}
The important part to note is how we close the output channels once the input channel has been exhausted. Also if one of the output channels blocks on the send it will hold up the send on the other output channels. We control the amount of lag by setting the buffer size of the channels.
This solution below is a bit contrived, but it works for me:
package main
import (
"fmt"
"time"
"crypto/rand"
"encoding/binary"
)
func handleNewChannels(arrchangen chan [](chan uint32),
intchangen chan (chan uint32)) {
currarr := []chan uint32{}
arrchangen <- currarr
for {
newchan := <-intchangen
currarr = append(currarr, newchan)
arrchangen <- currarr
}
}
func sendToChannels(arrchangen chan [](chan uint32)) {
tick := time.Tick(1 * time.Second)
currarr := <-arrchangen
for {
select {
case <-tick:
sent := false
var n uint32
binary.Read(rand.Reader, binary.LittleEndian, &n)
for i := 0 ; i < len(currarr) ; i++ {
currarr[i] <- n
sent = true
}
if sent {
fmt.Println("Sent generated ", n)
}
case newarr := <-arrchangen:
currarr = newarr
}
}
}
func handleChannel(tchan chan uint32) {
for {
val := <-tchan
fmt.Println("Got the value ", val)
}
}
func createChannels(intchangen chan (chan uint32)) {
othertick := time.Tick(5 * time.Second)
for {
<-othertick
fmt.Println("Creating new channel! ")
newchan := make(chan uint32)
intchangen <- newchan
go handleChannel(newchan)
}
}
func main() {
arrchangen := make(chan [](chan uint32))
intchangen := make(chan (chan uint32))
go handleNewChannels(arrchangen, intchangen)
go sendToChannels(arrchangen)
createChannels(intchangen)
}
First, see related question What is the neatest idiom for producer/consumer in Go? and One thread showing interest in another thread (consumer / producer). Also, take a look to producer-consumer problem. About concurrency see how to achieve concurrency In Google Go.
We can handle multiple consumers without making the copy of channel data for each consumer.
Go playground: https://play.golang.org/p/yOKindnqiZv
package main
import (
"fmt"
"sync"
)
type data struct {
msg string
consumers int
}
func main() {
ch := make(chan *data) // both block or non-block are ok
var wg sync.WaitGroup
consumerCount := 3 // specify no. of consumers
producer := func() {
obj := &data {
msg: "hello everyone!",
consumers: consumerCount,
}
ch <- obj
}
consumer := func(idx int) {
defer wg.Done()
obj := <-ch
fmt.Printf("consumer %d received data %v\n", idx, obj)
obj.consumers--
if obj.consumers > 0 {
ch <- obj // forward to others
} else {
fmt.Printf("last receiver: %d\n", idx)
}
}
go producer()
for i:=1; i<=consumerCount; i++ {
wg.Add(1)
go consumer(i)
}
wg.Wait()
}

Resources