The following program never prints "Full". With fmt.Println(len(choke)) uncommented, the program outputs "Full" when the channel is full.
package main
import (
"fmt"
)
func main() {
choke := make(chan string, 150000)
go func() {
for i := 0; i < 10000000; i++ {
choke <- string(i)
fmt.Println("i=", i)
}
}()
for {
//fmt.Println(len(choke))
if len(choke) >= 150000 {
fmt.Println("Full")
}
}
}
#tim-heckman explained the cause of this behavior in OP.
How do I detect a channel is full without using a hot loop?
Use a select statement on the write side. It will write to the channel if there is buffer available or a receiver waiting; it will fallthrough to the default case if the channel is full.
func main() {
choke := make(chan string, 150000)
var i int
for {
select {
case choke <- string(i):
i++
default:
fmt.Println("Full")
return
}
}
}
Related
I use sync.WaitGroup with goroutine before, but I want to control the goroutine concurrency,
so I write my waitgroup with concurrency limit like:
package wglimit
import (
"sync"
)
// WaitGroupLimit ...
type WaitGroupLimit struct {
ch chan int
wg *sync.WaitGroup
}
// New ...
func New(size int) *WaitGroupLimit {
if size <= 0 {
size = 1
}
return &WaitGroupLimit{
ch: make(chan int, size), // buffer chan to limit concurrency
wg: &sync.WaitGroup{},
}
}
// Add ...
func (wgl *WaitGroupLimit) Add(delta int) {
for i := 0; i < delta; i++ {
wgl.ch <- 1
wgl.wg.Add(1)
}
}
// Done ...
func (wgl *WaitGroupLimit) Done() {
wgl.wg.Done()
<-wgl.ch
}
// Wait ...
func (wgl *WaitGroupLimit) Wait() {
close(wgl.ch)
wgl.wg.Wait()
}
And then I use it to control the goroutine concurrency, for example:
jobs := ["1", "2", "3", "4"] // some jobs
// wg := sync.WaitGroup{} // have no concurrency limit
wg := wglimit.New(2) // limit 2 goroutine
for _, job := range jobs {
wg.Add(1)
go func(job string) {
// job worker
defer wg.Done()
}(job)
}
wg.Wait()
And it looks like worked when running.
But Test Failed:
package wglimit
import (
"runtime"
"testing"
"time"
)
func TestGoLimit(t *testing.T) {
var limit int = 5
wglimit := New(limit)
for i := 0; i < 10000; i++ {
wglimit.Add(1)
go func() {
defer wglimit.Done()
time.Sleep(time.Millisecond)
if runtime.NumGoroutine() > limit+2 {
println(runtime.NumGoroutine()) // will print 9 , cocurrent limit fail ?
t.Errorf("FAIL")
}
}()
}
wglimit.Wait()
}
When testing, the goroutine numbers is bigger than my limit, it seems like the cocurrent limit fail.
Anything wrong with my WaitGroupLimit code and why?
Anything wrong with my WaitGroupLimit code [...]?
No.
The problem is runtime.NumGoroutine() doesn't do what you seem to think it does. It counts all goroutines, i.e. not only the ones you start but also the goroutines the runtime uses itself, e.g. for concurrent garbage collection. NumGoroutine is thus higher than your limit.
Your code is fine, your test isn't. Do not try to get clever in testing and test what you code really does: It blocks on Add until the limited resource is available. Test that and not a goroutine count which is just a (bad) proxy for the desired behaviour in your test.
I'm starting out with Go and I'm now writing a simple program which reads out data from a sensor and puts that into a channel to do some calculations with it. I now have it working as follows:
package main
import (
"fmt"
"time"
"strconv"
)
func get_sensor_data(c chan float64) {
time.Sleep(1 * time.Second) // wait a second before sensor data starts pooring in
c <- 2.1 // Sensor data starts being generated
c <- 2.2
c <- 2.3
c <- 2.4
c <- 2.5
}
func main() {
s := 1.1
c := make(chan float64)
go get_sensor_data(c)
for {
select {
case s = <-c:
fmt.Println("the next value of s from the channel: " + strconv.FormatFloat(s, 'f', 1, 64))
default:
// no new values in the channel
}
fmt.Println(s)
time.Sleep(500 * time.Millisecond) // Do heavy "work"
}
}
This works fine, but the sensor generates a lot of data, and I'm always only interested in the latest data. With this setup however, it only reads out the next item with every loop, which means that if the channel at some point contains 20 values, the newest value only is read out after 10 seconds.
Is there a way for a channel to always only contain one value at a time, so that I always only get the data I'm interested in, and no unnecessary memory is used by the channel (although the memory is the least of my worries)?
Channels are best thought of as queues (FIFO). Therefore you can't really skip around. However there are libraries out there that do stuff like this: https://github.com/cloudfoundry/go-diodes is an atomic ring buffer that will overwrite old data. You can set a smaller size if you like.
All that being said, it doesn't sound like you need a queue (or ring buffer). You just need a mutex:
type SensorData struct{
mu sync.RWMutex
last float64
}
func (d *SensorData) Store(data float64) {
mu.Lock()
defer mu.Unlock()
d.last = data
}
func (d *SensorData) Get() float64 {
mu.RLock()
defer mu.RUnlock()
return d.last
}
This uses a RWMutex which means many things can read from it at the same time while only a single thing can write. It will store a single entry much like you said.
No. Channels are FIFO buffers, full stop. That is how channels work and their only purpose. If you only want the latest value, consider just using a single variable protected by a mutex; write to it whenever new data comes in, and whenever you read it, you will always be reading the latest value.
Channels serves a specific purpose. You might want to use a code that is inside a lock and update the variable whenever new value is to be set.
This way reciever will always get the latest value.
You cannot get that from one channel directly, but you can use one channel per value and get notified when there are new values:
package main
import (
"fmt"
"strconv"
"sync"
"time"
)
type LatestChannel struct {
n float64
next chan struct{}
mu sync.Mutex
}
func New() *LatestChannel {
return &LatestChannel{next: make(chan struct{})}
}
func (c *LatestChannel) Push(n float64) {
c.mu.Lock()
c.n = n
old := c.next
c.next = make(chan struct{})
c.mu.Unlock()
close(old)
}
func (c *LatestChannel) Get() (float64, <-chan struct{}) {
c.mu.Lock()
n := c.n
next := c.next
c.mu.Unlock()
return n, next
}
func getSensorData(c *LatestChannel) {
time.Sleep(1 * time.Second)
c.Push(2.1)
time.Sleep(100 * time.Millisecond)
c.Push(2.2)
time.Sleep(100 * time.Millisecond)
c.Push(2.3)
time.Sleep(100 * time.Millisecond)
c.Push(2.4)
time.Sleep(100 * time.Millisecond)
c.Push(2.5)
}
func main() {
s := 1.1
c := New()
_, hasNext := c.Get()
go getSensorData(c)
for {
select {
case <-hasNext:
s, hasNext = c.Get()
fmt.Println("the next value of s from the channel: " + strconv.FormatFloat(s, 'f', 1, 64))
default:
// no new values in the channel
}
fmt.Println(s)
time.Sleep(250 * time.Millisecond) // Do heavy "work"
}
}
If you do not need the notify about new value, you can try to read Channels inside channels pattern in Golang.
Try this package https://github.com/subbuv26/chanup
It allows the producer to update the channel with latest value, which replaces the latest value. And produces does not get blocked. (with this, stale values gets overridden).
So, on the consumer side, always only the latest item gets read.
import "github.com/subbuv26/chanup"
ch := chanup.GetChan()
_ := ch.Put(testType{
a: 10,
s: "Sample",
})
_ := ch.Update(testType{
a: 20,
s: "Sample2",
})
// Continue updating with latest values
...
...
// On consumer end
val := ch.Get()
// val contains latest value
There is another way to solve this problem (trick)
sender work faster: sender remove channel if channel_length > 1
go func() {
for {
msg:=strconv.Itoa(int(time.Now().Unix()))
fmt.Println("make: ",msg," at:",time.Now())
messages <- msg
if len(messages)>1{
//remove old message
<-messages
}
time.Sleep(2*time.Second)
}
}()
receiver work slower:
go func() {
for {
channLen :=len(messages)
fmt.Println("len is ",channLen)
fmt.Println("received",<-messages)
time.Sleep(10*time.Second)
}
}()
OR, we can delete old message from receiver side
(read message like delete it)
There is an elegant channel-only solution. If you're OK with adding one more channel and goroutine - you can introduce a buferless channel and a goroutine that tries to send the latest value from your channel to it:
package main
import (
"fmt"
"time"
)
func wrapLatest(ch <-chan int) <-chan int {
result := make(chan int) // important that this one i unbuffered
go func() {
defer close(result)
value, ok := <-ch
if !ok {
return
}
LOOP:
for {
select {
case value, ok = <-ch:
if !ok {
return
}
default:
break LOOP
}
}
for {
select {
case value, ok = <-ch:
if !ok {
return
}
case result <- value:
if value, ok = <-ch; !ok {
return
}
}
}
}()
return result
}
func main() {
sendChan := make(chan int, 10) // may be buffered or not
for i := 0; i < 10; i++ {
sendChan <- i
}
go func() {
for i := 10; i < 20; i++ {
sendChan <- i
time.Sleep(time.Second)
}
close(sendChan)
}()
recvChan := wrapLatest(sendChan)
for i := range recvChan {
fmt.Println(i)
time.Sleep(time.Second * 2)
}
}
I have a simple concurrency use case in go, and I cannot figure out an elegant solution to my problem.
I want to write a method fetchAll that queries an unspecified number of resources from remote servers in parallel. If any of the fetches fails, I want to return that first error immediately.
My initial implementation leaks goroutines:
package main
import (
"fmt"
"math/rand"
"sync"
"time"
)
func fetchAll() error {
wg := sync.WaitGroup{}
errs := make(chan error)
leaks := make(map[int]struct{})
defer fmt.Println("these goroutines leaked:", leaks)
// run all the http requests in parallel
for i := 0; i < 4; i++ {
leaks[i] = struct{}{}
wg.Add(1)
go func(i int) {
defer wg.Done()
defer delete(leaks, i)
// pretend this does an http request and returns an error
time.Sleep(time.Duration(rand.Intn(100)) * time.Millisecond)
errs <- fmt.Errorf("goroutine %d's error returned", i)
}(i)
}
// wait until all the fetches are done and close the error
// channel so the loop below terminates
go func() {
wg.Wait()
close(errs)
}()
// return the first error
for err := range errs {
if err != nil {
return err
}
}
return nil
}
func main() {
fmt.Println(fetchAll())
}
Playground: https://play.golang.org/p/Be93J514R5
I know from reading https://blog.golang.org/pipelines that I can create a signal channel to cleanup the other threads. Alternatively, I could probably use context to accomplish it. But it seems like such a simple use case should have a simpler solution that I'm missing.
Using Error Group makes this even simpler. This automatically waits for all the supplied Go Routines to complete successfully, or cancels all those remaining in the case of any one routine returning an error (in which case that error is the one bubble back up to the caller).
package main
import (
"context"
"fmt"
"math/rand"
"time"
"golang.org/x/sync/errgroup"
)
func fetchAll(ctx context.Context) error {
errs, ctx := errgroup.WithContext(ctx)
// run all the http requests in parallel
for i := 0; i < 4; i++ {
errs.Go(func() error {
// pretend this does an http request and returns an error
time.Sleep(time.Duration(rand.Intn(100)) * time.Millisecond)
return fmt.Errorf("error in go routine, bailing")
})
}
// Wait for completion and return the first error (if any)
return errs.Wait()
}
func main() {
fmt.Println(fetchAll(context.Background()))
}
All but one of your goroutines are leaked, because they're still waiting to send to the errs channel - you never finish the for-range that empties it. You're also leaking the goroutine who's job is to close the errs channel, because the waitgroup is never finished.
(Also, as Andy pointed out, deleting from map is not thread-safe, so that'd need protection from a mutex.)
However, I don't think maps, mutexes, waitgroups, contexts etc. are even necessary here. I'd rewrite the whole thing to just use basic channel operations, something like the following:
package main
import (
"fmt"
"math/rand"
"time"
)
func fetchAll() error {
var N = 4
quit := make(chan bool)
errc := make(chan error)
done := make(chan error)
for i := 0; i < N; i++ {
go func(i int) {
// dummy fetch
time.Sleep(time.Duration(rand.Intn(100)) * time.Millisecond)
err := error(nil)
if rand.Intn(2) == 0 {
err = fmt.Errorf("goroutine %d's error returned", i)
}
ch := done // we'll send to done if nil error and to errc otherwise
if err != nil {
ch = errc
}
select {
case ch <- err:
return
case <-quit:
return
}
}(i)
}
count := 0
for {
select {
case err := <-errc:
close(quit)
return err
case <-done:
count++
if count == N {
return nil // got all N signals, so there was no error
}
}
}
}
func main() {
rand.Seed(time.Now().UnixNano())
fmt.Println(fetchAll())
}
Playground link: https://play.golang.org/p/mxGhSYYkOb
EDIT: There indeed was a silly mistake, thanks for pointing it out. I fixed the code above (I think...). I also added some randomness for added Realism™.
Also, I'd like to stress that there really are multiple ways to approach this problem, and my solution is but one way. Ultimately it comes down to personal taste, but in general, you want to strive towards "idiomatic" code - and towards a style that feels natural and easy to understand for you.
Here's a more complete example using errgroup suggested by joth. It shows processing successful data, and will exit on the first error.
https://play.golang.org/p/rU1v-Mp2ijo
package main
import (
"context"
"fmt"
"golang.org/x/sync/errgroup"
"math/rand"
"time"
)
func fetchAll() error {
g, ctx := errgroup.WithContext(context.Background())
results := make(chan int)
for i := 0; i < 4; i++ {
current := i
g.Go(func() error {
// Simulate delay with random errors.
time.Sleep(time.Duration(rand.Intn(100)) * time.Millisecond)
if rand.Intn(2) == 0 {
return fmt.Errorf("goroutine %d's error returned", current)
}
// Pass processed data to channel, or receive a context completion.
select {
case results <- current:
return nil
// Close out if another error occurs.
case <-ctx.Done():
return ctx.Err()
}
})
}
// Elegant way to close out the channel when the first error occurs or
// when processing is successful.
go func() {
g.Wait()
close(results)
}()
for result := range results {
fmt.Println("processed", result)
}
// Wait for all fetches to complete.
return g.Wait()
}
func main() {
fmt.Println(fetchAll())
}
As long as each goroutine completes, you won't leak anything. You should create the error channel as buffered with the buffer size equal to the number of goroutines so that the send operations on the channel won't block. Each goroutine should always send something on the channel when it finishes, whether it succeeds or fails. The loop at the bottom can then just iterate for the number of goroutines and return if it gets a non-nil error. You don't need the WaitGroup or the other goroutine that closes the channel.
I think the reason it appears that goroutines are leaking is that you return when you get the first error, so some of them are still running.
By the way, maps are not goroutine safe. If you share a map among goroutines and some of them are making changes to the map, you need to protect it with a mutex.
This answer includes the ability to get the responses back into doneData -
package main
import (
"fmt"
"math/rand"
"os"
"strconv"
)
var doneData []string // responses
func fetchAll(n int, doneCh chan bool, errCh chan error) {
partialDoneCh := make(chan string)
for i := 0; i < n; i++ {
go func(i int) {
if r := rand.Intn(100); r != 0 && r%10 == 0 {
// simulate an error
errCh <- fmt.Errorf("e33or for reqno=" + strconv.Itoa(r))
} else {
partialDoneCh <- strconv.Itoa(i)
}
}(i)
}
// mutation of doneData
for d := range partialDoneCh {
doneData = append(doneData, d)
if len(doneData) == n {
close(partialDoneCh)
doneCh <- true
}
}
}
func main() {
// rand.Seed(1)
var n int
var e error
if len(os.Args) > 1 {
if n, e = strconv.Atoi(os.Args[1]); e != nil {
panic(e)
}
} else {
n = 5
}
doneCh := make(chan bool)
errCh := make(chan error)
go fetchAll(n, doneCh, errCh)
fmt.Println("main: end")
select {
case <-doneCh:
fmt.Println("success:", doneData)
case e := <-errCh:
fmt.Println("failure:", e, doneData)
}
}
Execute using go run filename.go 50 where N=50 i.e amount of parallelism
This question already has answers here:
What is channel buffer size?
(3 answers)
Closed 7 years ago.
I went through a series of definitions to figure out how the buffer works but I just don't get it. Here is an example below, I changed the value of the buffer but I have no clue about what it does. Can some one explain it to me based on this example and provide some test cases of how/why it's working? Thanks.
package main
import (
"fmt"
"time"
)
func send(out, finish chan bool) {
for i := 0; i < 5; i++ {
out <- true
time.Sleep(1 * time.Second)
fmt.Println("Fin d'une écriture")
}
finish <- true
close(out)
}
func recv(in, finish chan bool) {
for _ = range in {
fmt.Println("Fin d'une lecture")
time.Sleep(10 * time.Second)
}
finish <- true
}
func main() {
chanFoo := make(chan bool, 3)
chanfinish := make(chan bool)
go send(chanFoo, chanfinish)
go recv(chanFoo, chanfinish)
<-chanfinish
<-chanfinish
}
If a channel does not have a buffer then only a single item can be sent on it at a time. This means code that sends on it will block until some receiver reads the item out of the channel. Here's a contrived example; https://play.golang.org/p/HM8jdIFqsN
package main
import (
"fmt"
)
func main() {
blocker := make(chan bool)
nonBlocker := make(chan bool, 5)
for i := 0; i < 5; i++ {
nonBlocker <- true
fmt.Println("We keep going")
}
go func () {
for i := 0; i < 5; i++ {
blocker <- true
fmt.Println("We block cause that channel is full")
} }()
}
There's plenty of other things I could do to demonstrate the same but the basic idea is, if you pass a channel into some goroutine and the channel is not buffered, the goroutine which sends on the channel will block until the item it sent is received. With a buffered channel you can send as long as the buffer isn't at capacity. Basically, if you spin up goroutines which are doing work and returning the results and they're moving faster than the code that spawned them, you may want to use a buffered channel to open up that bottle neck.
EDIT: If it's still not obvious what's happening look at this; https://play.golang.org/p/9SXc4M1to4
package main
import (
"fmt"
)
func main() {
blocker := make(chan bool)
nonBlocker := make(chan bool, 5)
for i := 0; i < 5; i++ {
nonBlocker <- true
fmt.Println("We keep going")
}
go func () {
for i := 0; i < 5; i++ {
blocker <- true
fmt.Println("Now we see this cause the reciever keeps opening the channel up again!")
} }()
for i := 0; i < 5; i++ {
<-blocker
}
}
I am studying ‘Go Concurrency Pattern' from https://talks.golang.org/2012/concurrency.slide#25'
Question:
How the channel share variable from it's outside ? In this case i has been shared.
Variable at point A and point B seems to have some special relation ? What is it ?
What does it means for ?
for i := 0; ; i++
Main code:
package main
import (
"fmt"
"math/rand"
"time"
)
func boring(msg string) <-chan string { // Returns receive-only channel of strings.
c := make(chan string)
go func() { // We launch the goroutine from inside the function.
for i := 0; ; i++ { // <--------- point B
c <- fmt.Sprintf("%s %d", msg, i)
time.Sleep(time.Duration(rand.Intn(1e3)) * time.Millisecond)
}
}()
return c // Return the channel to the caller.
}
func main() {
c := boring("boring!") // Function returning a channel.
for i := 0; i < 5; i++ { // <--------- point A
fmt.Printf("You say: %q\n", <-c)
}
fmt.Println("You're boring; I'm leaving.")
}
output:
You say: "boring! 0"
You say: "boring! 1"
You say: "boring! 2"
You say: "boring! 3"
You say: "boring! 4"
You're boring; I'm leaving.
Program exited.
The for (i := 0; ; i++) { } creates an index the increments forever.
When you make(chan string ) you've created a read/write channel. You also reference the channel inside the go function and pass it out as the return value. Go does an analysis of how variables are used called "escape analysis" and chooses whether to make the channel on the heap or on the stack, in your case on the heap so that when the function creating the channel exits the channel doesn't get deallocated.