1-what is the condition that make chan break?
deliveries <-chan amqp.Delivery
for d:= range deliveries{
..
}
If there is no more data in chan deliveries about a few minutes,that it will break.
Is the code up is same to below?
deliveries <- chan amqp.Delivery
for{
d,ok:=<-deliveries
if !ok{
break
}
//code
}
2-Why does chan not only return data but also status?And what does the "ok" mean?
3-How does the chan realize?"ok" is the status about client,Why can it return the "ok"?
I will answer question 2 and 3 first because the answer provides context for my answer to question 1.
2, 3) The builtin function close(c) records that no more values will be sent to the channel c.
The second result in a receive expression is a bool indicating if the operation was successful. The second result is true if a sent value was received or false if the zero value was received because the channel was closed.
1) Range over a channel receives values sent on the channel until the channel is closed.
The following loops are very similar. They both receive values until the channel is closed.
for v := range c {
// code
}
for {
v, ok := <-c
if != ok {
break
}
// code
}
The main difference between these loops is the scope of of the variable v. The scope of v is outside of the first loop and inside the second loop. This distinction is important if you use a closure and goroutine in the loop.
1) Code 1 and 2 differ: The second also fetches ok which indicates whether the channel was closed by the sender. This makes your code more robust.
2) Channels can only transfer one type of message. If you need status code you ahve to put it inside the message.
Related
While trying few experiments in channels, I came up with below code:
var strChannel = make(chan string, 30)
var mutex = &sync.Mutex{}
func main() {
go sampleRoutine()
for i := 0; i < 10; i++ {
mutex.Lock()
strChannel <- strconv.FormatInt(int64(i), 10)
mutex.Unlock()
time.Sleep(1 * time.Second)
}
time.Sleep(10 * time.Second)
}
func sampleRoutine() {
/* A: for msg := range strChannel{*/
/* B: for {
msg := <-strChannel*/
log.Println("got message ", msg, strChannel)
if msg == "3" {
mutex.Lock()
strChannel = make(chan string, 20)
mutex.Unlock()
}
}
}
Basically here while listening to a given channel, I am assigning the channel variable to a new channel in a specific condition (here when msg == 3).
When I use the code in comment block B it works as expected, i.e. the loop moves on to the newly created channel and prints 4-10.
However comment block A which I believe is just a different way to write the loop doesn't work i.e. after printing "3" it stops.
Could someone please let me know the reason for this behavior?
And also is code like this where a routine listening on a channel, creating a new one safe?
In Go, for statement evaluate the value on the right side of range before the loop begins.
That means changing the value of the variable on the right side of range will take no effects. So in your code, in Variant A, msg is ever-iterating over the orignal channel and never changed. In Varaint B, it works as intended as the channel is being evaluated per iteration.
The concept is a little bit tricky. It does not mean you cannot modify items of the slice or map on the right side of range. If you look deeper into it, you will find that in Go, map and slice stores a pointer, and modify its item does not change that pointer, so it has effects.
It is even more trickier in case of array. Modifying item of an array on the right side of range has no effects. This is due to Go's mechanism about storing array as a value.
Playground examples: https://play.golang.org/p/wzPfGHFYrnv
Sending and reading from a channel do not need to be protected by a mutex: They act as synchronisation primitives by themself (that's one of the main ideas behind sending/receiving) on a channel.
There is no difference between variant A and B because you do not close the channel. But...
Resigning a new channel to strChannel while iterating the old channel is wrong. Don't do that. Not even with the B variant. Think about range strChannel as "please range over the values in the channel currently stored in the variable strChannel". This range will "continue on the original channel" and storing a new channel in the same variable doesn't change this. Avoid such code!
func First(query string, replicas ...Search) Result {
c := make(chan Result)
searchReplica := func(i int) {
c <- replicas[i](query)
}
for i := range replicas {
go searchReplica(i)
}
return <-c
}
This function is from the slides of Rob Pike on go concurrency patterns in 2012. I think there is a resource leak in this function. As the function return after the first send & receive pair happens on channel c, the other go routines try to send on channel c. So there is a resource leak here. Anyone knows golang well can confirm this? And how can I detect this leak using what kind of golang tooling?
Yes, you are right (for reference, here's the link to the slide). In the above code only one launched goroutine will terminate, the rest will hang on attempting to send on channel c.
Detailing:
c is an unbuffered channel
there is only a single receive operation, in the return statement
A new goroutine is launched for each element of replicas
each launched goroutine sends a value on channel c
since there is only 1 receive from it, one goroutine will be able to send a value on it, the rest will block forever
Note that depending on the number of elements of replicas (which is len(replicas)):
if it's 0: First() would block forever (no one sends anything on c)
if it's 1: would work as expected
if it's > 1: then it leaks resources
The following modified version will not leak goroutines, by using a non-blocking send (with the help of select with default branch):
searchReplica := func(i int) {
select {
case c <- replicas[i](query):
default:
}
}
The first goroutine ready with the result will send it on channel c which will be received by the goroutine running First(), in the return statement. All other goroutines when they have the result will attempt to send on the channel, and "seeing" that it's not ready (send would block because nobody is ready to receive from it), the default branch will be chosen, and thus the goroutine will end normally.
Another way to fix it would be to use a buffered channel:
c := make(chan Result, len(replicas))
And this way the send operations would not block. And of course only one (the first sent) value will be received from the channel and returned.
Note that the solution with any of the above fixes would still block if len(replicas) is 0. To avoid that, First() should check this explicitly, e.g.:
func First(query string, replicas ...Search) Result {
if len(replicas) == 0 {
return Result{}
}
// ...rest of the code...
}
Some tools / resources to detect leaks:
https://github.com/fortytw2/leaktest
https://github.com/zimmski/go-leak
https://medium.com/golangspec/goroutine-leak-400063aef468
https://blog.minio.io/debugging-go-routine-leaks-a1220142d32c
I am learning about channels and concurrency in Go and I am confused on how the code below works.
package main
import (
"fmt"
"time"
"sync/atomic"
)
var workerID int64
var publisherID int64
func main() {
input := make(chan string)
go workerProcess(input)
go workerProcess(input)
go workerProcess(input)
go publisher(input)
go publisher(input)
go publisher(input)
go publisher(input)
time.Sleep(1 * time.Millisecond)
}
// publisher pushes data into a channel
func publisher(out chan string) {
atomic.AddInt64(&publisherID, 1)
thisID := atomic.LoadInt64(&publisherID)
dataID := 0
for {
dataID++
fmt.Printf("publisher %d is pushing data\n", thisID)
data := fmt.Sprintf("Data from publisher %d. Data %d", thisID, dataID)
out <- data
}
}
func workerProcess(in <-chan string) {
atomic.AddInt64(&workerID, 1)
thisID := atomic.LoadInt64(&workerID)
for {
fmt.Printf("%d: waiting for input...\n", thisID)
input := <-in
fmt.Printf("%d: input is: %s\n", thisID, input)
}
}
This is what I understand please correct me if I'm wrong:
workerProcess Goroutine:
Input channel taken as the argument which is assigned as the in channel.
In the for loop, the first printf is executed.
The value off the in channel is assigned to the variable input.
The last printf is executed to show the value of thisID and input.(This doesn't really execute till after other Goroutines run).
publisher Goroutine:
Input channel taken as the argument which is assigned as the out channel.
In the for loop, dataID is incremented then first printf is executed.
The string is assigned to data.
The value of data is passed to out.
Questions:
Where is the value from out getting passed to in the publisher Goroutine? It seems to be only in the scope of the for loop, wouldn't this cause deadlock. Since this is an unbuffered channel.
I don't get how the workerProcess gets the data from publisher if
all the Goroutines in main have the arguments as the channel input.
I'm used to having code written like this if the output of one function
is used in another:
foo = fcn1(input)
fcn2(foo)
I suspect it has something with how Goroutines run in the background but I'm not to sure I would appreciate an explanation.
Why isn't the last printf statement in workerProcess not executed? My guess is the channel is empty, so it's waiting for a value.
Part of the output:
1: waiting for input...
publisher 1 is pushing data
publisher 1 is pushing data
1: input is: Data from publisher 1. Data 1
1: waiting for input...
1: input is: Data from publisher 1. Data 2
1: waiting for input...
publisher 1 is pushing data
publisher 1 is pushing data
publisher 2 is pushing data
2: waiting for input..
You have one channel, it is made with make in main.
This one single channel is named in in workerProcess and out in publisher. out is a function argument to publisher which answers question 1.
Question 2: That's the whole purpose of a channel. What you stuff into a channel on its input side comes out at its output side. If a function has reference to (one end of) such a channel it may communicate with someone else having reference to the same channel (its other end). What producer send to the channel is received by workerProcess. This send and receive is done through the special operator <- on Go. A fact you got slightly wrong in your explanation.
out <- data takes data and sends it through the channel named out until <- in reads it from the channel (remember in and out name the same channel from main). That's how workerProcess and publisher communicate.
Question 3 is a duplicate. Your whole program terminates once main is done (in your case after 1 millisecond. Nothing happens after termination of the program. Give the program more time to execute. (The non-printing is unrelated to the channel).
Unbuffered channels block receivers until data is available on the channel. It's not clear to me how this blocking behaves with multiple receivers on the same channel (say when using goroutines). I am sure they would all block as long as there is no data sent on that channel.
But what happens once I send a single value to that channel? Which receiver/goroutine will get the data and therefore unblock? All of them? The first in line? Random?
A single random (non-deterministic) one will receive it.
See the language spec:
Execution of a "select" statement proceeds in several steps:
For all the cases in the statement, the channel operands of receive operations and the channel and right-hand-side expressions of send
statements are evaluated exactly once, in source order, upon entering
the "select" statement. The result is a set of channels to receive
from or send to, and the corresponding values to send. Any side
effects in that evaluation will occur irrespective of which (if any)
communication operation is selected to proceed. Expressions on the
left-hand side of a RecvStmt with a short variable declaration or
assignment are not yet evaluated.
If one or more of the communications can proceed, a single one that can proceed is chosen via a uniform pseudo-random selection.
Otherwise, if there is a default case, that case is chosen. If there
is no default case, the "select" statement blocks until at least one
of the communications can proceed.
Unless the selected case is the default case, the respective communication operation is executed.
If the selected case is a RecvStmt with a short variable declaration or an assignment, the left-hand side expressions are
evaluated and the received value (or values) are assigned.
The statement list of the selected case is executed.
By default the goroutine communication is synchronous and unbuffered: sends do not complete until there is a receiver to accept the value. There must be a receiver ready to receive data from the channel and then the sender can hand it over directly to the receiver.
So channel send/receive operations block until the other side is ready:
1. A send operation on a channel blocks until a receiver is available for the same channel: if there’s no recipient for the value on ch, no other value can be put in the channel. And the other way around: no new value can be sent in ch when the channel is not empty! So the send operation will wait until ch becomes available again.
2. A receive operation for a channel blocks until a sender is available for the same channel: if there is no value in the channel, the receiver blocks.
This is illustrated in the below example:
package main
import "fmt"
func main() {
ch1 := make(chan int)
go pump(ch1) // pump hangs
fmt.Println(<-ch1) // prints only 0
}
func pump(ch chan int) {
for i:= 0; ; i++ {
ch <- i
}
}
Because there is no receiver the goroutine hangs and print only the first number.
To workaround this we need to define a new goroutine which reads from the channel in an infinite loop.
func receive(ch chan int) {
for {
fmt.Println(<- ch)
}
}
Then in main():
func main() {
ch := make(chan int)
go pump(ch)
receive(ch)
}
If the program is allowing multiple goroutines to receive on a single channel then the sender is broadcasting. Each receiver should be equally able to process the data. So it does not matter what mechanism the go runtime uses to decide which of the many goroutine receivers will run Cf. https://github.com/golang/go/issues/247. But only ONE will run for each sent item if the channel is unbuffered.
There have been some discussion about this
But what is established in the Go Memory Model is that it will be at most one of them.
Each send on a particular channel is matched to a corresponding receive from that channel, usually in a different goroutine.
That isn't as clear cut as I would like, but later down they give this example of a semaphore implementation
var limit = make(chan int, 3)
func main() {
for _, w := range work {
go func(w func()) {
limit <- 1
w()
// if it were possible for more than one channel to receive
// from a single send, it would be possible for this to release
// more than one "lock", making it an invalid semaphore
// implementation
<-limit
}(w)
}
select{}
}
So, right now, I just pass a pointer to a Queue object (implementation doesn't really matter) and call queue.add(result) at the end of goroutines that should add things to the queue.
I need that same sort of functionality—and of course doing a loop checking completion with the comma ok syntax is unacceptable in terms of performance versus the simple queue add function call.
Is there a way to do this better, or not?
There are actually two parts to your question: how does one queue data in Go, and how does one use a channel without blocking.
For the first part, it sounds like what you need to do is instead of using the channel to add things to the queue, use the channel as a queue. For example:
var (
ch = make(chan int) // You can add an int parameter to this make call to create a buffered channel
// Do not buffer these channels!
gFinished = make(chan bool)
processFinished = make(chan bool)
)
func f() {
go g()
for {
// send values over ch here...
}
<-gFinished
close(ch)
}
func g() {
// create more expensive objects...
gFinished <- true
}
func processObjects() {
for val := range ch {
// Process each val here
}
processFinished <- true
}
func main() {
go processObjects()
f()
<-processFinished
}
As for how you can make this more asynchronous, you can (as cthom06 pointed out) pass a second integer to the make call in the second line which will make send operations asynchronous until the channel's buffer is full.
EDIT: However (as cthom06 also pointed out), because you have two goroutines writing to the channel, one of them has to be responsible for closing the channel. Also, my previous revision would exit before processObjects could complete. The way I chose to synchronize the goroutines is by creating a couple more channels that pass around dummy values to ensure that the cleanup gets finished properly. Those channels are specifically unbuffered so that the sends happen in lock-step.