I'm new to Golang, and may I resize the buffered channel like:
var xxx // just declaration.
...
xxx = make(chan string, mysize) // I know `xxx := make(chan string, mysize)` work, but I don't want it.
...
xxx <- "123" // using the buffered channel.
Once a channel is created, its capacity (buffer size) cannot be changed. You can only create and assign a new channel with different capacity.
Note that just declaring a variable of channel type does not create and initialize a channel, it will be its zero value which is nil for channel types:
var xxx chan int
Here xxx is a variable of chan int, but it's not yet initialized, you cannot send any values on it. You have to create one with the builtin make() function, e.g.:
var xxx chan int = make(chan int, 10)
Or simply:
var xxx = make(chan int, 10)
Its capactiy will be 10 and it cannot be changed. You can however assign a new channel value to it, e.g.:
xxx = make(chan int, 20)
But know that if multiple goroutines access and use the xxx variable, the above operation may cause a data race, and changing the value of xxx should be synchronized with other goroutines that read it.
Also note that if goroutines do not access xxx but the channel is "passed" to them, then assigning a new channel value to xxx will not affect them, those goroutines will not know about the new channel and will continue to use the old, passed channel.
Related
I'm implementing a simple mechanism of passing variable between two goroutines with a channel. Here is my code:
pipe := make(chan string)
go func(out chan string, data string) { //1st goroutine
out <- DataSignerMd5(data)
}(pipe, data)
go func(in chan string) { //2nd goroutine
data := <-in
in <- DataSignerCrc32(data)
}(pipe)
crcMdData := <- pipe
More likely, crcMdData pulls a variable from pipe before 2nd goroutine. I guess that I simply can create another channel to make this work. But maybe it's possible with a single pipe?
You should use a second channel for what you want to do. You could get away with using a single channel and switching on the result, but that's not really ideal - you're basically trying to put two different types of objects into the same channel, and your program will end up being a lot cleaner and easier to reason about if you just have one channel per data type / intended transformation.
I am learning about channels and concurrency in Go and I am confused on how the code below works.
package main
import (
"fmt"
"time"
"sync/atomic"
)
var workerID int64
var publisherID int64
func main() {
input := make(chan string)
go workerProcess(input)
go workerProcess(input)
go workerProcess(input)
go publisher(input)
go publisher(input)
go publisher(input)
go publisher(input)
time.Sleep(1 * time.Millisecond)
}
// publisher pushes data into a channel
func publisher(out chan string) {
atomic.AddInt64(&publisherID, 1)
thisID := atomic.LoadInt64(&publisherID)
dataID := 0
for {
dataID++
fmt.Printf("publisher %d is pushing data\n", thisID)
data := fmt.Sprintf("Data from publisher %d. Data %d", thisID, dataID)
out <- data
}
}
func workerProcess(in <-chan string) {
atomic.AddInt64(&workerID, 1)
thisID := atomic.LoadInt64(&workerID)
for {
fmt.Printf("%d: waiting for input...\n", thisID)
input := <-in
fmt.Printf("%d: input is: %s\n", thisID, input)
}
}
This is what I understand please correct me if I'm wrong:
workerProcess Goroutine:
Input channel taken as the argument which is assigned as the in channel.
In the for loop, the first printf is executed.
The value off the in channel is assigned to the variable input.
The last printf is executed to show the value of thisID and input.(This doesn't really execute till after other Goroutines run).
publisher Goroutine:
Input channel taken as the argument which is assigned as the out channel.
In the for loop, dataID is incremented then first printf is executed.
The string is assigned to data.
The value of data is passed to out.
Questions:
Where is the value from out getting passed to in the publisher Goroutine? It seems to be only in the scope of the for loop, wouldn't this cause deadlock. Since this is an unbuffered channel.
I don't get how the workerProcess gets the data from publisher if
all the Goroutines in main have the arguments as the channel input.
I'm used to having code written like this if the output of one function
is used in another:
foo = fcn1(input)
fcn2(foo)
I suspect it has something with how Goroutines run in the background but I'm not to sure I would appreciate an explanation.
Why isn't the last printf statement in workerProcess not executed? My guess is the channel is empty, so it's waiting for a value.
Part of the output:
1: waiting for input...
publisher 1 is pushing data
publisher 1 is pushing data
1: input is: Data from publisher 1. Data 1
1: waiting for input...
1: input is: Data from publisher 1. Data 2
1: waiting for input...
publisher 1 is pushing data
publisher 1 is pushing data
publisher 2 is pushing data
2: waiting for input..
You have one channel, it is made with make in main.
This one single channel is named in in workerProcess and out in publisher. out is a function argument to publisher which answers question 1.
Question 2: That's the whole purpose of a channel. What you stuff into a channel on its input side comes out at its output side. If a function has reference to (one end of) such a channel it may communicate with someone else having reference to the same channel (its other end). What producer send to the channel is received by workerProcess. This send and receive is done through the special operator <- on Go. A fact you got slightly wrong in your explanation.
out <- data takes data and sends it through the channel named out until <- in reads it from the channel (remember in and out name the same channel from main). That's how workerProcess and publisher communicate.
Question 3 is a duplicate. Your whole program terminates once main is done (in your case after 1 millisecond. Nothing happens after termination of the program. Give the program more time to execute. (The non-printing is unrelated to the channel).
Unbuffered channels block receivers until data is available on the channel. It's not clear to me how this blocking behaves with multiple receivers on the same channel (say when using goroutines). I am sure they would all block as long as there is no data sent on that channel.
But what happens once I send a single value to that channel? Which receiver/goroutine will get the data and therefore unblock? All of them? The first in line? Random?
A single random (non-deterministic) one will receive it.
See the language spec:
Execution of a "select" statement proceeds in several steps:
For all the cases in the statement, the channel operands of receive operations and the channel and right-hand-side expressions of send
statements are evaluated exactly once, in source order, upon entering
the "select" statement. The result is a set of channels to receive
from or send to, and the corresponding values to send. Any side
effects in that evaluation will occur irrespective of which (if any)
communication operation is selected to proceed. Expressions on the
left-hand side of a RecvStmt with a short variable declaration or
assignment are not yet evaluated.
If one or more of the communications can proceed, a single one that can proceed is chosen via a uniform pseudo-random selection.
Otherwise, if there is a default case, that case is chosen. If there
is no default case, the "select" statement blocks until at least one
of the communications can proceed.
Unless the selected case is the default case, the respective communication operation is executed.
If the selected case is a RecvStmt with a short variable declaration or an assignment, the left-hand side expressions are
evaluated and the received value (or values) are assigned.
The statement list of the selected case is executed.
By default the goroutine communication is synchronous and unbuffered: sends do not complete until there is a receiver to accept the value. There must be a receiver ready to receive data from the channel and then the sender can hand it over directly to the receiver.
So channel send/receive operations block until the other side is ready:
1. A send operation on a channel blocks until a receiver is available for the same channel: if there’s no recipient for the value on ch, no other value can be put in the channel. And the other way around: no new value can be sent in ch when the channel is not empty! So the send operation will wait until ch becomes available again.
2. A receive operation for a channel blocks until a sender is available for the same channel: if there is no value in the channel, the receiver blocks.
This is illustrated in the below example:
package main
import "fmt"
func main() {
ch1 := make(chan int)
go pump(ch1) // pump hangs
fmt.Println(<-ch1) // prints only 0
}
func pump(ch chan int) {
for i:= 0; ; i++ {
ch <- i
}
}
Because there is no receiver the goroutine hangs and print only the first number.
To workaround this we need to define a new goroutine which reads from the channel in an infinite loop.
func receive(ch chan int) {
for {
fmt.Println(<- ch)
}
}
Then in main():
func main() {
ch := make(chan int)
go pump(ch)
receive(ch)
}
If the program is allowing multiple goroutines to receive on a single channel then the sender is broadcasting. Each receiver should be equally able to process the data. So it does not matter what mechanism the go runtime uses to decide which of the many goroutine receivers will run Cf. https://github.com/golang/go/issues/247. But only ONE will run for each sent item if the channel is unbuffered.
There have been some discussion about this
But what is established in the Go Memory Model is that it will be at most one of them.
Each send on a particular channel is matched to a corresponding receive from that channel, usually in a different goroutine.
That isn't as clear cut as I would like, but later down they give this example of a semaphore implementation
var limit = make(chan int, 3)
func main() {
for _, w := range work {
go func(w func()) {
limit <- 1
w()
// if it were possible for more than one channel to receive
// from a single send, it would be possible for this to release
// more than one "lock", making it an invalid semaphore
// implementation
<-limit
}(w)
}
select{}
}
I am writing a DNS protocol parser in golang, the idea is to use a map like this
var tidMap map[uint16] (chan []byte)
So for the tidMap map, key is the tid (transaction ID), value is a byte array channel.
The idea is that a goroutine will try get value from the channel, another goroutine will try read bytes by listening every imcoming packet, and once found transaction ID, will set response data to the tidMap, so the former goroutine will continue handle the response.
One problem with the design is that I need the make sure the channel has buffer length of 1, so extra values can be pushed into channel without blocking.
So how can I specify channel buffer length in tidMap declaration?
var tidMap map[int] make(chan int, 1)
You can't use make() there.
The length of the channel buffer doesn't convey type, so you will have to add logic to test if the map entry exists, if it doesn't:
tidMap[0] = make(chan int, 1)
The short answer: you can't. When you make a map, you define the data types of its keys and values, and the capacity of a channel is not part of its type.
The longer answer is: create an abstract data type that hides this implementation detail. Something like this:
type ChannelMap struct {
tidMap map[int](chan []byte)
}
func NewChannelMap() *ChannelMap { ... }
func (c *ChannelMap) Put(tid int) (chan int) {
res := make(chan int, 1)
c.tidMap[tid] = res
return res
}
func (c *ChannelMap) Get(tid int) (chan int) {
return c.tidMap[tid]
}
And just to be sure: giving the channel a capacity of 1 does not ensure that senders will never block; if your channel consumers are too slow, producers can fill the channel up to its capacity and will then block until the channel has room again.
So, right now, I just pass a pointer to a Queue object (implementation doesn't really matter) and call queue.add(result) at the end of goroutines that should add things to the queue.
I need that same sort of functionality—and of course doing a loop checking completion with the comma ok syntax is unacceptable in terms of performance versus the simple queue add function call.
Is there a way to do this better, or not?
There are actually two parts to your question: how does one queue data in Go, and how does one use a channel without blocking.
For the first part, it sounds like what you need to do is instead of using the channel to add things to the queue, use the channel as a queue. For example:
var (
ch = make(chan int) // You can add an int parameter to this make call to create a buffered channel
// Do not buffer these channels!
gFinished = make(chan bool)
processFinished = make(chan bool)
)
func f() {
go g()
for {
// send values over ch here...
}
<-gFinished
close(ch)
}
func g() {
// create more expensive objects...
gFinished <- true
}
func processObjects() {
for val := range ch {
// Process each val here
}
processFinished <- true
}
func main() {
go processObjects()
f()
<-processFinished
}
As for how you can make this more asynchronous, you can (as cthom06 pointed out) pass a second integer to the make call in the second line which will make send operations asynchronous until the channel's buffer is full.
EDIT: However (as cthom06 also pointed out), because you have two goroutines writing to the channel, one of them has to be responsible for closing the channel. Also, my previous revision would exit before processObjects could complete. The way I chose to synchronize the goroutines is by creating a couple more channels that pass around dummy values to ensure that the cleanup gets finished properly. Those channels are specifically unbuffered so that the sends happen in lock-step.