I am writing a DNS protocol parser in golang, the idea is to use a map like this
var tidMap map[uint16] (chan []byte)
So for the tidMap map, key is the tid (transaction ID), value is a byte array channel.
The idea is that a goroutine will try get value from the channel, another goroutine will try read bytes by listening every imcoming packet, and once found transaction ID, will set response data to the tidMap, so the former goroutine will continue handle the response.
One problem with the design is that I need the make sure the channel has buffer length of 1, so extra values can be pushed into channel without blocking.
So how can I specify channel buffer length in tidMap declaration?
var tidMap map[int] make(chan int, 1)
You can't use make() there.
The length of the channel buffer doesn't convey type, so you will have to add logic to test if the map entry exists, if it doesn't:
tidMap[0] = make(chan int, 1)
The short answer: you can't. When you make a map, you define the data types of its keys and values, and the capacity of a channel is not part of its type.
The longer answer is: create an abstract data type that hides this implementation detail. Something like this:
type ChannelMap struct {
tidMap map[int](chan []byte)
}
func NewChannelMap() *ChannelMap { ... }
func (c *ChannelMap) Put(tid int) (chan int) {
res := make(chan int, 1)
c.tidMap[tid] = res
return res
}
func (c *ChannelMap) Get(tid int) (chan int) {
return c.tidMap[tid]
}
And just to be sure: giving the channel a capacity of 1 does not ensure that senders will never block; if your channel consumers are too slow, producers can fill the channel up to its capacity and will then block until the channel has room again.
Related
I'm new to Golang, and may I resize the buffered channel like:
var xxx // just declaration.
...
xxx = make(chan string, mysize) // I know `xxx := make(chan string, mysize)` work, but I don't want it.
...
xxx <- "123" // using the buffered channel.
Once a channel is created, its capacity (buffer size) cannot be changed. You can only create and assign a new channel with different capacity.
Note that just declaring a variable of channel type does not create and initialize a channel, it will be its zero value which is nil for channel types:
var xxx chan int
Here xxx is a variable of chan int, but it's not yet initialized, you cannot send any values on it. You have to create one with the builtin make() function, e.g.:
var xxx chan int = make(chan int, 10)
Or simply:
var xxx = make(chan int, 10)
Its capactiy will be 10 and it cannot be changed. You can however assign a new channel value to it, e.g.:
xxx = make(chan int, 20)
But know that if multiple goroutines access and use the xxx variable, the above operation may cause a data race, and changing the value of xxx should be synchronized with other goroutines that read it.
Also note that if goroutines do not access xxx but the channel is "passed" to them, then assigning a new channel value to xxx will not affect them, those goroutines will not know about the new channel and will continue to use the old, passed channel.
There is queue of not important structs Message, which has the classic push and pop methods:
type Queue struct {
messages list.List
}
//The implementation is not relevant for the sake of the question
func (q *Queue) Push(msg Message) { /*...*/ }
func (q *Queue) Pop() (Message, bool) { /*...*/ }
/*
* NewTimedChannel runs a goroutine which pops a message from the queue every
* given time duration and sends it over the returned channel
*/
func (q *Queue) NewTimedChannel(t time.Duration) (<-chan Message) {/*...*/}
The client of the Push function will be a web gui in which users will post their messages.
The client of the channel returned by NewTimedChannel will be a service which sends each message to a not relevant endpoint over the network.
I'm a newbie in concurrency and go and I have the following question:
I know that since Queue.messages is a shared state between the main goroutine which deals with pushing the message after the user submit a web form and the ones created for each NewTimedChannel invocation, I need to lock it.
Do I need to lock and unlock using the sync.Mutex in all the Push, Pop and NewTimedChannel methods?
And is there a more idiomatic way to handle this specific problem in the go environment?
As others have pointed out, it requires synchronization or there will be a data race.
There is a saying in Go, "Don't communicate by sharing memory, share memory by communicating." As in this case, I think an idomatic way is to make channels send to a seprate goroutine which synchronize all the operations together using select. The code can easily be extended by adding more channels to support more kinds of operations (like the timed channel in your code which I don't fully understand what does it do), and by using select and other utils, it can easily handle more complex synchronizing by using locks. I write some sample code:
type SyncQueue struct {
Q AbsQueue
pushCh,popMsgCh chan Message
popOkCh chan bool
popCh chan struct{}
}
// An abstract of the Queue type. You can remove the abstract layer.
type AbsQueue interface {
Push(Message)
Pop() (Message,bool)
}
func (sq SyncQueue) Push(m Message) {
sq.pushCh <- m
}
func (sq SyncQueue) Pop() (Message,bool) {
sq.popCh <- struct{}{} // send a signal for pop. struct{}{} cost no memory at all.
return <-sq.popMsgCh,<-sq.popOkCh
}
// Every pop and push get synchronized here.
func (sq SyncQueue) Run() {
for {
select {
case m:=<-pushCh:
Q.Push(m)
case <-popCh:
m,ok := Q.Pop()
sq.popMsgCh <- m
sq.popOkCh <- ok
}
}
}
func NewSyncQueue(Q AbsQueue) *SyncQueue {
sq:=SyncQueue {
Q:Q,
pushCh: make(chan Message),popMsgCh: make(chan Message),
pushOkCh: make(chan bool), popCh: make(chan struct{}),
}
go sq.Run()
return &sq
}
Note that for simpilicity, I did not use a quit channel or a context.Context, so the goroutine of sq.Run() has no way of exiting and would cause a memory leak.
Do I need to lock and unlock using the sync.Mutex in all the Push, Pop and NewTimedChannel methods?
Yes.
And is there a more idiomatic way to handle this specific problem in
the go environment?
For insight, have a look at the last answer for this question:
How do I (succinctly) remove the first element from a slice in Go?
Below is an example of how to use mutex lock in order to safely access data. How would I go about doing the same with the use of CSP (communication sequential processes) instead of using mutex lock’s and unlock’s?
type Stack struct {
top *Element
size int
sync.Mutex
}
func (ss *Stack) Len() int {
ss.Lock()
size := ss.size
ss.Unlock()
return size
}
func (ss *Stack) Push(value interface{}) {
ss.Lock()
ss.top = &Element{value, ss.top}
ss.size++
ss.Unlock()
}
func (ss *SafeStack) Pop() (value interface{}) {
ss.Lock()
size := ss.size
ss.Unlock()
if size > 0 {
ss.Lock()
value, ss.top = ss.top.value, ss.top.next
ss.size--
ss.Unlock()
return
}
return nil
}
If you actually were to look at how Go implements channels, you'd essentially see a mutex around an array with some additional thread handling to block execution until the value is passed through. A channel's job is to move data from one spot in memory to another with ease. Therefore where you have locks and unlocks, you'd have things like this example:
func example() {
resChan := make(int chan)
go func(){
resChan <- 1
}()
go func(){
res := <-resChan
}
}
So in the example, the first goroutine is blocked after sending the value until the second goroutine reads from the channel.
To do this in Go with mutexes, one would use sync.WaitGroup which will add one to the group on setting the value, then release it from the group and the second goroutine will lock and then unlock the value.
The oddities in your example are 1 no goroutines, so it's all happening in a single main goroutine and the locks are being used more traditionally (as in c thread like) so channels won't really accomplish anything. The example you have would be considered an anti-pattern, like the golang proverb says "Don't communicate by sharing memory, share memory by communicating."
How can I make a channel 2-way (I don't know if this is right term) in the same function. If I have the following code, then:
func server (a <-chan string) {
data:= <-a
// now is there a way I can send data through the same channel
// data <- "yet another string"
}
Is there anyother way of implementing this ? Appreciate any help.
As commented by #Warrior:
In the code referred above, the directional pointer with channel restricts the function to do any other activity on that channel instead of the one that is allowed. So, making it:
func server (a chan string) {
instead of
func server (a <-chan string) {
will allow the function to send as well as receive data through the same channel.
My program has a pipeline structure, and I just implemented a caching filter that sends stuff directly to output if the already processed version of data is in the cache.
func Run(in chan downloader.ReadyDownload) chan CCFile {
out := make(chan CCFile)
processQueue := make(chan downloader.ReadyDownload)
go cache.BypassFilter(in, processQueue, out)
// writes the cached, already processed version to out if it exists
// otherwise redirects the input to processQueue
go process(processQueue, out)
return out
}
The problem is that my program has multiple places like this, and many kind of structs (like ReadyDownload and CCFile in this snippet) are being passed through the channels. They all implement this interface
type ProcessItem interface {
Source() string
Target() string
Key() string
}
so my BypassFilter() function signature looks like this:
func (c Cache) BypassFilter(in chan ProcessItem, process chan ProcessItem, bypass chan ProcessItem)
But this brings about the following error:
cannot use in (type chan downloader.ReadyDownload) as type chan utils.ProcessItem in function argument
Although ReadyDownload certainly implements ProcessItem. For example, this works without problems:
foo := downloader.ReadyDownload{}
var bar utils.ProcessItem
bar = foo
So, my (yet) very limited understanding of Go types and interfaces brings me to ask this question: Is it the fact that they are channels of something and something else, that makes the types incompatible? What should I do to make it work? Let's say that I've got a channel of ReadyDownloads. Is the only way to forward the data to a function that takes, let's say channel of interface{}s as a parameter, to create a new channel of interface{}s, pass that to the function and read stuff from the channel of ReadyDownloads and feed them to the other channel?
These two are different types:
processQueue chan ReadyDownload
process chan ProcessItem
You can put a ReadyDownloader value in a channel of type chan ProcessItem (if it implements the interface), but you cannot convert one channel type to another, in the same way that you cannot convert a []T slice into a []interface{} slice, another common confusion similar to this one.
What you need to do is make all the channels of type chan ProcessItem:
func Run(in chan ProcessItem) chan CCFile {
out := make(chan CCFile)
processQueue := make(chan ProcessItem)
go cache.BypassFilter(in, processQueue, out)
// writes the cached, already processed version to out if it exists
// otherwise redirects the input to processQueue
go process(processQueue, out)
return out
}
To read more about why this is (for slices, but the same applies for channels), you can read the following go-wiki page:
http://code.google.com/p/go-wiki/wiki/InterfaceSlice
Changing every channels to struct channels might work here, but in general, you might want to treat your struct type as interfaces for processing down the road. Fortunately, go gives us many solutions. Here is one.
Consider this very simple set up, where we want to use a Object struct type as several interfaces:
// Get the objects
func ParseFile(fileName string, co chan Object) {
for _, object := range DoStuff(fileName) {
co <- object
}
}
// Use some saving functionality that is defined elsewhere as:
func Archive(cs chan Saveable) {
for saveable := range cs {
saveable.Save()
}
}
type Saveable interface {
Save()
}
//Implement the interfaces...
func (*Object) Save() {
fmt.Println("Naa, I'm lazy")
}
// Or some throwing functionality?
func ThrowOnTheWall(ct chan Throwable) {
for throwable := range cs {
throwable.Throw()
}
}
//...
co := make(chan Object)
go ParseFile("file.xml", co)
Archive(co) // Will NOT work, co is of the wrong type.
Here, using everywhere some chan Object is not suitable, because you might want to throw on the wall something different than an object (e.g., type Defecation struct {...} that you would implement as a Throwable too.).
You could use a go routine to do the casting in the background:
func ObjectToSaveable(from chan Object) chan Saveable {
to := make(chan Saveable)
go func() {
for object := range from {
to <- &object
}
close(to)
}()
return to
}
And then use it to encapsulate the initial channel:
co := make(chan Object)
go ParseFile("file.xml", co)
Archive(ObjectToSaveable(co))