How can I make a channel 2-way (I don't know if this is right term) in the same function. If I have the following code, then:
func server (a <-chan string) {
data:= <-a
// now is there a way I can send data through the same channel
// data <- "yet another string"
}
Is there anyother way of implementing this ? Appreciate any help.
As commented by #Warrior:
In the code referred above, the directional pointer with channel restricts the function to do any other activity on that channel instead of the one that is allowed. So, making it:
func server (a chan string) {
instead of
func server (a <-chan string) {
will allow the function to send as well as receive data through the same channel.
Related
There is queue of not important structs Message, which has the classic push and pop methods:
type Queue struct {
messages list.List
}
//The implementation is not relevant for the sake of the question
func (q *Queue) Push(msg Message) { /*...*/ }
func (q *Queue) Pop() (Message, bool) { /*...*/ }
/*
* NewTimedChannel runs a goroutine which pops a message from the queue every
* given time duration and sends it over the returned channel
*/
func (q *Queue) NewTimedChannel(t time.Duration) (<-chan Message) {/*...*/}
The client of the Push function will be a web gui in which users will post their messages.
The client of the channel returned by NewTimedChannel will be a service which sends each message to a not relevant endpoint over the network.
I'm a newbie in concurrency and go and I have the following question:
I know that since Queue.messages is a shared state between the main goroutine which deals with pushing the message after the user submit a web form and the ones created for each NewTimedChannel invocation, I need to lock it.
Do I need to lock and unlock using the sync.Mutex in all the Push, Pop and NewTimedChannel methods?
And is there a more idiomatic way to handle this specific problem in the go environment?
As others have pointed out, it requires synchronization or there will be a data race.
There is a saying in Go, "Don't communicate by sharing memory, share memory by communicating." As in this case, I think an idomatic way is to make channels send to a seprate goroutine which synchronize all the operations together using select. The code can easily be extended by adding more channels to support more kinds of operations (like the timed channel in your code which I don't fully understand what does it do), and by using select and other utils, it can easily handle more complex synchronizing by using locks. I write some sample code:
type SyncQueue struct {
Q AbsQueue
pushCh,popMsgCh chan Message
popOkCh chan bool
popCh chan struct{}
}
// An abstract of the Queue type. You can remove the abstract layer.
type AbsQueue interface {
Push(Message)
Pop() (Message,bool)
}
func (sq SyncQueue) Push(m Message) {
sq.pushCh <- m
}
func (sq SyncQueue) Pop() (Message,bool) {
sq.popCh <- struct{}{} // send a signal for pop. struct{}{} cost no memory at all.
return <-sq.popMsgCh,<-sq.popOkCh
}
// Every pop and push get synchronized here.
func (sq SyncQueue) Run() {
for {
select {
case m:=<-pushCh:
Q.Push(m)
case <-popCh:
m,ok := Q.Pop()
sq.popMsgCh <- m
sq.popOkCh <- ok
}
}
}
func NewSyncQueue(Q AbsQueue) *SyncQueue {
sq:=SyncQueue {
Q:Q,
pushCh: make(chan Message),popMsgCh: make(chan Message),
pushOkCh: make(chan bool), popCh: make(chan struct{}),
}
go sq.Run()
return &sq
}
Note that for simpilicity, I did not use a quit channel or a context.Context, so the goroutine of sq.Run() has no way of exiting and would cause a memory leak.
Do I need to lock and unlock using the sync.Mutex in all the Push, Pop and NewTimedChannel methods?
Yes.
And is there a more idiomatic way to handle this specific problem in
the go environment?
For insight, have a look at the last answer for this question:
How do I (succinctly) remove the first element from a slice in Go?
I'm writing a package to control a Canon DSLR using their EDSDK DLL from Go.
This is a personal project for a photo booth to use at our wedding at my partners request, which I'll be happy to post on GitHub when complete :).
Looking at the examples of using the SDK elsewhere, it isn't threadsafe and uses thread-local resources, so I'll need to make sure I'm calling it from a single thread during usage. While not ideal, it looks like Go provides a "runtime.LockOSThread" function for doing just that, although this does get called by the core DLL interop code itself, so I'll have to wait and find out if that interferes or not.
I want the rest of the application to be able to call the SDK using a higher level interface without worrying about the threading, so I need a way to pass function call requests to the locked thread/Goroutine to execute there, then pass the results back to the calling function outside of that Goroutine.
So far, I've come up with this working example of using very broad function definitions using []interface{} arrays and passing back and forward via channels. This would take a lot of mangling of input/output data on every call to do type assertions back out of the interface{} array, even if we know what we should expect for each function ahead of time, but it looks like it'll work.
Before I invest a lot of time doing it this way for possibly the worst way to do it - does anyone have any better options?
package edsdk
import (
"fmt"
"runtime"
)
type CanonSDK struct {
FChan chan functionCall
}
type functionCall struct {
Function func([]interface{}) []interface{}
Arguments []interface{}
Return chan []interface{}
}
func NewCanonSDK() (*CanonSDK, error) {
c := &CanonSDK {
FChan: make(chan functionCall),
}
go c.BackgroundThread(c.FChan)
return c, nil
}
func (c *CanonSDK) BackgroundThread(fcalls <-chan functionCall) {
runtime.LockOSThread()
for f := range fcalls {
f.Return <- f.Function(f.Arguments)
}
runtime.UnlockOSThread()
}
func (c *CanonSDK) TestCall() {
ret := make(chan []interface{})
f := functionCall {
Function: c.DoTestCall,
Arguments: []interface{}{},
Return: ret,
}
c.FChan <- f
results := <- ret
close(ret)
fmt.Printf("%#v", results)
}
func (c *CanonSDK) DoTestCall([]interface{}) []interface{} {
return []interface{}{ "Test", nil }
}
For similar embedded projects I've played with, I tend to create a single goroutine worker that listens on a channel to perform all the work over that USB device. And any results sent back out on another channel.
Talk to the device with channels only in Go in a one-way exchange. LIsten for responses from the other channel.
Since USB is serial and polling, I had to setup a dedicated channel with another goroutine that justs picks items off the channel when they were pushed into it from the worker goroutine that just looped.
I have a struct called Hub with a Run() method which is executed in its own goroutine. This method sequentially handles incoming messages. Messages arrive concurrently from multiple producers (separate goroutines). Of course I use a channel to accomplish this task. But now I want to hide the Hub behind an interface to be able to choose from its implementations. So, using a channel as a simple Hub's field isn't appropriate.
package main
import "fmt"
import "time"
type Hub struct {
msgs chan string
}
func (h *Hub) Run() {
for {
msg, hasMore := <- h.msgs
if !hasMore {
return
}
fmt.Println("hub: msg received", msg)
}
}
func (h *Hub) SendMsg(msg string) {
h.msgs <- msg
}
func send(h *Hub, prefix string) {
for i := 0; i < 5; i++ {
fmt.Println("main: sending msg")
h.SendMsg(fmt.Sprintf("%s %d", prefix, i))
}
}
func main() {
h := &Hub{make(chan string)}
go h.Run()
for i := 0; i < 10; i++ {
go send(h, fmt.Sprintf("msg sender #%d", i))
}
time.Sleep(time.Second)
}
So I've introduced Hub.SendMsg(msg string) function that just calls h.msgs <- msg and which I can add to the HubInterface. And as a Go-newbie I wonder, is it safe from the concurrency perspective? And if so - is it a common approach in Go?
Playground here.
Channel send semantics do not change when you move the send into a method. Andrew's answer points out that the channel needs to be created with make to send successfully, but that was always true, whether or not the send is inside a method.
If you are concerned about making sure callers can't accidentally wind up with invalid Hub instances with a nil channel, one approach is to make the struct type private (hub) and have a NewHub() function that returns a fully initialized hub wrapped in your interface type. Since the struct is private, code in other packages can't try to initialize it with an incomplete struct literal (or any struct literal).
That said, it's often possible to create invalid or nonsense values in Go and that's accepted: net.IP("HELLO THERE BOB") is valid syntax, or net.IP{}. So if you think it's better to expose your Hub type go ahead.
Easy answer
Yes
Better answer
No
Channels are great for emitting data from unknown go-routines. They do so safely, however I would recommend being careful with a few parts. In the listed example the channel is created with the construction of the struct by the consumer (and not not by a consumer).
Say the consumer creates the Hub like the following: &Hub{}. Perfectly valid... Apart from the fact that all the invokes of SendMsg() will block for forever. Luckily you placed those in their own go-routines. So you're still fine right? Wrong. You are now leaking go-routines. Seems fine... until you run this for a period of time. Go encourages you to have valid zero values. In this case &Hub{} is not valid.
Ensuring SendMsg() won't block could be achieved via a select{} however you then have to decide what to do when you encounter the default case (e.g. throw data away). The channel could block for more reasons than bad setup too. Say later you do more than simply print the data after reading from the channel. What if the read gets very slow, or blocks on IO. You then will start pushing back on the producers.
Ultimately, channels allow you to not think much about concurrency... However if this is something of high-throughput, then you have quite a bit to consider. If it is production code, then you need to understand that your API here involves SendMsg() blocking.
My program has a pipeline structure, and I just implemented a caching filter that sends stuff directly to output if the already processed version of data is in the cache.
func Run(in chan downloader.ReadyDownload) chan CCFile {
out := make(chan CCFile)
processQueue := make(chan downloader.ReadyDownload)
go cache.BypassFilter(in, processQueue, out)
// writes the cached, already processed version to out if it exists
// otherwise redirects the input to processQueue
go process(processQueue, out)
return out
}
The problem is that my program has multiple places like this, and many kind of structs (like ReadyDownload and CCFile in this snippet) are being passed through the channels. They all implement this interface
type ProcessItem interface {
Source() string
Target() string
Key() string
}
so my BypassFilter() function signature looks like this:
func (c Cache) BypassFilter(in chan ProcessItem, process chan ProcessItem, bypass chan ProcessItem)
But this brings about the following error:
cannot use in (type chan downloader.ReadyDownload) as type chan utils.ProcessItem in function argument
Although ReadyDownload certainly implements ProcessItem. For example, this works without problems:
foo := downloader.ReadyDownload{}
var bar utils.ProcessItem
bar = foo
So, my (yet) very limited understanding of Go types and interfaces brings me to ask this question: Is it the fact that they are channels of something and something else, that makes the types incompatible? What should I do to make it work? Let's say that I've got a channel of ReadyDownloads. Is the only way to forward the data to a function that takes, let's say channel of interface{}s as a parameter, to create a new channel of interface{}s, pass that to the function and read stuff from the channel of ReadyDownloads and feed them to the other channel?
These two are different types:
processQueue chan ReadyDownload
process chan ProcessItem
You can put a ReadyDownloader value in a channel of type chan ProcessItem (if it implements the interface), but you cannot convert one channel type to another, in the same way that you cannot convert a []T slice into a []interface{} slice, another common confusion similar to this one.
What you need to do is make all the channels of type chan ProcessItem:
func Run(in chan ProcessItem) chan CCFile {
out := make(chan CCFile)
processQueue := make(chan ProcessItem)
go cache.BypassFilter(in, processQueue, out)
// writes the cached, already processed version to out if it exists
// otherwise redirects the input to processQueue
go process(processQueue, out)
return out
}
To read more about why this is (for slices, but the same applies for channels), you can read the following go-wiki page:
http://code.google.com/p/go-wiki/wiki/InterfaceSlice
Changing every channels to struct channels might work here, but in general, you might want to treat your struct type as interfaces for processing down the road. Fortunately, go gives us many solutions. Here is one.
Consider this very simple set up, where we want to use a Object struct type as several interfaces:
// Get the objects
func ParseFile(fileName string, co chan Object) {
for _, object := range DoStuff(fileName) {
co <- object
}
}
// Use some saving functionality that is defined elsewhere as:
func Archive(cs chan Saveable) {
for saveable := range cs {
saveable.Save()
}
}
type Saveable interface {
Save()
}
//Implement the interfaces...
func (*Object) Save() {
fmt.Println("Naa, I'm lazy")
}
// Or some throwing functionality?
func ThrowOnTheWall(ct chan Throwable) {
for throwable := range cs {
throwable.Throw()
}
}
//...
co := make(chan Object)
go ParseFile("file.xml", co)
Archive(co) // Will NOT work, co is of the wrong type.
Here, using everywhere some chan Object is not suitable, because you might want to throw on the wall something different than an object (e.g., type Defecation struct {...} that you would implement as a Throwable too.).
You could use a go routine to do the casting in the background:
func ObjectToSaveable(from chan Object) chan Saveable {
to := make(chan Saveable)
go func() {
for object := range from {
to <- &object
}
close(to)
}()
return to
}
And then use it to encapsulate the initial channel:
co := make(chan Object)
go ParseFile("file.xml", co)
Archive(ObjectToSaveable(co))
I am writing a DNS protocol parser in golang, the idea is to use a map like this
var tidMap map[uint16] (chan []byte)
So for the tidMap map, key is the tid (transaction ID), value is a byte array channel.
The idea is that a goroutine will try get value from the channel, another goroutine will try read bytes by listening every imcoming packet, and once found transaction ID, will set response data to the tidMap, so the former goroutine will continue handle the response.
One problem with the design is that I need the make sure the channel has buffer length of 1, so extra values can be pushed into channel without blocking.
So how can I specify channel buffer length in tidMap declaration?
var tidMap map[int] make(chan int, 1)
You can't use make() there.
The length of the channel buffer doesn't convey type, so you will have to add logic to test if the map entry exists, if it doesn't:
tidMap[0] = make(chan int, 1)
The short answer: you can't. When you make a map, you define the data types of its keys and values, and the capacity of a channel is not part of its type.
The longer answer is: create an abstract data type that hides this implementation detail. Something like this:
type ChannelMap struct {
tidMap map[int](chan []byte)
}
func NewChannelMap() *ChannelMap { ... }
func (c *ChannelMap) Put(tid int) (chan int) {
res := make(chan int, 1)
c.tidMap[tid] = res
return res
}
func (c *ChannelMap) Get(tid int) (chan int) {
return c.tidMap[tid]
}
And just to be sure: giving the channel a capacity of 1 does not ensure that senders will never block; if your channel consumers are too slow, producers can fill the channel up to its capacity and will then block until the channel has room again.