I'd like some feedback on the implementation details of what I'm trying to build. What I want to achieve is have multiple functions with different signatures that can be called concurrently.
Calling the functions in coroutines sequentially works fine, but I'm wondering if there's a way to do this in a more idiomatic way, e.g. iterate over a slice of functions.
Since each function has different arguments and return values though, I have trouble figuring out what the best approach would be. An example that is a bit similar to my goal can be seen here: Golang - How do you create a slice of functions with different signatures?, but there the code just calls the functions and doesn't account for any return values.
Is what I have in mind even possible?
You can use code from linked question and just wrap the v.Call(params) into an anonymous function executing in its own goroutine like this:
...
// WaitGroup to wait on goroutines to finish their execution
var wg sync.WaitGroup
for a, v := range f {
v := reflect.TypeOf(v)
//calling the function from reflect
val := reflect.ValueOf(f[a])
params := make([]reflect.Value, v.NumIn())
if v.NumIn() == 1 {
params[0] = reflect.ValueOf(1564)
} else if v.NumIn() == 2 {
params[0] = reflect.ValueOf("Test FROM reflect")
params[1] = reflect.ValueOf(float32(123456))
}
// Run them in parallel
wg.Add(1)
go func() {
defer wg.Done()
val.Call(params)
}()
}
wg.Wait()
See it on Go Playground
As for return values Value.Call() returns []Value which is slice of return values - so you are covered here too. Your question doesn't specify what you intend to do with results but given they will be generated in parallel you'll probably need to send them through a channel(s) - you can do that in anonymous function (after processing return slice) too.
go func() { MyPackage.MyFunc(with, whatsoever, signature); }() - roughtly, that's what you need. You span as many goroutines (using the go keyword) as there are concurrent functions.
There is no notion of "returned value" from goroutine. For that you have to use channels. They are primary communication mechanism. So, you span a new goroutine with some function f of arbitrary signature and when it's done and you got some result, you send it to some channel shared between goroutines for communication.
Channels are thread-safe and were carefully designed to handle such a communication gracefully. Go, as programming language, provides few keywords that deal with reading/writing to/from channels. So there are pretty fundamental to (concurrent) programming in Go.
However, of course, you can handle it differently. Sharing some mutable memory protected by some kind of locking, or relying upon lockless compareAndSet fashion. Arguably, that's less idiomatic way and generally have to be avoided. Always prefer channels.
Related
I am trying to learn goLang by studying different examples online. in this one example it is a quiz test where a CSV file is given to the user with questions and answers, the timer fires if the user does not answer the Q in a given time. What I don’t understand in the code below why are we creating the answer channel in for loop each time for every different question. why can’t we define the timer outside the for loop and use that for every question isn’t that inefficient coding?
problemloop:
for i, p := range problems {
fmt.Printf("Problem #%d: %s = ", i+1, p.q)
answerCh := make(chan string)
go func() {
var answer string
fmt.Scanf("%s\n", &answer)
answerCh <- answer
}()
select {
case <-timer.C:
fmt.Println()
break problemloop
case answer := <-answerCh:
if answer == p.a {
correct++
}
}
}
In Go, creating channels is very cheap. It's a common idiom, therefore, to create an "answer channel" you pass to a goroutine. Goroutines can't just return a value to a caller the way a function can do. When the goroutine is done, it sends its answer/result to the channel. Receiving on this channel in the main (or some other consumer) goroutine serves as a sync point. And allows to do timeouts, like your example demonstrates.
I wouldn't worry about efficiency here unless you can prove with profiling that this is the hot path. It's likely that this code could have been written with a single channel, but it's hard to say looking at the small snippet you provided.
I encounter a situation that I can not understand. In my code, I use functions have the need to read a map (but not write, only loop through a snapshot of existing datas in this map). There is my code :
type MyStruct struct {
*sync.RWMutex
MyMap map[int]MyDatas
}
var MapVar = MyStruct{ &sync.RWMutex{}, make(map[int]MyDatas) }
func MyFunc() {
MapVar.Lock()
MapSnapshot := MapVar.MyMap
MapVar.Unlock()
for _, a := range MapSnapshot { // Map concurrent write/read occur here
//Some stuff
}
}
main() {
go MyFunc()
}
The function "MyFunc" is run in a go routine, only once, there is no multiple runs of this func. Many other functions are accessing to the same "MapVar" with the same method and it randomly produce a "map concurrent write/read". I hope someone will explain to me why my code is wrong.
Thank you for your time.
edit: To clarify, I am just asking why my range MapSnapshot produce a concurrent map write/read. I cant understand how this map can be concurrently used since I save the real global var (MapVar) in a local var (MapSnapshot) using a sync mutex.
edit: Solved. To copy the content of a map in a new variable without using the same reference (and so to avoid map concurrent read/write), I must loop through it and write each index and content to a new map with a for loop.
Thanks xpare and nilsocket.
there is no multiple runs of this func. Many other functions are accessing to the same "MapVar" with the same method and it randomly produce a "map concurrent write/read"
When you pass the value of MapVar.MyMap to MapSnapshot, the Map concurrent write/read will never be occur, because the operation is wrapped with mutex.
But on the loop, the error could happen since practically reading process is happening during loop. So better to wrap the loop with mutex as well.
MapVar.Lock() // lock begin
MapSnapshot := MapVar.MyMap
for _, a := range MapSnapshot {
// Map concurrent write/read occur here
// Some stuff
}
MapVar.Unlock() // lock end
UPDATE 1
Here is my response to your argument below:
This for loop takes a lot of time, there is many stuff in this loop, so locking will slow down other routines
As per your statement The function "MyFunc" is run in a go routine, only once, there is no multiple runs of this func, then I think making the MyFunc to be executed as goroutine is not a good choice.
And to increase the performance, better to make the process inside the loop to be executed in a goroutine.
func MyFunc() {
for _, a := range MapVar.MyMap {
go func(a MyDatas) {
// do stuff here
}(a)
}
}
main() {
MyFunc() // remove the go keyword
}
UPDATE 2
If you really want to copy the MapVar.MyMap into another object, passing it to another variable will not solve that (map is different type compared to int, float32 or other primitive type).
Please refer to this thread How to copy a map?
I am developing a simple go library for jsonrpc over http.
There is the following method:
rpcClient.Call("myMethod", myParam1, myParam2)
This method internally does a http.Get() and returns the result or an error (tuple).
This is of course synchron for the caller and returns when the Get() call returns.
Is this the way to provide libraries in go? Should I leave it to the user of my library to make it asynchron if she wants to?
Or should I provide a second function called:
rpcClient.CallAsync()
and return a channel here? Because channels cannot provide tuples I have to pack the (response, error) tuple in a struct and return that struct instead.
Does this make sense?
Otherwise the user would have to wrap every call in an ugly method like:
result := make(chan AsyncResponse)
go func() {
res, err := rpcClient.Call("myMethod", myParam1, myParam2)
result <- AsyncResponse{res, err}
}()
Is there a best practice for go libraries and asynchrony?
The whole point of go's execution model is to hide the asynchronous operations from the developer, and behave like a threaded model with blocking operations. Behind the scenes there are green-threads and asynchronous IO and a very sophisticated scheduler.
So no, you shouldn't provide an async API to your library. Networking in go is done in a pseudo-blocking way from the code's perspective, and you open as many goroutines as needed, as they are very cheap.
So your last example is the way to go, and I don't consider it ugly. Because it allows the developer to choose the concurrency model. In the context of an http server, where each command is handled in separate goroutine, I'd just call rpcClient.Call("myMethod", myParam1, myParam2).
Or if I want a fanout - I'll create fanout logic.
You can also create a convenience function for executing the call and returning on a channel:
func CallAsync(method, p1, p2) chan AsyncResponse {
result := make(chan AsyncResponse)
go func() {
res, err := rpcClient.Call(method, p1, p2)
result <- AsyncResponse{res, err}
}()
return result
}
The below code gives compilation error saying 'unexpected go':
x := go doSomething(arg)
func doSomething(arg int) int{
...
return my_int_value
}
I know, I can fetch the return value if I call the function normally i.e. without using goroutine or I can use channels etc.
My question is why is it not possible to fetch a return value like this from a goroutine.
Why is it not possible to fetch a return value from a goroutine assigning it to a variable?
Run goroutine (asynchronously) and fetch return value from function are essentially contradictory actions. When you say go you mean "do it asynchronously" or even simpler: "Go on! Don't wait for the function execution be finished". But when you assign function return value to a variable you are expecting to have this value within the variable. So when you do that x := go doSomething(arg) you are saying: "Go on, don't wait for the function! Wait-wait-wait! I need a returned value be accessible in x var right in the next line below!"
Channels
The most natural way to fetch a value from a goroutine is channels. Channels are the pipes that connect concurrent goroutines. You can send values into channels from one goroutine and receive those values into another goroutine or in a synchronous function. You could easily obtain a value from a goroutine not breaking concurrency using select:
func main() {
c1 := make(chan string)
c2 := make(chan string)
go func() {
time.Sleep(time.Second * 1)
c1 <- "one"
}()
go func() {
time.Sleep(time.Second * 2)
c2 <- "two"
}()
for i := 0; i < 2; i++ {
// Await both of these values
// simultaneously, printing each one as it arrives.
select {
case msg1 := <-c1:
fmt.Println("received", msg1)
case msg2 := <-c2:
fmt.Println("received", msg2)
}
}
}
The example is taken from Go By Example
CSP & message-passing
Go is largerly based on CSP theory. The naive description from above could be precisely outlined in terms of CSP (although I believe it is out of scope of the question). I strongly recommend to familiarize yourself with CSP theory at least because it is RAD. These short quotations give a direction of thinking:
As its name suggests, CSP allows the description of systems in terms of component processes that operate independently, and interact with each other solely through message-passing communication.
In computer science, message passing sends a message to a process and relies on the process and the supporting infrastructure to select and invoke the actual code to run. Message passing differs from conventional programming where a process, subroutine, or function is directly invoked by name.
The strict answer is that you can do that. It's just probably not a good idea. Here's code that would do that:
var x int
go func() {
x = doSomething()
}()
This will spawn off a new goroutine which will calculate doSomething() and then assign the result to x. The problem is: how are you going to use x from the original goroutine? You probably want to make sure the spawned goroutine is done with it so that you don't have a race condition. But if you want to do that, you'll need a way to communicate with the goroutine, and if you've got a way to do that, why not just use it to send the value back?
The idea of the go keyword is that you run the doSomething function asynchronously, and continue the current goroutine without waiting for the result, kind of like executing a command in a Bash shell with an '&' after it. If you want to do
x := doSomething(arg)
// Now do something with x
then you need the current goroutine to block until doSomething finishes. So why not just call doSomething in the current goroutine? There are other options (like, doSomething could post a result to a channel, which the current goroutine receives values from) but simply calling doSomething and assigning the result to a variable is obviously simpler.
It's a design choice by Go creators. There's a whole lot of abstractions/APIs to represent the value of async I/O operations - promise, future, async/await, callback, observable, etc. These abstractions/APIs are inherently tied to the unit of scheduling - coroutines - and these abstractions/APIs dictate how coroutines (or more precisely the return value of async I/O represented by them) can be composed.
Go chose message passing (aka channels) as the abstraction/API to represent the return value of async I/O operations. And of course, goroutines and channels give you a composable tool to implement async I/O operations.
Why not use a channel to write into?
chanRes := make(chan int, 1)
go doSomething(arg, chanRes)
//blocks here or you can use some other sync mechanism (do something else) and wait
x := <- chanRes
func doSomething(arg int, out chan<- int){
...
out <- my_int_value
}
All the examples I've seen so far involve blocking to get the result (via the <-chan operator).
My current approach involves passing a pointer to a struct:
type goresult struct {
result resultType;
finished bool;
}
which the goroutine writes upon completion. Then it's a simple matter of checking finished whenever convenient. Do you have better alternatives?
What I'm really aiming for is a Qt-style signal-slot system. I have a hunch the solution will look almost trivial (chans have lots of unexplored potential), but I'm not yet familiar enough with the language to figure it out.
You can use the "comma, ok" pattern (see their page on "effective go"):
foo := <- ch; // This blocks.
foo, ok := <- ch; // This returns immediately.
Select statements allows you to check multiple channels at once, taking a random branch (of the ones where communication is waiting):
func main () {
for {
select {
case w := <- workchan:
go do_work(w)
case <- signalchan:
return
// default works here if no communication is available
default:
// do idle work
}
}
}
For all the send and receive
expressions in the "select" statement,
the channel expressions are evaluated,
along with any expressions that appear
on the right hand side of send
expressions, in top-to-bottom order.
If any of the resulting operations can
proceed, one is chosen and the
corresponding communication and
statements are evaluated. Otherwise,
if there is a default case, that
executes; if not, the statement blocks
until one of the communications can
complete.
You can also peek at the channel buffer to see if it contains anything by using len:
if len(channel) > 0 {
// has data to receive
}
This won't touch the channel buffer, unlike foo, gotValue := <- ch which removes a value when gotValue == true.