I had read various pages such as https://github.com/golang/go/wiki/CommonMistakes which outlined the issues with using closures and goroutines in a loop. As such I wrote my original loops as follows:
for outstanding < threads {
ttl += 1;
outstanding += 1;
go func (ttl int, results chan Result) {
results <- pw.SendTTL(ttl, dest)
results <- pw.Recv(3)
}(ttl, results)
}
Passing the changing TTL as an argument to the anonymous function. I ended up getting a random assortment of values over the range. Say if I was expecting 1-5 I'd get a couple 1's, a couple 3's, maybe a 4.
So I tried the following, in case there was something about specifically using the variable instantiated by the loop. Yes I know I'm sort of abusing the for loop here...
for i := ttl; outstanding < threads; i++ {
go func (ttl int, results chan Result) {
results <- pw.SendTTL(ttl, dest)
results <- pw.Recv(3)
}(i, results)
outstanding++;
}
No joy. Same experience.
I also tried the other suggested option where you use a local variable in the loop, and use that within the closure. Same experience.
What am I doing wrong here? What boat did I miss?
In writing out the requested example I think I realized what my problem was. The various go routines were clobbering each others TTL settings when attempting to share the socket.
Question withdrawn ;)
Edit: To clarify, the right value was in fact being passed to the routines in either case. It was an underlying sharing of resources that was the problem.
Related
I am trying to learn goLang by studying different examples online. in this one example it is a quiz test where a CSV file is given to the user with questions and answers, the timer fires if the user does not answer the Q in a given time. What I don’t understand in the code below why are we creating the answer channel in for loop each time for every different question. why can’t we define the timer outside the for loop and use that for every question isn’t that inefficient coding?
problemloop:
for i, p := range problems {
fmt.Printf("Problem #%d: %s = ", i+1, p.q)
answerCh := make(chan string)
go func() {
var answer string
fmt.Scanf("%s\n", &answer)
answerCh <- answer
}()
select {
case <-timer.C:
fmt.Println()
break problemloop
case answer := <-answerCh:
if answer == p.a {
correct++
}
}
}
In Go, creating channels is very cheap. It's a common idiom, therefore, to create an "answer channel" you pass to a goroutine. Goroutines can't just return a value to a caller the way a function can do. When the goroutine is done, it sends its answer/result to the channel. Receiving on this channel in the main (or some other consumer) goroutine serves as a sync point. And allows to do timeouts, like your example demonstrates.
I wouldn't worry about efficiency here unless you can prove with profiling that this is the hot path. It's likely that this code could have been written with a single channel, but it's hard to say looking at the small snippet you provided.
I'd like some feedback on the implementation details of what I'm trying to build. What I want to achieve is have multiple functions with different signatures that can be called concurrently.
Calling the functions in coroutines sequentially works fine, but I'm wondering if there's a way to do this in a more idiomatic way, e.g. iterate over a slice of functions.
Since each function has different arguments and return values though, I have trouble figuring out what the best approach would be. An example that is a bit similar to my goal can be seen here: Golang - How do you create a slice of functions with different signatures?, but there the code just calls the functions and doesn't account for any return values.
Is what I have in mind even possible?
You can use code from linked question and just wrap the v.Call(params) into an anonymous function executing in its own goroutine like this:
...
// WaitGroup to wait on goroutines to finish their execution
var wg sync.WaitGroup
for a, v := range f {
v := reflect.TypeOf(v)
//calling the function from reflect
val := reflect.ValueOf(f[a])
params := make([]reflect.Value, v.NumIn())
if v.NumIn() == 1 {
params[0] = reflect.ValueOf(1564)
} else if v.NumIn() == 2 {
params[0] = reflect.ValueOf("Test FROM reflect")
params[1] = reflect.ValueOf(float32(123456))
}
// Run them in parallel
wg.Add(1)
go func() {
defer wg.Done()
val.Call(params)
}()
}
wg.Wait()
See it on Go Playground
As for return values Value.Call() returns []Value which is slice of return values - so you are covered here too. Your question doesn't specify what you intend to do with results but given they will be generated in parallel you'll probably need to send them through a channel(s) - you can do that in anonymous function (after processing return slice) too.
go func() { MyPackage.MyFunc(with, whatsoever, signature); }() - roughtly, that's what you need. You span as many goroutines (using the go keyword) as there are concurrent functions.
There is no notion of "returned value" from goroutine. For that you have to use channels. They are primary communication mechanism. So, you span a new goroutine with some function f of arbitrary signature and when it's done and you got some result, you send it to some channel shared between goroutines for communication.
Channels are thread-safe and were carefully designed to handle such a communication gracefully. Go, as programming language, provides few keywords that deal with reading/writing to/from channels. So there are pretty fundamental to (concurrent) programming in Go.
However, of course, you can handle it differently. Sharing some mutable memory protected by some kind of locking, or relying upon lockless compareAndSet fashion. Arguably, that's less idiomatic way and generally have to be avoided. Always prefer channels.
In Go, a sync.Mutex or chan is used to prevent concurrent access of shared objects. However, in some cases I am just interested in the "latest" value of a variable or field of an object.
Or I like to write a value and do not care if another go-routine overwrites it later or has just overwritten it before.
Update: TLDR; Just don't do this. It is not safe. Read the answers, comments, and linked documents!
Update 2021: The Go memory model is going to be specified more thoroughly and there are three great articles by Russ Cox that will teach you more about the surprising effects of unsynchronized memory access. These articles summarize a lot of the below discussions and learnings.
Here are two variants good and bad of an example program, where both seem to produce "correct" output using the current Go runtime:
package main
import (
"flag"
"fmt"
"math/rand"
"time"
)
var bogus = flag.Bool("bogus", false, "use bogus code")
func pause() {
time.Sleep(time.Duration(rand.Uint32()%100) * time.Millisecond)
}
func bad() {
stop := time.After(100 * time.Millisecond)
var name string
// start some producers doing concurrent writes (DANGER!)
for i := 0; i < 10; i++ {
go func(i int) {
pause()
name = fmt.Sprintf("name = %d", i)
}(i)
}
// start consumer that shows the current value every 10ms
go func() {
tick := time.Tick(10 * time.Millisecond)
for {
select {
case <-stop:
return
case <-tick:
fmt.Println("read:", name)
}
}
}()
<-stop
}
func good() {
stop := time.After(100 * time.Millisecond)
names := make(chan string, 10)
// start some producers concurrently writing to a channel (GOOD!)
for i := 0; i < 10; i++ {
go func(i int) {
pause()
names <- fmt.Sprintf("name = %d", i)
}(i)
}
// start consumer that shows the current value every 10ms
go func() {
tick := time.Tick(10 * time.Millisecond)
var name string
for {
select {
case name = <-names:
case <-stop:
return
case <-tick:
fmt.Println("read:", name)
}
}
}()
<-stop
}
func main() {
flag.Parse()
if *bogus {
bad()
} else {
good()
}
}
The expected output is as follows:
...
read: name = 3
read: name = 3
read: name = 5
read: name = 4
...
Any combination of read: and read: name=[0-9] is correct output for this program. Receiving any other string as output would be an error.
When running this program with go run --race bogus.go it is safe.
However, go run --race bogus.go -bogus warns of the concurrent reads and writes.
For map types and when appending to slices I always need a mutex or a similar method of protection to avoid segfaults or unexpected behavior. However, reading and writing literals (atomic values) to variables or field values seems to be safe.
Question: Which Go data types can I safely read and safely write concurrently without a mutext and without producing segfaults and without reading garbage from memory?
Please explain why something is safe or unsafe in Go in your answer.
Update: I rewrote the example to better reflect the original code, where I had the the concurrent writes issue. The important leanings are already in the comments. I will accept an answer that summarizes these learnings with enough detail (esp. on the Go-runtime).
However, in some cases I am just interested in the latest value of a variable or field of an object.
Here is the fundamental problem: What does the word "latest" mean?
Suppoose that, mathematically speaking, we have a sequence of values Xi, with 0 <= i < N. Then obviously Xj is "later than" Xi if j > i. That's a nice simple definition of "latest" and is probably the one you want.
But when two separate CPUs within a single machine—including two goroutines in a Go program—are working at the same time, time itself loses meaning. We cannot say whether i < j, i == j, or i > j. So there is no correct definition for the word latest.
To solve this kind of problem, modern CPU hardware, and Go as a programming language, gives us certain synchronization primitives. If CPUs A and B execute memory fence instructions, or synchronization instructions, or use whatever other hardware provisions exist, the CPUs (and/or some external hardware) will insert whatever is required for the notion of "time" to regain its meaning. That is, if the CPU uses barrier instructions, we can say that a memory load or store that was executed before the barrier is a "before" and a memory load or store that is executed after the barrier is an "after".
(The actual implementation, in some modern hardware, consists of load and store buffers that can rearrange the order in which loads and stores go to memory. The barrier instruction either synchronizes the buffers, or places an actual barrier in them, so that loads and stores cannot move across the barrier. This particular concrete implementation gives an easy way to think about the problem, but isn't complete: you should think of time as simply not existing outside the hardware-provided synchronization, i.e., all loads from, and stores to, some location are happening simultaneously, rather than in some sequential order, except for these barriers.)
In any case, Go's sync package gives you a simple high level access method to these kinds of barriers. Compiled code that executes before a mutex Lock call really does complete before the lock function returns, and the code that executes after the call really does not start until after the lock function returns.
Go's channels provide the same kinds of before/after time guarantees.
Go's sync/atomic package provides much lower level guarantees. In general you should avoid this in favor of the higher level channel or sync.Mutex style guarantees. (Edit to add note: You could use sync/atomic's Pointer operations here, but not with the string type directly, as Go strings are actually implemented as a header containing two separate values: a pointer, and a length. You could solve this with another layer of indirection, by updating a pointer that points to the string object. But before you even consider doing that, you should benchmark the use of the language's preferred methods and verify that these are a problem, because code that works at the sync/atomic level is hard to write and hard to debug.)
Which Go data types can I safely read and safely write concurrently without a mutext and without producing segfaults and without reading garbage from memory?
None.
It really is that simple: You cannot, under no circumstance whatsoever, read and write concurrently to anything in Go.
(Btw: Your "correct" program is not correct, it is racy and even if you get rid of the race condition it would not deterministically produce the output.)
Why can't you use channels
package main
import (
"fmt"
"sync"
)
func main() {
var wg sync.WaitGroup // wait group to close channel
var buffer int = 1 // buffer of the channel
// channel to get the share data
cName := make(chan string, buffer)
for i := 0; i < 10; i++ {
wg.Add(1) // add to wait group
go func(i int) {
cName <- fmt.Sprintf("name = %d", i)
wg.Done() // decrease wait group.
}(i)
}
go func() {
wg.Wait() // wait of wait group to be 0
close(cName) // close the channel
}()
// process all the data
for n := range cName {
println("read:", n)
}
}
The above code returns the following output
read: name = 0
read: name = 5
read: name = 1
read: name = 2
read: name = 3
read: name = 4
read: name = 7
read: name = 6
read: name = 8
read: name = 9
https://play.golang.org/p/R4n9ssPMOeS
Article about channels
I was asked about finding a bug in some Go code and I am stumped as to what this obvious bug should be. I was asked the following:
What is wrong with this code? It’s incomplete, but the snippet should be enough to spot the bug. How would you fix it? What other improvements can you propose?
Tried different coding solutions to example and find the bug, but don't see an obvious bug in the code because I can get this code to work fine (depending on how I code for it). I told them that the code isn't waiting on the worker processes or checking that they're completing before closing the channel. I stated that the code is too abstract to say for sure and it depends on the goal of the code, but that it can be made to work fine (not that I'd code it that way, of course).
const nworkers = 5
type Task struct {
...
}
func worker(queue chan *Task) {
for task := range queue {
task.execute()
}
}
func main() {
tasks := getTasks()
queue := make(chan *Task, nworkers)
for i := 0; i < nworkers; i++ {
go worker(queue)
}
for i := 0; i < len(tasks); i++ {
queue <- tasks[i]
}
close(queue)
}
/*
Again, I had created some code to populate the missing code areas and
added wait to the workers and checks to ensure they completed, passing
the structure to the function call, added code to populate the struct
data and passed the data with queue <- &tasks[i] to reference (in how I
was doing it with one example) and so on. I literally can't see anything
obvious as per the bug in the code example if I fill in some blanks to
make it work).
*/
No specific expected output, just trying to locate this bug so I can learn from this question as I delve further into the Go language. Can anyone spot what should be an obvious bug and explain where/what the bug is?
This code is from the most popular go matrix package https://github.com/skelterjohn/go.matrix/blob/go1/util.go
I googled this function and seems like it is for computing the fractal dimension. But in this package, this function is never used so I am having a hard time understanding this.
func countBoxes(start, cap int) chan box {
ints := make(chan box)
go func() {
for i := start; i < cap; i++ {
ints <- i
}
close(ints)
}()
return ints
}
Why do we need goroutine when we have only one anonymous function here?
And does anybody know what this function does in terms of matrix work?
Thanks in advance.
It returns a channel with cap - start queued integer events. (i.e. You can 'read'
start,start+1..,cap from the channel and then it closes ).
If you poke around in the code, it uses a similar kind of construct to create an iterator for the the indices of non-zero entries of sparse matrices. Look in sparse.go.
It's not used anywhere in the code that I can find, it may have been just to test
the idea.