I'm trying to implement a very simple test function to verify results coming from my solutions for Euler problems.
In the following code I've created a map of slices where on the index 0, I call the function which return a integer and on the index 1, the result I expect from that function.
package euler
import "testing"
func TestEulers(t *testing.T) {
tests := map[string][]int{
"Euler1": {Euler1(), 233168},
"Euler2": {Euler2(), 4613732},
"Euler3": {Euler3(), 6857},
"Euler4": {Euler4(), 906609},
"Euler5": {Euler5(), 232792560},
"Euler6": {Euler6(), 25164150},
}
for key, value := range tests {
if value[0] != value[1] {
t.Errorf("%s\nExpected: %d\nGot:%d",
key, value[0], value[1])
}
}
}
For that map, every function works fine and return the result I expect if I run one by one or if I comment, let's say, half part of those keys/values.
For example, if I call the the function above with these lines commented the test will PASS.
tests := map[string][]int{
"Euler1": {Euler1(), 233168},
// "Euler2": {Euler2(), 4613732},
"Euler3": {Euler3(), 6857},
"Euler4": {Euler4(), 906609},
// "Euler5": {Euler5(), 232792560},
// "Euler6": {Euler6(), 25164150},
}
But if I arrange the comments on that next way, for example, the test wouldn't.
tests := map[string][]int{
//"Euler1": {Euler1(), 233168},
"Euler2": {Euler2(), 4613732},
"Euler3": {Euler3(), 6857},
"Euler4": {Euler4(), 906609},
//"Euler5": {Euler5(), 232792560},
// "Euler6": {Euler6(), 25164150},
}
The test will give me an error:
WARNING: DATA RACE
Write by goroutine 6:
runtime.closechan()
/private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/runtime/chan.go:295 +0x0
github.com/alesr/project-euler.Euler2()
/Users/Alessandro/GO/src/github.com/alesr/project-euler/euler.go:40 +0xd7
github.com/alesr/project-euler.TestEulers()
/Users/Alessandro/GO/src/github.com/alesr/project-euler/euler_test.go:9 +0x46
testing.tRunner()
/private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:456 +0xdc
Previous read by goroutine 7:
runtime.chansend()
/private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/runtime/chan.go:107 +0x0
github.com/alesr/numbers.FibonacciGen.func1()
/Users/Alessandro/GO/src/github.com/alesr/numbers/numbers.go:103 +0x59
Goroutine 6 (running) created at:
testing.RunTests()
/private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:561 +0xaa3
testing.(*M).Run()
/private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:494 +0xe4
main.main()
github.com/alesr/project-euler/_test/_testmain.go:54 +0x20f
Goroutine 7 (running) created at:
github.com/alesr/numbers.FibonacciGen()
/Users/Alessandro/GO/src/github.com/alesr/numbers/numbers.go:105 +0x60
github.com/alesr/project-euler.Euler2()
/Users/Alessandro/GO/src/github.com/alesr/project-euler/euler.go:27 +0x32
github.com/alesr/project-euler.TestEulers()
/Users/Alessandro/GO/src/github.com/alesr/project-euler/euler_test.go:9 +0x46
testing.tRunner()
/private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:456 +0xdc
==================
panic: send on closed channel
goroutine 36 [running]:
github.com/alesr/numbers.FibonacciGen.func1(0xc8200a01e0)
/Users/Alessandro/GO/src/github.com/alesr/numbers/numbers.go:103 +0x5a
created by github.com/alesr/numbers.FibonacciGen
/Users/Alessandro/GO/src/github.com/alesr/numbers/numbers.go:105 +0x61
goroutine 1 [chan receive]:
testing.RunTests(0x24d038, 0x2f7340, 0x1, 0x1, 0xf78401)
/private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:562 +0xafa
testing.(*M).Run(0xc82004df00, 0x1ff0e8)
/private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:494 +0xe5
main.main()
github.com/alesr/project-euler/_test/_testmain.go:54 +0x210
goroutine 17 [syscall, locked to thread]:
runtime.goexit()
/private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/runtime/asm_amd64.s:1696 +0x1
goroutine 35 [runnable]:
github.com/alesr/strings.Flip(0xc8200727a0, 0x6, 0x0, 0x0)
/Users/Alessandro/GO/src/github.com/alesr/strings/strings.go:33 +0x17e
github.com/alesr/project-euler.Euler4(0x1ac9)
/Users/Alessandro/GO/src/github.com/alesr/project-euler/euler.go:73 +0x95
github.com/alesr/project-euler.TestEulers(0xc8200b6000)
/Users/Alessandro/GO/src/github.com/alesr/project-euler/euler_test.go:11 +0x63
testing.tRunner(0xc8200b6000, 0x2f7340)
/private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:456 +0xdd
created by testing.RunTests
/private/var/folders/q8/bf_4b1ts2zj0l7b0p1dv36lr0000gp/T/workdir/go/src/testing/testing.go:561 +0xaa4
exit status 2
FAIL github.com/alesr/project-euler 0.022s
But still, I checked every single function and they work just as expected.
You can access the Euler source code or the packages numbers and strings if you want.
At Euler2 function I have a defer statement to close the channel which is receiving from FibonacciGen.
And on FibonacciGen I do have another defer statement to close the same channel.
It seems that's the my first error. I should have just one and not two statements to close the channel, since they are trying to close the same thing. Is that correct?
Second (and here I'm even a little more unsure), the defer statement will prevent the function to be called until the main goroutine returns, right? Independently if I call it on the package main or not?
Plus, since the data is flowing through the channel from FibonacciGen to the main function. It seems for me, that if I close the channel at FibonacciGen I don't need to notify the main function. But If I close the channel on the main function I do have to notify FibonacciGen to stop trying to send to this channel.
In your Euler2() you don't check if the channel has been closed. Once it's closed it's unblocked, so it tries to send a value to a now closed channel.
If you only run Euler2() your program might just exit before you send the value to the closed channel.
Thank you all. With your help I could understand that I was closing the channel in the wrong way.
Now works correctly.
func Euler2() int {
c := make(chan int)
done := make(chan bool)
go numbers.FibonacciGen(c, done)
sum := 0
var f int
for {
f = <-c
if f < 4000000 {
if f%2 == 0 {
sum += f
}
} else {
close(done)
return sum
}
}
}
func FibonacciGen(c chan int, done chan bool) {
for {
select {
case <-done:
return
default:
for i, j := 0, 1; ; i, j = i+j, i {
c <- i
}
}
}
}
Related
I'm learning concurrency-related issues in Golang. I wrote some code:
package main
import (
"fmt"
"time"
)
func incr(num *int) {
*num = *num + 1
}
func main() {
var a = 0
for i := 0; i < 50; i++ {
go incr(&a)
}
incr(&a)
time.Sleep(1 * time.Second)
fmt.Println(a)
}
The result of this code is: 51
In this code I've declared a variable which I'm increasing in 50 running goroutines. What I've read and unsterstood this code should fail because multiple goroutines are writing to same memory address. In this case I should add sync.Mutex lock in order to fix that.
Code is available in the playground: https://play.golang.org/p/Tba9pfpxaHY
Could You explain what is really happening in this program?
Guess what? I ran your app and I get varying outputs: sometimes 49, sometimes 48, sometimes 50 (and sometimes 51).
If you run your app with the race detector enabled (go run -race play.go), it tells you have data races:
==================
WARNING: DATA RACE
Read at 0x00c00009a010 by goroutine 7:
main.incr()
/home/icza/gows/src/play/play.go:9 +0x3a
Previous write at 0x00c00009a010 by goroutine 6:
main.incr()
/home/icza/gows/src/play/play.go:9 +0x50
Goroutine 7 (running) created at:
main.main()
/home/icza/gows/src/play/play.go:17 +0x83
Goroutine 6 (finished) created at:
main.main()
/home/icza/gows/src/play/play.go:17 +0x83
==================
When you have data races, the behavior of your app is undefined. "Seemingly working sometimes" also fits into the "undefined" behavior, but undefined also means it can do anything else too.
See related questions:
Assign a map to another map is safety in golang?
Is it safe to read a function pointer concurrently without a lock?
golang struct concurrent read and write without Lock is also running ok?
Why does this code cause data race?
I have already used atomic add.
package main
import (
"sync/atomic"
"time"
)
var a int64
func main() {
for {
if a < 100 {
atomic.AddInt64(&a, 1)
go run()
}
}
}
func run() {
<-time.After(5 * time.Second)
atomic.AddInt64(&a, -1)
}
I run command go run --race with this code and get:
==================
WARNING: DATA RACE
Write at 0x000001150f30 by goroutine 8:
sync/atomic.AddInt64()
/usr/local/Cellar/go/1.11.2/libexec/src/runtime/race_amd64.s:276 +0xb
main.run()
/Users/flask/test.go:22 +0x6d
Previous read at 0x000001150f30 by main goroutine:
main.main()
/Users/flask/test.go:12 +0x3a
Goroutine 8 (running) created at:
main.main()
/Users/flask/test.go:15 +0x75
==================
Could you help me explain this?
And how to fix this warning?
Thanks!
You didn't use the atomic package at all places where you accessed the variable. All access must be synchronized to variables that are accessed from multiple goroutines concurrently, including reads:
for {
if value := atomic.LoadInt64(&a); value < 100 {
atomic.AddInt64(&a, 1)
go run()
}
}
With that change, the race condition goes away.
If you just want to inspect the value, you don't even need to store it in a variable, so you may simply do:
for {
if atomic.LoadInt64(&a) < 100 {
atomic.AddInt64(&a, 1)
go run()
}
}
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 4 years ago.
Improve this question
A seemingly clever trick to avoid locking in concurrent C code goes like this: I have a global variable ptr which points to a mystruct and I want to update that structure. So I'll allocate a new mystruct, fill the data in and only then I'll make the change visible to the world by pointing ptr to the new mystruct object.
This is incorrect as it depends on the ordering of writes and there's no guarantee that the write to ptr will become visible to other threads after all stores to the new mystruct have taken place. Therefore, the new mystruct object can be returned partially initialized.
My question is: can this happen in Go, too? I think it can, but I have to say I found The Go Memory Model a little incomprehensible.
I wrote a bit of Go code to test it out, but on my machine, the bad behavirour does not manifest itself:
package main
import (
"fmt"
"time"
)
type mystruct struct {
a int
b int
}
var (
ptr *mystruct
counter int
)
func writer() {
for {
counter += 1
s := mystruct{a: counter, b: counter}
ptr = &s
}
}
func reader() {
time.Sleep(time.Millisecond)
for {
if ptr.a != ptr.b {
fmt.Println("Oh no, I'm so buggy!")
}
}
}
func main() {
go writer()
go reader()
select {}
}
This of course proves nothing.
Can you please provide a very brief comparison of memory guarantees provided by Go's goroutines with (almost no guarantees) provided by a POSIX thread in C?
The Go Memory Model
Version of May 31, 2014
Advice
If you must read the rest of this document to understand the behavior
of your program, you are being too clever.
Don't be clever.
Introducing the Go Race Detector
I [David] wrote a bit of Go code to test it out.
Your Go program has data races. The results are undefined.
$ go run -race david.go
==================
WARNING: DATA RACE
Read at 0x000000596cc0 by goroutine 7:
main.reader()
/home/peter/gopath/src/david.go:29 +0x4b
Previous write at 0x000000596cc0 by goroutine 6:
main.writer()
/home/peter/gopath/src/david.go:22 +0xf8
Goroutine 7 (running) created at:
main.main()
/home/peter/gopath/src/david.go:37 +0x5a
Goroutine 6 (running) created at:
main.main()
/home/peter/gopath/src/david.go:36 +0x42
==================
==================
WARNING: DATA RACE
Read at 0x00c0000cc270 by goroutine 7:
main.reader()
/home/peter/gopath/src/david.go:29 +0x5b
Previous write at 0x00c0000cc270 by goroutine 6:
main.writer()
/home/peter/gopath/src/david.go:21 +0xd2
Goroutine 7 (running) created at:
main.main()
/home/peter/gopath/src/david.go:37 +0x5a
Goroutine 6 (running) created at:
main.main()
/home/peter/gopath/src/david.go:36 +0x42
==================
==================
WARNING: DATA RACE
Read at 0x00c0000cda38 by goroutine 7:
main.reader()
/home/peter/gopath/src/david.go:29 +0x7f
Previous write at 0x00c0000cda38 by goroutine 6:
main.writer()
/home/peter/gopath/src/david.go:21 +0xd2
Goroutine 7 (running) created at:
main.main()
/home/peter/gopath/src/david.go:37 +0x5a
Goroutine 6 (running) created at:
main.main()
/home/peter/gopath/src/david.go:36 +0x42
==================
<<SNIP>>
Your Go program: david.go:
package main
import (
"fmt"
"time"
)
type mystruct struct {
a int
b int
}
var (
ptr *mystruct
counter int
)
func writer() {
for {
counter += 1
s := mystruct{a: counter, b: counter}
ptr = &s
}
}
func reader() {
time.Sleep(time.Millisecond)
for {
if ptr.a != ptr.b {
fmt.Println("Oh no, I'm so buggy!")
}
}
}
func main() {
go writer()
go reader()
select {}
}
Playground: https://play.golang.org/p/XKywmzrRRRw
Is it thread-safe to modify the channel that a consumer is reading from?
Consider the following code:
func main(){
channel := make(chan int, 3)
channel_ptr := &channel
go supplier (channel_ptr)
go consumer (channel_ptr)
temp = *channel_ptr
// Important bit
*channel_ptr = make(chan int, 5)
more := true
for more{
select {
case msg := <-temp:
*channel_ptr <- msg
default:
more = false
}
}
// Block main indefinitely to keep the children alive
<-make(chan bool)
}
func consumer(c *chan int){
for true{
fmt.Println(<-(*c))
}
}
func supplier(c *chan int){
for i :=0; i < 5; i ++{
(*c)<-i
}
}
If channels and make work the way that I want them to, I should get the following properties:
The program always outputs 0 1 2 3 4
The program will never panic from trying to read from a non-initialized channel (IE, the part I labelled Important bit is atomic)
From several test runs, this seems to be true, but I can't find it anywhere in the documentation and I'm worried about subtle race conditions.
Update
Yeah, what I was doing doesn't work. This thread is probably buried at this point, but does anybody know how to dynamically resize a buffered channel?
It's not thread safe.
If you run with -race flag to use race detector, you'll see the bug:
$ run -race t.go
==================
WARNING: DATA RACE
Write at 0x00c420086018 by main goroutine:
main.main()
/Users/kjk/src/go/src/github.com/kjk/go-cookbook/start-mysql-in-docker-go/t.go:14 +0x128
Previous read at 0x00c420086018 by goroutine 6:
main.supplier()
/Users/kjk/src/go/src/github.com/kjk/go-cookbook/start-mysql-in-docker-go/t.go:37 +0x51
Goroutine 6 (running) created at:
main.main()
/Users/kjk/src/go/src/github.com/kjk/go-cookbook/start-mysql-in-docker-go/t.go:9 +0xb4
0
==================
1
2
3
==================
WARNING: DATA RACE
Read at 0x00c420086018 by goroutine 6:
main.supplier()
/Users/kjk/src/go/src/github.com/kjk/go-cookbook/start-mysql-in-docker-go/t.go:37 +0x51
Previous write at 0x00c420086018 by main goroutine:
main.main()
/Users/kjk/src/go/src/github.com/kjk/go-cookbook/start-mysql-in-docker-go/t.go:14 +0x128
Goroutine 6 (running) created at:
main.main()
/Users/kjk/src/go/src/github.com/kjk/go-cookbook/start-mysql-in-docker-go/t.go:9 +0xb4
==================
4
As a rule of thumb, you should never pass channel as a pointer. Channel already is a pointer internally.
Stepping back a bit: I don't understand what you're trying to achieve.
I guess there's a reason you're trying to pass a channel as a pointer. The pattern of using channels in Go is: you create it once and you pass it around as value. You don't pass a pointer to it and you never modify it after creation.
In your example the problem is that you have a shared piece of memory (memory address pointed to by channel_ptr) and you write to that memory in one thread while some other thread reads it. That's data race.
It's not specific to a channel, you would have the same issue if it was pointer to an int and two threads were modifying the value of an int.
I'm getting a strange data race reported when I use xml.Marshal on a struct. I am 99% sure that I am not sending the same variable or something like that - instead I get intermittent errors in which the marshal function believes that it is causing a data race.
Here's the code (simplified a bit but all functional elements are there):
// this is run prior to any calls being sent to the below functions
func Setup() (descriptor *serviceDescriptor) {
descriptor = new(serviceDescriptor)
wpChan := make(chan *Call)
for i := 1; i < 100; i++ {
go serviceWorker(wpChan)
}
descriptor.wpChan = wpChan
return
}
// called externally to initiate a call
func (s *serviceDescriptor) Add(c *Call) {
go s.makeCall(c)
}
// sends a call to the worker pool set up in the Setup() function
func (s *serviceDescriptor) makeCall(c *Call) {
cw := new(callwrapper)
cw.internal = new(etFullCall)
cw.internal.Calldata = c
/// this is a channel that the next function listens to
s.wpChan <- ct
// the result is sent back on a channel and processed here
}
// this is a function
func worker(wpChan chan *callwrapper) {
for cw := range wpChan {
v := new(Result)
// this is where the data race occurs.
byt, err := xml.Marshal(cw.internal)
// do stuff with the result down here
}
}
Here's the error:
WARNING: DATA RACE
Write by goroutine 115:
runtime.copy()
/usr/lib/go/src/pkg/runtime/slice.c:120 +0x0
encoding/xml.(*parentStack).push()
/usr/lib/go/src/pkg/encoding/xml/marshal.go:901 +0x2bd
encoding/xml.(*printer).marshalStruct()
/usr/lib/go/src/pkg/encoding/xml/marshal.go:819 +0x58c
encoding/xml.(*printer).marshalValue()
/usr/lib/go/src/pkg/encoding/xml/marshal.go:524 +0x12a8
encoding/xml.(*Encoder).Encode()
/usr/lib/go/src/pkg/encoding/xml/marshal.go:153 +0x83
encoding/xml.Marshal()
/usr/lib/go/src/pkg/encoding/xml/marshal.go:72 +0x9d
exacttarget.serviceWorker()
/service.go:94 +0x20e
Previous write by goroutine 114:
runtime.copy()
/usr/lib/go/src/pkg/runtime/slice.c:120 +0x0
encoding/xml.(*parentStack).push()
/usr/lib/go/src/pkg/encoding/xml/marshal.go:901 +0x2bd
encoding/xml.(*printer).marshalStruct()
/usr/lib/go/src/pkg/encoding/xml/marshal.go:819 +0x58c
encoding/xml.(*printer).marshalValue()
/usr/lib/go/src/pkg/encoding/xml/marshal.go:524 +0x12a8
encoding/xml.(*Encoder).Encode()
/usr/lib/go/src/pkg/encoding/xml/marshal.go:153 +0x83
encoding/xml.Marshal()
/usr/lib/go/src/pkg/encoding/xml/marshal.go:72 +0x9d
exacttarget.serviceWorker()
/service.go:94 +0x20e
Goroutine 115 (running) created at:
service.Setup()
/service.go:39 +0x112
/src.processSingleUser()
/main_test.go:405 +0xdf
/src.testAddLists()
/main_test.go:306 +0x1f2
Goroutine 114 (running) created at:
service.Setup()
/service.go:39 +0x112
/src.processSingleUser()
/main_test.go:405 +0xdf
src.testAddLists()
/main_test.go:306 +0x1f2