concurrent map read and write when there is no concurrency [closed] - go

Closed. This question needs debugging details. It is not currently accepting answers.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
Closed 5 years ago.
Improve this question
The following go play example shows in a simplistic way what I have defined. I am passing a map as a copied value to a function (not a reference) as well as there is a recursion in my function which I assume passes by value as well.
https://play.golang.org/p/na6y6Wih4M
// this function has no write operations to dataMap, just reads
// dataMap, in fact, has no write operations since it was copied
func findParentAncestors(ID int, dataMap map[int]Data) []Data {
results := []Data{}
if _, ok := dataMap[ID]; ok {
if parentData, ok := dataMap[dataMap[ID].ParentID]; ok {
results = append(results, parentData)
// recursion
results = append(results, findParentAncestors(parentData.ID, dataMap)...)
}
}
return results
}
PROBLEM: somehow along my program execution, which involves much more data than this example (obviusly), an error "fatal error: concurrent map read and map write" points function findParentAncestors():
main.findParentAncestors(0x39e3, 0xc82013ac90, 0x0, 0x0, 0x0)
/opt/test/src/test.go:17 +0xa6 fp=0xc820269fb8 sp=0xc820269bd0
main.findParentAncestors(0x5d25, 0xc82013ac90, 0x0, 0x0, 0x0)
/opt/test/src/test.go:21 +0x239 fp=0xc82026a3a0 sp=0xc820269fb8

From your example, https://play.golang.org/p/na6y6Wih4M:
// the orignalMap is defined elsewhere in the program (here represented)
originalMap := map[int]Data{}
originalMap[0] = Data{ID: 0, ParentID: -1, Name: "zero"}
originalMap[1] = Data{ID: 1, ParentID: 0, Name: "one"}
originalMap[2] = Data{ID: 2, ParentID: 1, Name: "two"}
// copies the original map from a global location (here represented)
copiedMap := originalMap
// identifies ancestors unsing the copied map
parents := findParentAncestors(2, copiedMap)
This is a misnomer, copiedMap := originalMap, you are not copying the map.
In Go all arguments are passed by value. It's equivalent to assigning each argument to each parameter. For a map, assignment, copiedMap := originalMap, or passing by value, findParentAncestors(2, copiedMap), copies the map descriptor which is a pointer to the map descriptor struct which contains a pointer to the map key-value data. Obviously you have a potential race condition if there are any writes to the map.
You are using go version go1.6.3 linux/amd64, so run the race detector.
Go 1.6 Release Notes
Runtime
The runtime has added lightweight, best-effort detection of concurrent
misuse of maps. As always, if one goroutine is writing to a map, no
other goroutine should be reading or writing the map concurrently. If
the runtime detects this condition, it prints a diagnosis and crashes
the program. The best way to find out more about the problem is to run
the program under the race detector, which will more reliably identify
the race and give more detail.
Command go
Compile packages and dependencies
-race
enable data race detection.
Supported only on linux/amd64, freebsd/amd64, darwin/amd64 and windows/amd64.
Also, compile and run your program using Go 1.8, the current release of Go, which significantly improves concurrent map misuse.
Go 1.8 Release Notes
Concurrent Map Misuse
In Go 1.6, the runtime added lightweight, best-effort detection of
concurrent misuse of maps. This release improves that detector with
support for detecting programs that concurrently write to and iterate
over a map.
As always, if one goroutine is writing to a map, no other goroutine
should be reading (which includes iterating) or writing the map
concurrently. If the runtime detects this condition, it prints a
diagnosis and crashes the program. The best way to find out more about
the problem is to run the program under the race detector, which will
more reliably identify the race and give more detail.

Related

Question on the go memory model,the last example

i have a question on the go memory model.
in the last example:
type T struct {
msg string
}
var g *T
func setup() {
t := new(T)
t.msg = "hello, world"
g = t
}
func main() {
go setup()
for g == nil {
}
print(g.msg)
}
In my opnion,reads and writes of values with a single machine word is a atomic behavior.I try many times to run the test but it is always can be observed.
So please tell me why g.msg is not guarntee to observed? I want to know the reason in detail,please.
Because there are 2 write operations in the launched goroutine:
t := new(T) // One
t.msg = "hello, world" // Two
g = t
It may be that the main goroutine will observe the non-nil pointer assignment to g in the last line, but since there is no explicit synchronization between the 2 goroutines, the compiler is allowed to reorder the operations (that doesn't change the behavior in the launched goroutine), e.g. to the following:
t := new(T) // One
g = t
t.msg = "hello, world" // Two
If operations would be rearranged like this, the behavior of the launched goroutine (setup()) would not change, so a compiler is allowed to to this. And in this case the main goroutine could observe the effect of g = t, but not t.msg = "hello, world".
Why would a compiler reorder the operations? E.g. because a different order may result in a more efficient code. E.g. if the pointer assigned to t is already in a register, it can also be assigned to g right away, without having to reload it again if the assignment to g would not be executed right away.
This is mentioned in the Happens Before section:
Within a single goroutine, reads and writes must behave as if they executed in the order specified by the program. That is, compilers and processors may reorder the reads and writes executed within a single goroutine only when the reordering does not change the behavior within that goroutine as defined by the language specification. Because of this reordering, the execution order observed by one goroutine may differ from the order perceived by another. For example, if one goroutine executes a = 1; b = 2;, another might observe the updated value of b before the updated value of a.
If you use proper synchronization, that will forbid the compiler to perform such rearranging that would change the observed behavior from other goroutines.
Running your example any number of times and not observing this does not mean anything. It may be the problem will never arise, it may be it will arise on a different architecture, or on a different machine, or when compiled with a different (future) version of Go. Simply do not rely on such behavior that is not guaranteed. Always use proper synchronization, never leave any data races in your app.

How to Disable Stack Trace Info in Go for any OS Executing the Binary

I am currently building a CLI using Go and am trying to disable any backtrace that is produced as a result of a panic. I believe my code has great error handling, but would now like to suppress any panic messages (fairly new to Go).
I currently put in the following in my main.go function (to purposely cause a panic):
var myarr [2]string
myarr[0] = "Foo"
myarr[1] = "Bar"
for i := range myarr {
fmt.Println(myarr[i+1])
}
And I get this as a result:
goroutine 1 [running]:
Bar
panic: runtime error: index out of range [2] with length 2
main.main()
{directory where main.go is located}/main.go:23 +0x207
How can I suppress this error such that anyone with the executable binary file will not be able to see this error?
I've tried utilizing the GOBACKTRACE environment variable when building my binary and setting its value to GOBACKTRACE=none, but this has no effect on other operating systems I've tested on.
I've tried utilizing the GOBACKTRACE environment variable when building my binary and setting its value to GOBACKTRACE=none, but this has no effect on other operating systems I've tested on.
The environment variable is called GOTRACEBACK, not GOBACKTRACE.
Also, you can use debug.SetTraceback("none") to achieve the same effect, although this can still be overridden via the GOTRACEBACK environment variable.
If you use the correct naming, it should work. If it does not work, congratulations: you found a bug in golang and you should likely report it.
As #Burak mentioned you want to use the built-in Go function recover. There's a good Go blog post on all the subtleties of panic and recovery.
If you want to blanket cover your entire application stack, then simply register recover via a defer function at the main level:
func main() {
defer func() {
if r := recover(); r != nil {
fmt.Println("unexpected problem encountered - aborting")
// optionally log `r` to an exception log - so users may email to developer
os.Exit(1)
}
}()
run(os.Args...)
}
https://play.golang.org/p/b8qYnlNZsr5

Golang error function arguments too large for new goroutine

I am running a program with go 1.4 and I am trying to pass a large struct to a go function.
go ProcessImpression(network, &logImpression, campaign, actualSpent, partnerAccount, deviceId, otherParams)
I get this error:
runtime.newproc: function arguments too large for new goroutine
I have moved to pass by reference which helps but I am wondering if there is some way to pass large structs in a go function.
Thanks,
No, none I know of.
I don't think you should be too aggressive tuning to avoid copying, but it appears from the source that this error is emitted when parameters exceed the usable stack space for a new goroutine, which should be kilobytes. The copying overhead is real at that point, especially if this isn't the only time these things are copied. Perhaps some struct either explicitly is larger than expected thanks to a large struct member (1kb array rather than a slice, say) or indirectly. If not, just using a pointer as you have makes sense, and if you're worried about creating garbage, recycle the structs pointed to using sync.Pool.
I was able to fix this issue by changing the arguments from
func doStuff(prev, next User)
to
func doStuff(prev, next *User)
The answer from #twotwotwo in here is very helpful.
Got this issue at processing list of values([]BigType) of big struct:
for _, stct := range listBigStcts {
go func(stct BigType) {
...process stct ...
}(stct) // <-- error occurs here
}
Workaround is to replace []BigType with []*BigType

Number of specific go routines running

In go's runtime lib we have NumGoroutine() to return the total number of go routines running at that time. I was wondering if there was a simple way of getting the number of go routines running of a specific function?
Currently I have it telling me I have 1000, or whatever, of all go routines but I would like to know I have 500 go routines running func Foo. Possible? Simple? Shouldn't bother with it?
I'm afraid you have to count the goroutines on your own if you are interested in those numbers. The cheapest way to achieve your goal would be to use the sync/atomic package directly.
import "sync/atomic"
var counter int64
func example() {
atomic.AddInt64(&counter, 1)
defer atomic.AddInt64(&counter, -1)
// ...
}
Use atomic.LoadInt64(&counter) whenever you want to read the current value of the counter.
And it's not uncommon to have a couple of such counters in your program, so that you can monitor it easily. For example, take a look at the CacheStats struct of the recently published groupcache source.

How to interpret negative line number in stack trace

I made some changes to a fairly large project of mine today, and now I'm getting some odd behavior. Because I'm a knucklehead, I can't go back and figure out what I did.
But the main thrust of my question is how I should understand the negative line number in the stack trace that is printed. The -1218 below is the one that I mean.
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x1 pc=0x80501f2]
goroutine 1 [running]:
server.init() // vv-------------RIGHT HERE
/home/.../debugComponent.go:-1218 +0x282
_/home/.../possessions.init()
/home/.../possessions.go:29 +0x42
_/home/.../pageWrap.init()
/home/.../pageWrap.go:112 +0x47
main.init()
/home/.../main.go:0 +0x3c
goroutine 2 [syscall]:
goroutine 3 [runnable]:
The associated debugComponent.go file is pretty non-essential right now, so I removed it to see what would happen, and the file name just gets replaced with a different one, and a different negative number.
I've had to find plenty of bugs while developing this app, but this one has got me stumped.
If it helps, there's the main.go and then several packages in play. The three files listed above are all different packages, and this seems to be happening during the imports.
I hope you've read this far, because here's the strangest part. If I add this declaration to main.go, the error goes away!
var test = func() int { return 1 }() // Everything is fine now!
Very confusing! It doesn't fix it if I do var test = "foobar". It has to be the invoked func.
Any insight is appreciated, but mostly I'm curious about the -1218 in the trace.
Update
I'm trying to get this down to a small example that reproduces the issue. After working on it I reverted back to my original code, and restarted the machine.
The first time I tried to build and run, two new entries were added to the top of the stack trace. But only the first time.
goroutine 1 [syscall]:
syscall.Syscall()
/usr/local/go/src/pkg/syscall/asm_linux_386.s:14 +0x5
syscall.Mkdir(0x83a2f18, 0x2, 0x2, 0x806255e, 0x83a2f1c, ...)
/usr/local/go/src/pkg/syscall/zerrors_linux_386.go:2225 +0x80
server.init()
So this would be in line with my main question about interpreting stack trace. The -1218 is still there, but now there are these.
The asm_linux_386.s has this at line 14:
MOVL 4(SP), AX // syscall entry
I found the zerrors_linux_386.go too, but there's no line 2225. The file stops long before that line.
It's already reported and accepted as Issue 5243.
Program execution
A package with no imports is initialized by assigning initial values
to all its package-level variables and then calling any package-level
function with the name and signature of
func init()
defined in its source. A package-scope or file-scope identifier with
name init may only be declared to be a function with this signature.
Multiple such functions may be defined, even within a single source
file; they execute in unspecified order.
Within a package, package-level variables are initialized, and
constant values are determined, according to order of reference: if
the initializer of A depends on B, A will be set after B. Dependency
analysis does not depend on the actual values of the items being
initialized, only on their appearance in the source. A depends on B if
the value of A contains a mention of B, contains a value whose
initializer mentions B, or mentions a function that mentions B,
recursively. It is an error if such dependencies form a cycle. If two
items are not interdependent, they will be initialized in the order
they appear in the source, possibly in multiple files, as presented to
the compiler. Since the dependency analysis is done per package, it
can produce unspecified results if A's initializer calls a function
defined in another package that refers to B.
An init function cannot be referred to from anywhere in a program. In
particular, init cannot be called explicitly, nor can a pointer to
init be assigned to a function variable.
If a package has imports, the imported packages are initialized before
initializing the package itself. If multiple packages import a package
P, P will be initialized only once.
The importing of packages, by construction, guarantees that there can
be no cyclic dependencies in initialization.
A complete program is created by linking a single, unimported package
called the main package with all the packages it imports,
transitively. The main package must have package name main and declare
a function main that takes no arguments and returns no value.
func main() { … }
Program execution begins by initializing the main package and then
invoking the function main. When the function main returns, the
program exits. It does not wait for other (non-main) goroutines to
complete.
Package initialization—variable initialization and the invocation of
init functions—happens in a single goroutine, sequentially, one
package at a time. An init function may launch other goroutines, which
can run concurrently with the initialization code. However,
initialization always sequences the init functions: it will not start
the next init until the previous one has returned.
As your program begins execution, it initializes package variables and executes init functions. Adding package variables is going to change the initialization. It looks like the initialization failed in debugComponent.go on something related to server.init(). The negative line number is probably a bug.
Without the source code, it's hard to say more.

Resources