I am running a program with go 1.4 and I am trying to pass a large struct to a go function.
go ProcessImpression(network, &logImpression, campaign, actualSpent, partnerAccount, deviceId, otherParams)
I get this error:
runtime.newproc: function arguments too large for new goroutine
I have moved to pass by reference which helps but I am wondering if there is some way to pass large structs in a go function.
Thanks,
No, none I know of.
I don't think you should be too aggressive tuning to avoid copying, but it appears from the source that this error is emitted when parameters exceed the usable stack space for a new goroutine, which should be kilobytes. The copying overhead is real at that point, especially if this isn't the only time these things are copied. Perhaps some struct either explicitly is larger than expected thanks to a large struct member (1kb array rather than a slice, say) or indirectly. If not, just using a pointer as you have makes sense, and if you're worried about creating garbage, recycle the structs pointed to using sync.Pool.
I was able to fix this issue by changing the arguments from
func doStuff(prev, next User)
to
func doStuff(prev, next *User)
The answer from #twotwotwo in here is very helpful.
Got this issue at processing list of values([]BigType) of big struct:
for _, stct := range listBigStcts {
go func(stct BigType) {
...process stct ...
}(stct) // <-- error occurs here
}
Workaround is to replace []BigType with []*BigType
Related
Both functions are doing the same task which is initializing "Data struct". what are the Pros or Cons of each function? e.g. the function should unmarshal a big JSON file.
package main
type Data struct {
i int
}
func funcp(c chan *Data) {
var t *Data
t = <-c //receive
t.i = 10
}
func funcv(c chan Data) {
var t Data
t.i = 20
c <- t //send
}
func main() {
c := make(chan Data)
cp := make(chan *Data)
var t Data
go funcp(cp)
cp <- &t //send
println(t.i)
go funcv(c)
t = <- c //receive
println(t.i)
}
Link to Go Playground
The title of your question seems wrong. You are asking not about swapping things but rather about whether to send a pointer to some data or a copy of some data. More importantly, the overall thrust of your question lacks crucial information.
Consider two analogies:
Which is better, chocolate ice cream, or strawberry? That's probably a matter of opinion, but at least both with serve similar purposes.
Which is better, a jar of glue or a brick of C4? That depends on whether you want to build something, or blow something up, doesn't it?
If you send a copy of data through a channel, the receiver gets ... a copy. The receiver does not have access to the original. The copying process may take some time, but the fact that the receiver does not have to share access may speed things up. So this is something of an opinion, and if your question is about which is faster, well, you'll have to benchmark it. Be sure to benchmark the real problem, and not a toy example, because benchmarks on toy examples don't translate to real-world performance.
If you send a pointer to data through a channel, the receiver gets a copy of the pointer, and can therefore modify the original data. Copying the pointer is fast, but the fact that the receiver has to share access may slow things down. But if the receiver must be able to modify the data, you have no choice. You must use a tool that works, and not one that does not.
In your two functions, one generates values (funcv) so it does not have to send pointers. That's fine, and gives you the option. The other (funcp) receives objects but wants to update them so it must receive a pointer to the underlying object. That's fine too, but it means that you are now communicating by sharing (the underlying data structure), which requires careful coordination.
I saw the issue on Github which says sync.Pool should be used only with pointer types, for example:
var TPool = sync.Pool{
New: func() interface{} {
return new(T)
},
}
Does it make sense? What about return T{} and which is the better choice, why?
The whole point of sync.Pool is to avoid (expensive) allocations. Large-ish buffers, etc. You allocate a few buffers and they stay in memory, available for reuse. Hence the use of pointers.
But here you'll be copying the values on every step, defeating the purpose. (Assuming your T is a "normal" struct and not something like SliceHeader)
It is not necessary. In most cases it should be a pointer as you want to share an object, not to make copies.
In some use cases this can be a non pointer type, like an id of some external resource. I can imagine a pool of paths (mounted disk drives) represented with strings where some large file operations are being conducted.
Is it a good idea to generate a secure random hex string until the process succeeds?
All examples I've come across show that if rand.Read returns error, we should panic, os.Exit(1) or return empty string and the error.
I need my program to continue to function in case of such errors and wait until a random string is generated. Is it a good idea to loop until the string is generated, any pitfalls with that?
import "crypto/rand"
func RandomHex() string {
var buf [16]byte
for {
_, err := rand.Read(buf[:])
if err == nil {
break
}
}
return hex.EncodeToString(buf[:])
}
No. It may always return an error in certain contexts.
Example: playground: don't use /dev/urandom in crypto/rand
Imagine that a machine does not have the source that crypto/rand gets data from or the program runs in a context that doesn't have access to that source. In that case you might consider having the program return that error in a meaningful way rather than spin.
More explicitly, if you are serious in your use of crypto/rand then consider writing RandomHex such that it is exceptionally clear to the caller that it is meant for security contexts (possibly rename it) and return the error from RandomHex. The calling function needs to handle that error and let the user know that something is very wrong. For example in a rest api, I'd expect that error to surface to the request handler, fail & return a 500 at that point, and log a high severity error.
Is it a good idea to loop until the string is generated,
That depends. Probably yes.
any pitfalls with that?
You discard the random bytes read on error. And this in a tight loop.
This may drain you entropy source (depending on the OS) faster than
it can be filled.
Instead of an unbound infinite loop: Break after n rounds and give up.
Graceful degradation or stopping is best: If your program is stuck in
an endless loop it is also not "continue"ing.
Reading the code from the bufio package I've found such things:
// fill reads a new chunk into the buffer.
func (b *Reader) fill() {
...
if b.w >= len(b.buf) {
panic("bufio: tried to fill full buffer")
}
...
}
At the same time the Effective Go section about panic contains the next paragraph:
This is only an example but real library functions should avoid panic.
If the problem can be masked or worked around, it's always better to
let things continue to run rather than taking down the whole program.
So, I wonder, is the problem with a particular buffered reader so important to cause the panic call in the standard library code?
It may be questionable, but consider: fill is a private method, and b.w and b.buf are private fields. If the condition that causes the panic is ever true, it's due to a logic error in the implementation of bufio. Since it should never really be possible to get into that state in the first place (a "can't happen" condition), we don't really know how we got there, and it's unclear how much other state got corrupted before the problem was detected and what, if anything, the user can do about it. In that kind of situation, a panic seems reasonable.
Hi I am using the qml library for go to create UI's. I am trying to learn how to pass information from the UI (qml) to go to then "do something" with.
QML is working if it is just a UI. I can run that fine when I do:
func main() {
if len(os.Args) != 2 {
fmt.Fprintf(os.Stderr, "usage: %s <qml file>\n", os.Args[0])
os.Exit(1)
}
if err := qml.Run(run); err != nil {
fmt.Fprintf(os.Stderr, "error: %v\n", err)
os.Exit(1)
}
}
func run() error {
engine := qml.NewEngine()
engine.On("quit", func() { os.Exit(0) })
component, err := engine.LoadFile(os.Args[1])
if err != nil {
return err
}
window := component.CreateWindow(nil)
window.Show()
window.Wait()
return nil
}
However when I add some code, to try and "learn" something from the UI I get the run time error:
panic: runtime error: cgo argument has Go pointer to Go pointer
The code I am adding is:
window.On("visibleChanged", func(visible bool) {
if (visible) {
fmt.Println("Width:", window.Int("width"))
}
})
I am running "go version go1.6 darwin/amd64" on OSX El Capitan
Any ideas why? A google suggests this was an error in Go 1.6 Beta, but I am running the latest stable version (installed a couple of days ago).
If it's not a simple fix, can someone explain "why" this is occuring?
The problem is that when C code stores a Go pointer (in this case, a pointer to your callback function), the garbage collector cannot track that pointer in the C code, and may garbage collect the memory that the pointer is pointing to if no Go code is referencing it. This will cause the C code to crash when it attempts to access that memory. All the runtime knows is that the C code retained the pointer (that's why it can panic), but it doesn't know what the C code will do with it later and for how long it will keep it.
To avoid this, the trick used by most libraries was to hold on to a pointer in Go as well (e.g. in a global map), to ensure that the memory is protected from the garbage collector. go-qml uses this trick as well. This trick works, but the compiler and garbage collector have no idea that it does, they cannot verify that you're not making a mistake (e.g. deleting the Go pointer, while the C code still has its pointer).
With Go 1.6, the Go developers decided to be very strict about this, and they no longer allow C code to retain a Go pointer at all. However, if you disable this check, everything will still work in this case, because go-qml implements the trick correctly (it may break in the future however, e.g. if go implements a moving garbage collector).
Here's the issue about it: https://github.com/go-qml/qml/issues/170
Side note: In this specific case, what gets passed to C is a pointer to an interface{}, which itself contains a pointer to the function. That's why you get the error "cgo argument has Go pointer to Go pointer". The reason this isn't allowed is that it's more difficult to protect these pointers from the GC for the duration of the C call, and it's not worth it, so it's forbidden instead (https://github.com/golang/go/issues/12416#issuecomment-136473697).
However, even if this were allowed, the code would still be violating the rule about C code keeping a copy of the Go pointer.
This isn't actually a problem in Go 1.6, since it doesn't implement a moving garbage collector, but the rules were made so that it can be implemented later.
If you're just playing around, I suggest trying with go 1.5.3. Go 1.6 introduced a different set of constraints on pointers to memory when using cgo, a more restrictive set, and it's possible some go packages that were designed for the older version of go are now breaking a go rule or two.
If this is the case, getting the older package to work with go 1.6, where C is allowed to call go closures, could be harder to fix. But I don't have first hand experience with that yet.
Thanks for all the help here. I have written up a beginners tutorial on using QML with Go. It can be viewed here. I will continually update it if I run into any more errors and find fixes. Thanks everyone for your help. QML/GO is an awesome combination.
https://golog.co/blog/article/Using_Go_with_QML_part_1