I know go routine can have a few blocking actions, wonder if a goroutine can call a user-defined blocking function like a regular function. A user-defined blocking function has a few steps like, step1, step2.
In another word, I would like to find out whether we can have nested blocking calls in a go routine.
UPDATE:
Original intention was to find the stack size used by goroutine, especially with nested blocking calls. Sorry for the confusion. Thanks to the answer and comments, I created the following function that has 100,000 goroutines, it took 782MB of virtual memory and 416MB of Resident memory on my Ubuntu desktop. It evens out to be 78KB of memory for each go routine stack. Is this a correct statement?
package main
import (
"fmt"
"time"
)
func f(a int) {
x := f1(a);
f2(x);
}
func f1(a int) int {
r := step("1a", a);
r = step("1b", r);
return 1000 * a;
}
func f2(a int) {
r := step("2a", a);
r = step("2b", r);
}
func step(a string, b int) int{
fmt.Printf("%s %d\n", a, b);
time.Sleep(1000 * time.Second)
return 10 * b;
}
func main() {
for i := 0; i < 100000; i++ {
go f(i);
}
//go f(20);
time.Sleep(1000 * time.Second)
}
I believe you're right, though I'm unsure of the relationship between "virtual" and "resident" memory it's possible there's some overlap.
Some things to consider: you're running 100,000 it appears, not 10,000.
The stack itself might contain things like the strings used for the printfs, method parameters, etc.
As of go 1.2 the default stack size (per go routine) is 8KB which may explain some of it.
As of go 1.3 it also uses an exponentially increasing stack size, but I doubt that's the problem you're running into.
Short answer yes.
A goroutine is a "lightweight thread", that means it can do stuff independently from other code in your program. It's almost as if you started a new program, but you can communicate with your other code using the constructs golang provides (channels, locks, etc.).
P.S. Once the main function ends, all goroutines are killed (that's why you need the time.Sleep() in the example)
Here's the quick example (won't run in the golang playground because of their constraints):
package main
import (
"fmt"
"time"
)
func saySomething(a, b func()){
a()
b()
}
func foo() {
fmt.Println("foo")
}
func bar() {
fmt.Println("bar")
}
func talkForAWhile() {
for {
saySomething(foo, bar)
}
}
func main() {
go talkForAWhile()
time.Sleep(1 * time.Second)
}
Related
My project is a login-register web server that consists of multiple files and uses another package in which the Manager struct is defined.
Overview of my files:
my-package/
main.go
handlers.go
...
I have a variable: var M *Manager declared in main.go before definition of main() and it is assigned inside main():
var M *Manager
func main() {
...
M = InitManager(...)
...
}
handleLogin(...) and handleRegister(...) are functions defined in handlers.go that use the M variable:
func handleRegister(...){
...
fmt.Println("M:", M)
M.Log1("logging informations...")
...
}
func handleLogin(...) {
...
fmt.Println("M:", M)
M.GetAccount(login)
...
}
When I go to /loginor /register and the appropriate handle function is triggered it displays: M: <nil>
To find out something more I modified main() as shown below:
var M *Manager
func main() {
...
go func() { // for debugging
for {
fmt.Println("main() goloop1: M:", M)
time.Sleep(time.Second / 2)
}
}()
M = InitManager(...)
go func() { // for debugging
for {
fmt.Println("main() goloop2: M:", M)
time.Sleep(time.Second / 2)
}
}()
...
}
and the output:
main() goloop2: M: &{...data as expected...}
main() goloop1: M: <nil>
main() goloop2: M: &{...data as expected...}
main() goloop1: M: <nil>
...
My question are:
How do pointers work then if one pointer gives out two values?
How to fix my issue and properly plan code (if that was the cause) to avoid this in future?
Per the Go Memory Model, the provided code writes and reads M without proper synchronization, which is a data race and leads to undefined behavior (see icza's comment).
The compiler "assumes" that the code is properly synchronized (this is the responsibility of the developer) and so it is "allowed" to assume that M is never modified inside the infinite loops, so it may use a copy in a given register or stack memory location over and over, leading to the surprising output.
You may use a sync.Mutex to protect every access to the global *Manager variable M, as in this modified code.
Also beware of variable shadowing! It is possible to write M := f() instead of M = f(), resulting in an unrelated local variable, not affecting the global variable.
My solution:
I have added init.go:
my-package/
main.go
handlers.go
...
init.go
I moved global variables like M and CONFIG_MAP into the init.go file:
package main
import ...
var CONFIG_MAP = LoadConfig()
var M *asrv.Manager = InitManager()
func LoadConfig() map[string]string {
// return map from 'conf.json' file
}
func InitManager() *asrv.Manager {
// return Manager configured with CONFIG_MAP
// other functions also use CONFIG_MAP that is the reason why it is global
}
func init() {
LoadTemplatesFiles() // load templates and assign value
// to a variable declared in templates.go
}
This way handler functions (handleLoginGet etc.) in handlers.go could properly read M.
This fix just made my program work and I still don't know what is the proper way of handling this type of situation, that is why I added more info under EDIT in my question.
Is there a way to obtain the total number of finalizers registered using runtime.SetFinalizer and which have not yet run?
We are considering adding a struct with a registered finalizer to some of our products to release memory allocated using malloc, and the object could potentially have a relatively high allocation rate. It would be nice if we could monitor the number of finalizers, to make sure that they do not pile up and trigger out-of-memory errors (like they tend to with other garbage collectors).
(I'm aware that explicit deallocation would avoid this problem, but we cannot change the existing code, which does not call a Close function or something like that.)
You can keep keep a count of these objects by incrementing and decrementing a unexported package variable when a new object is created and finalized, respectively.
For example:
package main
import (
"fmt"
"runtime"
"sync/atomic"
)
var totalObjects int32
func TotalObjects() int32 {
return atomic.LoadInt32(&totalObjects)
}
type Object struct {
p uintptr // C allocated pointer
}
func NewObject() *Object {
o := &Object{
}
// TODO: perform other initializations
atomic.AddInt32(&totalObjects, 1)
runtime.SetFinalizer(o, (*Object).finalizer)
return o
}
func (o *Object) finalizer() {
atomic.AddInt32(&totalObjects, -1)
// TODO: perform finalizations
}
func main() {
fmt.Println("Total objects:", TotalObjects())
for i := 0; i < 100; i++ {
_ = NewObject()
runtime.GC()
}
fmt.Println("Total objects:", TotalObjects())
}
https://play.golang.org/p/n35QABBIcj
It's possible to make a wrapper on runtime.SetFinalizer which does the counting for you. Of course, it's a question of using it everywhere where you use SetFinalizer.
In case this is problematic, you can also modify SetFinalizer source code directly, but that requires a modified Go compiler.
Atomic integers are used as SetFinalizer may be called on different threads, and otherwise a counter may not be accurate as without those a race condition could possibly occur. Golang guarantees that finalizers are called from a single goroutine, so it's not needed for inner function.
https://play.golang.org/p/KKCH2UwTFYw
package main
import (
"fmt"
"reflect"
"runtime"
"sync/atomic"
)
var finalizersCreated int64
var finalizersRan int64
func SetFinalizer(obj interface{}, finalizer interface{}) {
finType := reflect.TypeOf(finalizer)
funcType := reflect.FuncOf([]reflect.Type{finType.In(0)}, nil, false)
f := reflect.MakeFunc(funcType, func(args []reflect.Value) []reflect.Value {
finalizersRan++
return reflect.ValueOf(finalizer).Call([]reflect.Value{args[0]})
})
runtime.SetFinalizer(obj, f.Interface())
atomic.AddInt64(&finalizersCreated, 1)
}
func main() {
v := "a"
SetFinalizer(&v, func(a *string) {
fmt.Println("Finalizer ran")
})
fmt.Println(finalizersRan, finalizersCreated)
runtime.GC()
fmt.Println(finalizersRan, finalizersCreated)
}
Below is an example of how to use mutex lock in order to safely access data. How would I go about doing the same with the use of CSP (communication sequential processes) instead of using mutex lock’s and unlock’s?
type Stack struct {
top *Element
size int
sync.Mutex
}
func (ss *Stack) Len() int {
ss.Lock()
size := ss.size
ss.Unlock()
return size
}
func (ss *Stack) Push(value interface{}) {
ss.Lock()
ss.top = &Element{value, ss.top}
ss.size++
ss.Unlock()
}
func (ss *SafeStack) Pop() (value interface{}) {
ss.Lock()
size := ss.size
ss.Unlock()
if size > 0 {
ss.Lock()
value, ss.top = ss.top.value, ss.top.next
ss.size--
ss.Unlock()
return
}
return nil
}
If you actually were to look at how Go implements channels, you'd essentially see a mutex around an array with some additional thread handling to block execution until the value is passed through. A channel's job is to move data from one spot in memory to another with ease. Therefore where you have locks and unlocks, you'd have things like this example:
func example() {
resChan := make(int chan)
go func(){
resChan <- 1
}()
go func(){
res := <-resChan
}
}
So in the example, the first goroutine is blocked after sending the value until the second goroutine reads from the channel.
To do this in Go with mutexes, one would use sync.WaitGroup which will add one to the group on setting the value, then release it from the group and the second goroutine will lock and then unlock the value.
The oddities in your example are 1 no goroutines, so it's all happening in a single main goroutine and the locks are being used more traditionally (as in c thread like) so channels won't really accomplish anything. The example you have would be considered an anti-pattern, like the golang proverb says "Don't communicate by sharing memory, share memory by communicating."
I wish I could do something like the following code with golang:
package main
import (
"fmt"
"time"
)
func getA() (int) {
fmt.Println("getA: Calculating...")
time.Sleep(300 * time.Millisecond)
fmt.Println("getA: Done!")
return 100
}
func getB() (int) {
fmt.Println("getB: Calculating...")
time.Sleep(400 * time.Millisecond)
fmt.Println("getB: Done!")
return 200
}
func main() {
A := go getA()
B := go getB()
C := A + B // waits till getA() and getB() return
fmt.Println("Result:", C)
fmt.Println("All done!")
}
More specifically, I wish Go could handle concurrency behind the scene.
This might be a bit off topic, but I am curious what people think of having such implicit concurrency support. Is it worth to put some effort on it? and what are the potential difficulties and drawbacks?
Edit:
To be clear, the question is not about "what Go is doing right now?", and not even "how it is implemented?" though I appreciate #icza post on what exactly we should expect from Go as it is right now. The question is why it does not or is not capable of returning values, and what are the potential complications of doing that?
Back to my simple example:
A := go getA()
B := go getB()
C := A + B // waits till getA() and getB() return
I do not see any issues concerning the scope of variables, at least from the syntax point of view. The scope of A, B, and C is clearly defined by the block they are living inside (in my example the scope is the main() function). However, a perhaps more legitimate question would be if those variables (here A and B) are "ready" to read from? Of course they should not be ready and accessible till getA() and getB() are finished. In fact, this is all I am asking for: The compiler could implement all the bell and whistles behind the scene to make sure the execution will be blocked till A and B are ready to consume (instead of forcing the programmer to explicitly implement those waits and whistles using channels).
This could make the concurrent programming much simpler, especially for the cases where computational tasks are independent of each other. The channels still could be used for explicit communication and synchronization, if really needed.
But this is very easily and idiomatically doable. The language provides the means: Channel types.
Simply pass a channel to the functions, and have the functions send the result on the channel instead of returning them. Channels are safe for concurrent use.
Only one goroutine has access to the value at any given time. Data races cannot occur, by design.
For more, check out the question: If I am using channels properly should I need to use mutexes?
Example solution:
func getA(ch chan int) {
fmt.Println("getA: Calculating...")
time.Sleep(300 * time.Millisecond)
fmt.Println("getA: Done!")
ch <- 100
}
func getB(ch chan int) {
fmt.Println("getB: Calculating...")
time.Sleep(400 * time.Millisecond)
fmt.Println("getB: Done!")
ch <- 200
}
func main() {
cha := make(chan int)
chb := make(chan int)
go getA(cha)
go getB(chb)
C := <-cha + <-chb // waits till getA() and getB() return
fmt.Println("Result:", C)
fmt.Println("All done!")
}
Output (try it on the Go Playground):
getB: Calculating...
getA: Calculating...
getA: Done!
getB: Done!
Result: 300
All done!
Note:
The above example can be implemented with a single channel too:
func main() {
ch := make(chan int)
go getA(ch)
go getB(ch)
C := <-ch + <-ch // waits till getA() and getB() return
fmt.Println("Result:", C)
fmt.Println("All done!")
}
Output is the same. Try this variant on the Go Playground.
Edit:
The Go spec states that the return values of such functions are discarded. More on this: What happens to return value from goroutine.
What you propose bleeds from multiple wounds. Each variable in Go has a scope (in which they can be referred to). Accessing variables do not block. Execution of statements or operators may block (e.g. Receive operator or Send statement).
Your proposal:
go A := getA()
go B := getB()
C := A + B // waits till getA() and getB() return
What is the scope of A and B? 2 reasonable answer would be a) from the go statement or after the go statement. Either way we should be able to access it after the go statement. After the go statement they would be in scope, reading/writing their values should not block.
But then if C := A + B would not block (because it is just reading a variable), then either
a) at this line A and B should already be populated which means the go statement would need to wait for getA() to complete (but then it defeats the purpose of go statement)
b) or else we would need some external code to synchronize but then again we don't gain anything (just make it worse compared to the solution with channels).
Do not communicate by sharing memory; instead, share memory by communicating.
By using channels, it is crystal clear what (may) block and what not. It is crystal clear that when a receive from the channel completes, the goroutine is done. And it gives us the mean to execute the receive whenever we're want to (at the point when its value is needed and we're willing to wait for it), and also the mean to check if the value is ready without blocking (comma-ok idiom).
I need to wait until x.Addr is being updated but it seems the for loop is not run. I suspect this is due the go scheduler and I'm wondering why it works this way or if there is any way I can fix it(without channels).
package main
import "fmt"
import "time"
type T struct {
Addr *string
}
func main() {
x := &T{}
go update(x)
for x.Addr == nil {
if x.Addr != nil {
break
}
}
fmt.Println("Hello, playground")
}
func update(x *T) {
time.Sleep(2 * time.Second)
y := ""
x.Addr = &y
}
There are two (three) problems with your code.
First, you are right that there is no point in the loop at which you give control to the scheduler and such it can't execute the update goroutine. To fix this you can set GOMAXPROCS to something bigger than one and then multiple goroutines can run in parallel.
(However, as it is this won't help as you pass x by value to the update function which means that the main goroutine will never see the update on x. To fix this problem you have to pass x by pointer. Now obsolete as OP fixed the code.)
Finally, note that you have a data race on Addr as you are not using atomic loads and stores.