I'm new to Go and I am also kinda having a hard time reading the documentation. Can anyone help explain what is wrong with the code below
package main
import (
"fmt"
)
func main() {
c := make(chan int)
for i := 0; i < 10; i++ {
go func() {
add(c)
}()
}
close(c)
for v := range c {
fmt.Println(v)
}
}
func add(c chan<- int) {
c<-123
}
Based on the suggestion from icza :
package main
import (
"fmt"
"sync"
)
func main() {
c := make(chan int)
var wgConsumer sync.WaitGroup
wgConsumer.Add(1)
go func() {
defer wgConsumer.Done()
for v := range c {
fmt.Println(v)
}
}()
var wg sync.WaitGroup
wg.Add(10)
for i := 0; i < 10; i++ {
go func() {
add(c)
defer wg.Done()
}()
}
wg.Wait()
close(c)
wgConsumer.Wait()
}
func add(c chan<- int) {
c <- 123
}
Output:
123
123
123
123
123
123
123
123
123
123
I tried some synchronization techniques to share state between goroutines and find out that incorrect variant (without sync) works slowly than same program with mutex.
Given the code:
package main
import (
"sync"
"time"
"fmt"
)
func main() {
var wg sync.WaitGroup
var mu sync.Mutex
hash := make(map[string]string)
hash["test"] = "string"
num := 40000000
wg.Add(num)
start := time.Now()
for i := 0; i < num; i++ {
go func() {
mu.Lock()
_, _ = hash["test"]
mu.Unlock()
wg.Done()
}()
}
wg.Wait()
fmt.Println(time.Since(start))
}
Perform on my laptop with 8 HT cores for 9-10 seconds.
But if just remove sync it works for 11-12 seconds:
package main
import (
"sync"
"time"
"fmt"
)
func main() {
var wg sync.WaitGroup
hash := make(map[string]string)
hash["test"] = "string"
num := 40000000
wg.Add(num)
start := time.Now()
for i := 0; i < num; i++ {
go func() {
_, _ = hash["test"]
wg.Done()
}()
}
wg.Wait()
fmt.Println(time.Since(start))
}
A synced version is faster and unitizes CPU much higher. Question is why?
My thought is about how goroutines scheduled and overhead for context switching because of the more GOMAXPROCS the greater the gap between these two versions. But I can't explain the real reason for that that happens under the hood of the scheduler.
I'm thinking start 1000 goroutines at the same time using for loop in Golang.
The problem is: I have to make sure that every goroutine has been executed.
Is it possible using channels to help me make sure of that?
The structure is kinda like this:
func main {
for i ... {
go ...
ch?
ch?
}
As #Andy mentioned You can use sync.WaitGroup to achieve this. Below is an example. Hope the code is self-explanatory.
package main
import (
"fmt"
"sync"
"time"
)
func dosomething(millisecs int64, wg *sync.WaitGroup) {
defer wg.Done()
duration := time.Duration(millisecs) * time.Millisecond
time.Sleep(duration)
fmt.Println("Function in background, duration:", duration)
}
func main() {
arr := []int64{200, 400, 150, 600}
var wg sync.WaitGroup
for _, n := range arr {
wg.Add(1)
go dosomething(n, &wg)
}
wg.Wait()
fmt.Println("Done")
}
To make sure the goroutines are done and collect the results, try this example:
package main
import (
"fmt"
)
const max = 1000
func main() {
for i := 1; i <= max; i++ {
go f(i)
}
sum := 0
for i := 1; i <= max; i++ {
sum += <-ch
}
fmt.Println(sum) // 500500
}
func f(n int) {
// do some job here and return the result:
ch <- n
}
var ch = make(chan int, max)
In order to wait for 1000 goroutines to finish, try this example:
package main
import (
"fmt"
"sync"
)
func main() {
wg := &sync.WaitGroup{}
for i := 0; i < 1000; i++ {
wg.Add(1)
go f(wg, i)
}
wg.Wait()
fmt.Println("Done.")
}
func f(wg *sync.WaitGroup, n int) {
defer wg.Done()
fmt.Print(n, " ")
}
I would suggest that you follow a pattern. Concurrency and Channel is Good but if you use it in a bad way, your program might became even slower than expected. The simple way to handle multiple go-routine and channel is by a worker pool pattern.
Take a close look at the code below
// In this example we'll look at how to implement
// a _worker pool_ using goroutines and channels.
package main
import "fmt"
import "time"
// Here's the worker, of which we'll run several
// concurrent instances. These workers will receive
// work on the `jobs` channel and send the corresponding
// results on `results`. We'll sleep a second per job to
// simulate an expensive task.
func worker(id int, jobs <-chan int, results chan<- int) {
for j := range jobs {
fmt.Println("worker", id, "started job", j)
time.Sleep(time.Second)
fmt.Println("worker", id, "finished job", j)
results <- j * 2
}
}
func main() {
// In order to use our pool of workers we need to send
// them work and collect their results. We make 2
// channels for this.
jobs := make(chan int, 100)
results := make(chan int, 100)
// This starts up 3 workers, initially blocked
// because there are no jobs yet.
for w := 1; w <= 3; w++ {
go worker(w, jobs, results)
}
// Here we send 5 `jobs` and then `close` that
// channel to indicate that's all the work we have.
for j := 1; j <= 5; j++ {
jobs <- j
}
close(jobs)
// Finally we collect all the results of the work.
for a := 1; a <= 5; a++ {
<-results
}
}
This simple example is taken from here . Also the results channel can help you keep track of all the go routines executing the jobs including failure notice.
I found strange occupy heap when convert byte[] to string with below code
package main
import (
"bytes"
"fmt"
"net/http"
_ "net/http/pprof"
"strings"
"time"
)
var (
c = make(chan int, 500000)
)
func main() {
go func() {
http.ListenAndServe(":8080", nil)
}()
f := func(ss []string) {
fmt.Println(ss)
time.Sleep(time.Millisecond)
<-c
}
for {
c <- 1
bs := bytes.NewBufferString("A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z").Bytes()
fmt.Println(bs) // will raise memory leak after marked as comment???
s := string(bs)
ss := strings.Split(s, ",")
go f(ss)
}
}
without fmt.Println(bs) will gradually exhausting memory.
with fmt.Println(bs) work fine. i can't understand what happened? i'm worked with version go1.9.2 darwin/amd64
No, there is no memory leak:
You are using 500000 concurrent goroutines, you just need to limit (reduce) the number of concurrent goroutines, e.g.:
c := make(chan int, runtime.NumCPU())
Try this ( and see the end of this edit):
package main
import (
"bytes"
"fmt"
"runtime"
"strings"
"time"
)
func main() {
c := make(chan int, runtime.NumCPU())
for {
c <- 1
bs := bytes.NewBufferString("A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z").Bytes()
s := string(bs)
ss := strings.Split(s, ",")
go func(ss []string) {
fmt.Println(ss)
time.Sleep(time.Millisecond)
<-c
}(ss)
}
}
Your Code:
package main
import (
"bytes"
"fmt"
"net/http"
_ "net/http/pprof"
"strings"
"time"
)
var (
c = make(chan int, 500000)
)
func main() {
go func() {
http.ListenAndServe(":8080", nil)
}()
f := func(ss []string) {
fmt.Println(ss)
time.Sleep(time.Millisecond)
<-c
}
for {
c <- 1
bs := bytes.NewBufferString("A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z").Bytes()
// fmt.Println(bs) // will raise memory leak after marked as comment???
s := string(bs)
ss := strings.Split(s, ",")
go f(ss)
}
}
It reaches steady state after a while and even reduced Memory usage:
// Mem CPU time:
// 5,464,208K 0:1:20
// 5,468,208K 0:2:20
// 5,469,608K 0:3:20
// 5,469,844K 0:4:20
// 5,469,844K 0:5:20
// 5,469,848K 0:6:20
// 5,469,848K 0:7:20 fixed
// 5,469,848K 0:8:20 fixed
// 5,469,616K 0:9:20 reduced
I am trying to understand concurrency and goroutines, and had a couple questions about the following experimental code:
Why does it create a memory leak? I thought that a return at the end of the goroutine would allow memory associated with it to get cleaned up.
Why do my loops almost never reach 999? In fact, when I output to a file and study the output, I notice that it rarely prints integers in double digits; the first time it prints "99" is line 2461, and for "999" line 6120. This behavior is unexpected to me, which clearly means I don't really understand what is going on with goroutine scheduling.
Disclaimer:
Be careful running the code below, it can crash your system if you don't stop it after a few seconds!
CODE
package main
import (
"fmt"
"sync"
)
func main() {
var wg sync.WaitGroup
for {
// spawn four worker goroutines
spawnWorkers(4, wg)
// wait for the workers to finish
wg.Wait()
}
}
func spawnWorkers(max int, wg sync.WaitGroup) {
for n := 0; n < max; n++ {
wg.Add(1)
go func() {
defer wg.Done()
f(n)
return
}()
}
}
func f(n int) {
for i := 0; i < 1000; i++ {
fmt.Println(n, ":", i)
}
}
Thanks to Tim Cooper, JimB, and Greg for their helpful comments. The corrected version of the code is posted below for reference.
The two fixes were to pass in the WaitGroup by reference, which fixed the memory leak, and to pass n correctly into the anonymous goroutine, and
package main
import (
"fmt"
"sync"
)
func main() {
var wg sync.WaitGroup
for {
// spawn four worker goroutines
spawnWorkers(4,&wg)
// wait for the workers to finish
wg.Wait()
}
}
func spawnWorkers(max int, wg *sync.WaitGroup) {
for n := 0; n < max; n++ {
wg.Add(1)
go func(n int) {
defer wg.Done()
f(n)
return
}(n)
}
}
func f(n int) {
for i := 0; i < 1000; i++ {
fmt.Println(n, ":", i)
}
}