I am struggling to understand where I should close my channels.
This bit of code takes about 0.7 seconds:
options := [3]string{"0", "1", "2"}
str := fmt.Sprintf("%6d ", id)
for j := 0; j < 40000; j++ {
str += options[rand.Intn(3)]
}
str += "\n"
Adding an io.Writestring does not make a difference to the time, so the problem is this bit.
I want roughly 100,000 records like these so I thought to put in a goroutine.
func main() {
file, _ := os.Create("myfile.txt")
ch := make(chan string)
for i := 0; i < 100000; i++ {
go generate(i, ch)
}
counter := 0
for result := range ch {
counter++
io.WriteString(file, result)
if counter == 100000 {
close(ch)
}
}
file.Close()
}
func generate(id int, c chan string) {
options := [3]string{"0", "1", "2"}
str := fmt.Sprintf("%6d ", id)
for j := 0; j < 40000; j++ {
str += options[rand.Intn(3)]
}
str += "\n"
c <- str
}
From what I understand is that I am closing the channel on the receiver side, which is not ideal? Also, this way all 100,000 should be sent to goroutines first, before I can receive any. Can I send requests to generate a record and start receiving at the same time?
Using a counter to close your channel is not a good practise.You can make use of sync.WaitGroup. This allows you to have a better control on when to close your channel:
func main() {
var wg sync.WaitGroup
ch := make(chan string)
file, _ := os.Create("myfile.txt")
for i := 0; i < 100000; i++ {
wg.Add(1)
go func(i int) {
defer wg.Done()
options := [3]string{"0", "1", "2"}
str := fmt.Sprintf("%6d ", i)
for j := 0; j < 40000; j++ {
str += options[rand.Intn(3)]
}
str += "\n"
ch <- str
}(i)
}
go func() {
wg.Wait()
close(ch)
}()
for result := range ch {
io.WriteString(file, result)
}
file.Close()
}
See if this solves your problem..
func main() {
file, _ := os.Create("myfile.txt")
ch := make(chan string)
wg := new(sync.WaitGroup)
for i := 0; i < 100000; i++ {
wg.Add(1)
go generate(i, ch)
}
go func(){wg.Wait();close(ch)}()
counter := 0
for result := range ch {
counter++
io.WriteString(file, result)
}
file.Close()
}
func generate(id int, c chan string) {
options := [3]string{"0", "1", "2"}
str := fmt.Sprintf("%6d ", id)
for j := 0; j < 40000; j++ {
str += options[rand.Intn(3)]
}
str += "\n"
c <- str
wg.Done()
}
Related
When not using goroutineļ¼500 * 100000000 times plus one
// 1m12.2857724s
start := time.Now()
for i := 0; i < 500; i++ {
res := 0
for j := 0; j < 100000000; j++ {
res++
}
}
duration := time.Since(start)
fmt.Println(duration)
When using goroutine, 10 goroutines execute 50 * 100000000 times plus one
// 1m12.0174541s
start := time.Now()
ch := make(chan bool)
for i := 0; i < 10; i++ {
go func(ch chan bool) {
for i := 0; i < 50; i++ {
res := 0
for j := 0; j < 100000000; j++ {
res++
}
}
ch <- true
}(ch)
<- ch
}
duration := time.Since(start)
fmt.Println(duration)
Why use goroutine does not save time
The ch channel is unbuffered. You launch a goroutine and send a value on the channel at the end, and right after that, before launching another goroutine you receive from it. This is a blocking operation. You won't start a new goroutine until one is finished. You gain nothing compared to the first solution.
One "solution" is to make the channel buffered, and only start receiving from it once all goroutines have been launched:
ch := make(chan bool, 10)
for i := 0; i < 10; i++ {
go func(ch chan bool) {
for i := 0; i < 50; i++ {
res := 0
for j := 0; j < 100000000; j++ {
res++
}
}
ch <- true
}(ch)
}
for i := 0; i < 10; i++ {
<-ch
}
This will result in almost 4x speedup on my computer (4 CPU cores).
A better, more idiomatic way to wait for all goroutines is to use sync.WaitGroup:
var wg sync.WaitGroup
for i := 0; i < 10; i++ {
wg.Add(1)
go func() {
defer wg.Done()
for i := 0; i < 50; i++ {
res := 0
for j := 0; j < 100000000; j++ {
res++
}
}
}()
}
wg.Wait()
Also note that using multiple goroutines is only worth it if the task they do is "significant", see:
Matrix multiplication with goroutine drops performance
Vectorise a function taking advantage of concurrency
You could effectively launch the 10 goroutines, and receive from each after they complete their work, in a simple way.
Check the CPU usage of your code, and this one
func add(ch chan int) {
r := 0
for i := 0; i < 50; i++ {
for j := 0; j < 100000000; j++ {
r++
}
}
ch <- r
}
func main() {
start := time.Now()
ch := make(chan int)
res := 0
for i := 0; i < 10; i++ {
go add(ch)
}
for i := 0; i < 10; i++ {
res += <-ch
}
duration := time.Since(start)
fmt.Println(duration, res)
}
Output
3.747405005s 50000000000
That will successfully launch the 10 goroutines, each of them perform their work and return the results down the channel once they're done.
Suppose I have a helper function helper(n int) which returns a slice of integers of variable length. I would like to run helper(n) in parallel for various values of n and collect the output in one big slice. My first attempt at this is the following:
package main
import (
"fmt"
"golang.org/x/sync/errgroup"
)
func main() {
out := make([]int, 0)
ch := make(chan int)
go func() {
for i := range ch {
out = append(out, i)
}
}()
g := new(errgroup.Group)
for n := 2; n <= 3; n++ {
n := n
g.Go(func() error {
for _, i := range helper(n) {
ch <- i
}
return nil
})
}
if err := g.Wait(); err != nil {
panic(err)
}
close(ch)
// time.Sleep(time.Second)
fmt.Println(out) // should have the same elements as [0 1 0 1 2]
}
func helper(n int) []int {
out := make([]int, 0)
for i := 0; i < n; i++ {
out = append(out, i)
}
return out
}
However, if I run this example I do not get all 5 expected values, instead I get
[0 1 0 1]
(If I uncomment the time.Sleep I do get all five values, [0 1 2 0 1], but this is not an acceptable solution).
It seems that the problem with this is that out is being updated in a goroutine, but the main function returns before it is done updating.
One thing that would work is using a buffered channel of size 5:
func main() {
ch := make(chan int, 5)
g := new(errgroup.Group)
for n := 2; n <= 3; n++ {
n := n
g.Go(func() error {
for _, i := range helper(n) {
ch <- i
}
return nil
})
}
if err := g.Wait(); err != nil {
panic(err)
}
close(ch)
out := make([]int, 0)
for i := range ch {
out = append(out, i)
}
fmt.Println(out) // should have the same elements as [0 1 0 1 2]
}
However, although in this simplified example I know what the size of the output should be, in my actual application this is not known a priori. Essentially what I would like is an 'infinite' buffer such that sending to the channel never blocks, or a more idiomatic way to achieve the same thing; I've read https://blog.golang.org/pipelines but wasn't able to find a close match to my use case. Any ideas?
In this version of the code, the execution is blocked until ch is closed.
ch is always closed at the end of a routine that is responsible to push into ch. Because the program pushes to ch in a routine, it is not needed to use a buffered channel.
package main
import (
"fmt"
"golang.org/x/sync/errgroup"
)
func main() {
ch := make(chan int)
go func() {
g := new(errgroup.Group)
for n := 2; n <= 3; n++ {
n := n
g.Go(func() error {
for _, i := range helper(n) {
ch <- i
}
return nil
})
}
if err := g.Wait(); err != nil {
panic(err)
}
close(ch)
}()
out := make([]int, 0)
for i := range ch {
out = append(out, i)
}
fmt.Println(out) // should have the same elements as [0 1 0 1 2]
}
func helper(n int) []int {
out := make([]int, 0)
for i := 0; i < n; i++ {
out = append(out, i)
}
return out
}
Here is the fixed version of the first code, it is convoluted but demonstrates the usage of sync.WaitGroup.
package main
import (
"fmt"
"sync"
"golang.org/x/sync/errgroup"
)
func main() {
out := make([]int, 0)
ch := make(chan int)
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
for i := range ch {
out = append(out, i)
}
}()
g := new(errgroup.Group)
for n := 2; n <= 3; n++ {
n := n
g.Go(func() error {
for _, i := range helper(n) {
ch <- i
}
return nil
})
}
if err := g.Wait(); err != nil {
panic(err)
}
close(ch)
wg.Wait()
// time.Sleep(time.Second)
fmt.Println(out) // should have the same elements as [0 1 0 1 2]
}
func helper(n int) []int {
out := make([]int, 0)
for i := 0; i < n; i++ {
out = append(out, i)
}
return out
}
How to print numbers in order using goroutine after all goroutines bing emited?
Here is the code which print numbers in random:
func main() {
var wg sync.WaitGroup
wg.Add(10)
for i := 1; i <= 10; i++ {
go func(i int) {
defer wg.Done()
fmt.Printf("i = %d\n", i)
}(i)
}
wg.Wait()
Emit goroutines in order to print numbers like below, it's not the solution I want.
func main() {
var wg sync.WaitGroup
for i := 1; i <= 10; i++ {
wg.Add(1)
go func(i int) {
defer wg.Done()
fmt.Printf("i = %d\n", i)
}(i)
wg.Wait()
}
I want all goroutines being emited, and after then make them print numbers in order.
func main() {
wg.Add(10)
count := 10
cBegins := make([]chan interface{}, count)
for i := range cBegins {
cBegins[i] = make(chan interface{})
}
go func() {
cBegins[0] <- struct {}{}
}()
for i := 0; i < count; i++ {
go func(i int) {
defer wg.Done()
<-cBegins[i]
fmt.Printf("i = %d\n", i)
if i < 9 {
cBegins[i+1] <- struct {}{}
}
}(i)
}
wg.Wait()
}
I am new to golang, and I am puzzled with this deadlock (run here)
package main
import (
"fmt"
"runtime"
"time"
)
func main() {
c := make(chan string)
work := make(chan int, 1)
clvl := runtime.NumCPU()
count := 0
for i := 0; i < clvl; i++ {
go func(i int) {
for jdId := range work {
time.Sleep(time.Second * 1)
c <- fmt.Sprintf("done %d", jdId)
}
}(i)
}
go func() {
for i := 0; i < 10; i++ {
work <- i
}
close(work)
}()
for resp := range c {
fmt.Println(resp, count)
count += 1
}
}
You never close c, so your for range loop waits forever. Close it like this:
var wg sync.WaitGroup
for i := 0; i < clvl; i++ {
wg.Add(1)
go func(i int) {
defer wg.Done()
for jdId := range work {
time.Sleep(time.Second * 1)
c <- fmt.Sprintf("done %d", jdId)
}
}(i)
}
go func() {
for i := 0; i < 10; i++ {
work <- i
}
close(work)
wg.Wait()
close(c)
}()
EDIT: Fixed the panic, thanks Crast
Why is there a deadlock even tho I just pass one and get one output from the channel?
package main
import "fmt"
import "math/cmplx"
func max(a []complex128, base int, ans chan float64, index chan int) {
fmt.Printf("called for %d,%d\n",len(a),base)
maxi_i := 0
maxi := cmplx.Abs(a[maxi_i]);
for i:=1 ; i< len(a) ; i++ {
if cmplx.Abs(a[i]) > maxi {
maxi_i = i
maxi = cmplx.Abs(a[i])
}
}
fmt.Printf("called for %d,%d and found %f %d\n",len(a),base,maxi,base+maxi_i)
ans <- maxi
index <- base+maxi_i
}
func main() {
ans := make([]complex128,128)
numberOfSlices := 4
incr := len(ans)/numberOfSlices
tmp_val := make([]chan float64,numberOfSlices)
tmp_index := make([]chan int,numberOfSlices)
for i,j := 0 , 0; i < len(ans); j++{
fmt.Printf("From %d to %d - %d\n",i,i+incr,len(ans))
go max(ans[i:i+incr],i,tmp_val[j],tmp_index[j])
i = i+ incr
}
//After Here is it stops deadlock
maximumFreq := <- tmp_index[0]
maximumMax := <- tmp_val[0]
for i := 1; i < numberOfSlices; i++ {
tmpI := <- tmp_index[i]
tmpV := <- tmp_val[i]
if(tmpV > maximumMax ) {
maximumMax = tmpV
maximumFreq = tmpI
}
}
fmt.Printf("Max freq = %d",maximumFreq)
}
For those reading this question and perhaps wondering why his code failed here's an explanation.
When he constructed his slice of channels like so:
tmp_val := make([]chan float64,numberOfSlices)
He made slice of channels where every index was to the channels zero value. A channels zero value is nil since channels are reference types and a nil channel blocks on send forever and since there is never anything in a nil channel it will also block on recieve forever. Thus you get a deadlock.
When footy changes his code to construct each channel individually using
tmp_val[i] = make(chan float64)
in a loop he constructs non-nil channels and everything is good.
I was wrong in making of the chan. Should have done
numberOfSlices := 4
incr := len(ans)/numberOfSlices
var tmp_val [4]chan float64
var tmp_index [4]chan int
for i := range tmp_val {
tmp_val[i] = make(chan float64)
tmp_index[i] = make(chan int)
}
for i,j := 0 , 0; i < len(ans); j++{
fmt.Printf("From %d to %d [j:%d] - %d\n",i,i+incr,j,len(ans))
go maximumFunc(ans[i:i+incr],i,tmp_val[j],tmp_index[j])
i = i+ incr
}