I've got an array X and I'm trying to range over it and delete some elements if a condition is true.
Doing this without any concurrency works fine, but when I try to do this concurrently I get the error:
"slice bounds out of range"
func main() {
X := make([][]int32, 10)
for i := 0; i < 10; i++ {
X[i] = []int32{int32(i), int32(i)}
}
ch := make(chan int, 20)
var wg sync.WaitGroup
fmt.Println(X)
for i := 0; i < 20; i++ {
wg.Add(1)
go func() {
defer wg.Done()
for index := range X {
// check some condition
if X[index][0]%2 == 0 && index < len(X)-1 {
ch <- index
break
}
}
}()
}
for {
ind, ok := <-ch
if ok {
X = append(X[:ind], X[ind+1:]...)
} else {
fmt.Println("closed chan")
break
}
}
wg.Wait()
fmt.Println(X)
}
https://play.golang.org/p/YeLamAU5_Rt
Is there any way to use the indexes and send it over a channel then delete the corresponding elements from different goroutines?
First: why are you trying to delete like this? Is there a larger problem you are trying to solve?
If this is how you want to do it, then:
You are using a buffered channel, so the deleting goroutine can be deleting items from the slice while the other goroutine is working on it, causing unexpected results. Also, you are not closing the channel, so the for-loop never terminates.
Use an unbuffered channel, and run the deletion loop in a separate goroutine. Then close the channel after wg.Wait to terminate the deleting goroutine.
Copy the elements that you want to keep to the beginning of the slice. Drop the goroutines.
keep := 0
for index := range X {
if X[index][0]%2 == 0 && index < len(X)-1 {
continue
}
X[keep] = X[index]
keep++
}
X = X[:keep]
Run it on the playground.
Related
I am reading values that are put into a channel ch via an infinite for. I would like some way to signal if a value has been read and operated upon (via the sq result) and add it to some sort of counter variable upon success. That way I have a way to check if my channel has been exhausted so that I can properly exit my infinite for loop.
Currently it is incrementing regardless if a value was read, thus causing it to exit early when the counter == num. I only want it to count when the value has been squared.
EDIT: Another approach I have tested is to receive the ok val out of the channel upon reading and setting val and then check if !ok { break }. However I receive a deadlock panic since the for did has not properly break. Example here: https://go.dev/play/p/RYNtTix2nm2
package main
import "fmt"
func main() {
num := 5
// Buffered channel with 5 values.
ch := make(chan int, num)
defer close(ch)
for i := 0; i < num; i++ {
go func(val int) {
fmt.Printf("Added value: %d to the channel\n", val)
ch <- val
}(i)
}
// Read from our channel infinitely and increment each time a value has been read and operated upon
counter := 0
for {
// Check our counter and if its == num then break the infinite loop
if counter == num {
break
}
val := <-ch
counter++
go func(i int) {
// I'd like to verify a value was read from ch & it was processed before I increment the counter
sq := i * i
fmt.Println(sq)
}(val)
}
}
let me try to help you in figuring out the issue.
Reading issue
The latest version of the code you put in the question is working except when you're about to read values from the ch channel. I mean with the following code snippet:
go func(i int) {
// I'd like to verify a value was read from ch & it was processed before I increment the counter
sq := i * I
fmt.Println(sq)
}(val)
In fact, it's not needed to spawn a new goroutine for each read. You can consume the messages as soon as they arrived in the ch channel. This is possible due to writing done inside goroutines. Thanks to them, the code can go ahead and reach the reading phase without being blocked.
Buffered vs unbuffered
In this scenario, you used a buffered channel with 5 slots for data. However, if you're relying on the buffered channel you should signal when you finish sending data to it. This is done with a close(ch) invocation after all of the Go routines finish their job. If you use an unbuffered channel it's fine to invoke defer close(ch) next to the channel initialization. In fact, this is done for cleanup and resource optimization tasks. Back to your example, you can change the implementation to use unbuffered channels.
Final Code
Just to recap, the two small changes that you've to do are:
Use an unbuffered channel instead of a buffered one.
Do Not use a Go routine when reading the messages from the channel.
Please be sure to understand exactly what's going on. Another tip can be to issue the statement: fmt.Println("NumGoroutine:", runtime.NumGoroutine()) to print the exact number of Go routines running in that specific moment.
The final code:
package main
import (
"fmt"
"runtime"
)
func main() {
num := 5
// Buffered channel with 5 values.
ch := make(chan int)
defer close(ch)
for i := 0; i < num; i++ {
go func(val int) {
fmt.Printf("Added value: %d to the channel\n", val)
ch <- val
}(i)
}
fmt.Println("NumGoroutine:", runtime.NumGoroutine())
// Read from our channel infinitely and increment each time a value has been read and operated upon
counter := 0
for {
// Check our counter and if its == num then break the infinite loop
if counter == num {
break
}
val := <-ch
counter++
func(i int) {
// I'd like to verify a value was read from ch & it was processed before I increment the counter
sq := i * i
fmt.Println(sq)
}(val)
}
}
Let me know if this helps you, thanks!
package main
import "fmt"
func main() {
c := make(chan int)
done := make(chan bool)
go func() {
for i := 0; i < 10; i++ {
c <- i
}
close(c)
}()
go func() {
for i := range c {
fmt.Println(i)
done <- true
}
close(done)
}()
for i := 0; i < 10; i++ {
<-done
}
}
In this example, the done channel is used to signal that a value has been read from the c channel. After each value is read from c, a signal is sent on the done channel. The main function blocks on the done channel, waiting for a signal before continuing. This ensures that all values from c have been processed before the program terminates.
I want to make X number of goroutines to update CountValue using parallelism (numRoutines are as much as how many CPU you have).
Solution 1:
func count(numRoutines int) (countValue int) {
var mu sync.Mutex
k := func(i int) {
mu.Lock()
defer mu.Unlock()
countValue += 5
}
for i := 0; i < numRoutines; i++ {
go k(i)
}
It becomes a data race and the returned countValue = 0.
Solution 2:
func count(numRoutines int) (countValue int) {
k := func(i int, c chan int) {
c <- 5
}
c := make(chan int)
for i := 0; i < numRoutines; i++ {
go k(i, c)
}
for i := 0; i < numRoutines; i++ {
countValue += <- c
}
return
}
I did a benchmark test on it and doing a sequential addition would work faster than using goroutines. I think it's because I have two for loops here as when I put countValue += <- c inside the first for loop, the code runs faster.
Solution 3:
func count(numRoutines int) (countValue int) {
var wg sync.WaitGroup
c := make(chan int)
k := func(i int) {
defer wg.Done()
c <- 5
}
for i := 0; i < numShards; i++ {
wg.Add(1)
go k(i)
}
go func() {
for i := range c {
countValue += i
}
}()
wg.Wait()
return
}
Still a race count :/
Is there any way better to do this?
There definitely is a better way to safely increment a variable: using sync/atomic:
import "sync/atomic"
var words int64
k := func() {
_ = atomic.AddInt64(&words, 5) // increment atomically
}
Using a channel basically eliminates the need for a mutex, or takes away the the risk of concurrent access to the variable itself, and a waitgroup here is just a bit overkill
Channels:
words := 0
done := make(chan struct{}) // or use context
ch := make(chan int, numRoutines) // buffer so each routine can write
go func () {
read := 0
for i := range ch {
words += 5 // or use i or something
read++
if read == numRoutines {
break // we've received data from all routines
}
}
close(done) // indicate this routine has terminated
}()
for i := 0; i < numRoutines; i++ {
ch <- i // write whatever value needs to be used in the counting routine on the channel
}
<- done // wait for our routine that increments words to return
close(ch) // this channel is no longer needed
fmt.Printf("Counted %d\n", words)
As you can tell, the numRoutines no longer is the number of routines, but rather the number of writes on the channel. You can move that to individual routines, still:
for i := 0; i < numRoutines; i++ {
go func(ch chan<- int, i int) {
// do stuff here
ch <- 5 * i // for example
}(ch, i)
}
WaitGroup:
Instead of using a context that you can cancel, or a channel, you can use a waitgroup + atomic to get the same result. The easiest way IMO to do so is to create a type:
type counter struct {
words int64
}
func (c *counter) doStuff(wg *sync.WaitGroup, i int) {
defer wg.Done()
_ = atomic.AddInt64(&c.words, i * 5) // whatever value you need to add
}
func main () {
cnt := counter{}
wg := sync.WaitGroup{}
wg.Add(numRoutines) // create the waitgroup
for i := 0; i < numRoutines; i++ {
go cnt.doStuff(&wg, i)
}
wg.Wait() // wait for all routines to finish
fmt.Println("Counted %d\n", cnt.words)
}
Fix for your third solution
As I mentioned in the comment: your third solution is still causing a race condition because the channel c is never closed, meaning the routine:
go func () {
for i := range c {
countValue += i
}
}()
Never returns. The waitgroup also only ensures that you've sent all values on the channel, but not that the countValue has been incremented to its final value. The fix would be to either close the channel after wg.Wait() returns so the routine can return, and add a done channel that you can close when this last routine returns, and add a <-done statement before returning.
func count(numRoutines int) (countValue int) {
var wg sync.WaitGroup
c := make(chan int)
k := func(i int) {
defer wg.Done()
c <- 5
}
for i := 0; i < numShards; i++ {
wg.Add(1)
go k(i)
}
done := make(chan struct{})
go func() {
for i := range c {
countValue += i
}
close(done)
}()
wg.Wait()
close(c)
<-done
return
}
This adds some clutter, though, and IMO is a bit messy. It might just be easier to just move the wg.Wait() call to a routine:
func count(numRoutines int) (countValue int) {
var wg sync.WaitGroup
c := make(chan int)
// add wg as argument, makes it easier to move this function outside of this scope
k := func(wg *sync.WaitGroup, i int) {
defer wg.Done()
c <- 5
}
wg.Add(numShards) // increment the waitgroup once
for i := 0; i < numShards; i++ {
go k(&wg, i)
}
go func() {
wg.Wait()
close(c) // this ends the loop over the channel
}()
// just iterate over the channel until it is closed
for i := range c {
countValue += i
}
// we've added all values to countValue
return
}
Im studying Golang now on my freetime and I am trying sample exams online to test what i learned,
I came about this coding exam task but I cant seem to make it work/run without a crash,
im getting fatal error: all goroutines are asleep - deadlock! error, can anybody help what I am doing wrong here?
func executeParallel(ch chan<- int, done chan<- bool, functions ...func() int) {
ch <- functions[1]()
done <- true
}
func exampleFunction(counter int) int {
sum := 0
for i := 0; i < counter; i++ {
sum += 1
}
return sum
}
func main() {
expensiveFunction := func() int {
return exampleFunction(200000000)
}
cheapFunction := func() int {return exampleFunction(10000000)}
ch := make(chan int)
done := make(chan bool)
go executeParallel(ch, done, expensiveFunction, cheapFunction)
var isDone = <-done
for result := range ch {
fmt.Printf("Result: %d\n", result)
if isDone {
break;
}
}
}
Your executeParallel function will panic if less than 2 functions are provided - and will only run the 2nd function:
ch <- functions[1]() // runtime panic if less then 2 functions
I think it should look more like this: running all input functions in parallel and grabbing the first result.
for _, fn := range functions {
fn := fn // so each iteration/goroutine gets the proper value
go func() {
select {
case ch <- fn():
// first (fastest worker) wins
default:
// other workers results are discarded (if reader has not read results yet)
// this ensure we don't leak goroutines - since reader only reads one result from channel
}
}()
}
As such there's no need for a done channel - as we just need to read the one and only (quickest) result:
ch := make(chan int, 1) // big enough to capture one result - even if reader is not reading yet
executeParallel(ch, expensiveFunction, cheapFunction)
fmt.Printf("Result: %d\n", <-ch)
https://play.golang.org/p/skXc3gZZmRn
package main
import "fmt"
func executeParallel(ch chan<- int, done chan<- struct{}, functions ...func() int) {
// Only execute the second function [1], if available.
if len(functions) > 1 {
ch <- functions[1]()
}
// Close the done channel to signal the for-select to break and the main returns.
close(done)
}
// example returns the number of iterations for [0..counter-1]
func example(counter int) int {
sum := 0
for i := 0; i < counter; i++ {
sum += 1
}
return sum
// NOTE(SS): This function could just return "counter-1"
// to avoid the unnecessary calculation done above.
}
func main() {
var (
cheap = func() int { return example(10000000) }
expensive = func() int { return example(200000000) }
ch = make(chan int)
done = make(chan struct{})
)
// executeParallel takes ch, done channel followed by variable
// number of functions where on the second i.e., indexed 1
// function is executed on a separated goroutine which is then
// sent to ch channel which is then received by the for-select
// reciever below i.e., <-ch is the receiver.
go executeParallel(ch, done, expensive, cheap)
for {
select {
// Wait for something to be sent to done or the done channel
// to be closed.
case <-done:
return
// Keep receiving from ch (if something is sent to it)
case result := <-ch:
fmt.Println("Result:", result)
}
}
}
I have commented on the code so that it's understandable. As you didn't the actual question the logic could be still wrong.
I am practicing golang by doing coding problems on LeetCode. I am trying to solve a simple sudoku puzzle (it just validates the board). No rows with same digits, no columns with same digits, no 3x3 blocks with same digits. I am trying to use concurrency for learning Go Routines/Channels/Etc...
I can't get the waitgroup to finalize
import (
"sync"
"fmt"
)
func isValidSlice(slice []byte, results chan<- bool, wg *sync.WaitGroup) {
fmt.Println(slice)
seen := make(map[byte]bool)
for _,val := range(slice) {
if seen[val] {
if val != '.'{
results <- false
defer wg.Done()
return
}
} else {
seen[val] = true
}
}
results <- true
defer wg.Done()
}
func isValidSudoku(board [][]byte) bool {
// Channel to receive solution
c := make(chan bool)
// Number of routines that will run (9 for rows, 9 for cols, 9 for 3x3 blocks)
var wg sync.WaitGroup
// Check every row
for x:= 0; x < 9; x++{
wg.Add(1)
go isValidSlice(append([]byte{}, board[x]...), c, &wg)
}
for y:= 0; y < 9; y++{
wg.Add(1)
go isValidSlice(append([]byte{}, board[0:9][y]...), c, &wg)
}
// Check every 3x3 block
for x:= 0; x <= 6; x += 3{
for y := 0; y <= 6; y += 3{
block_digits := append([]byte{}, board[x][y:y+3]...)
block_digits = append(block_digits, board[x+1][y:y+3]...)
block_digits = append(block_digits, board[x+2][y:y+3]...)
wg.Add(1)
go isValidSlice(block_digits, c, &wg)
}
}
fmt.Println("got here")
wg.Wait()
fmt.Println("never got here")
for result := range c{
if !result{
return false
}
}
return true
}
I'm expecting the wg.Wait() lock to release and code to move forward. Then I'm expecting to one of the results in the channel to be false and return false if so. Otherwise, after all elements in the channel are traverse and no false was found, I would expect a True.
Your goroutines cannot call wg.Done() because they all wait to add their value in the channel. But since you only consume the values from the channel after wg.Wait(), all goroutines but one never get to call wg.Done().
You actually don't need a WaitGroup, just remove it.
Other comments:
You should move defer wg.Done() to the first line of isValidSlice. Calling defer on the last line of your function does not make much sense.
You only need a WaitGroup if you want to close the channel properly, you can do that in an additional goroutine, see below for an example on how to do it.
func isValidSudoku(board [][]byte) bool {
// ...
fmt.Println("got here")
go func(){
wg.Wait()
close(c)
}()
fmt.Println("never got here")
for result := range c{
if !result{
go func(){
for _ := range c {
}
}()
return false
}
}
return true
}
// _Closing_ a channel indicates that no more values
// will be sent on it. This can be useful to communicate
// completion to the channel's receivers.
package main
import "fmt"
// In this example we'll use a `jobs` channel to
// communicate work to be done from the `main()` goroutine
// to a worker goroutine. When we have no more jobs for
// the worker we'll `close` the `jobs` channel.
func main() {
jobs := make(chan int, 5)
done := make(chan bool)
// Here's the worker goroutine. It repeatedly receives
// from `jobs` with `j, more := <-jobs`. In this
// special 2-value form of receive, the `more` value
// will be `false` if `jobs` has been `close`d and all
// values in the channel have already been received.
// We use this to notify on `done` when we've worked
// all our jobs.
for i := 1; i <= 3; i++ {
go func() {
for {
j, more := <-jobs
if more {
fmt.Println("received job", j)
} else {
fmt.Println("received all jobs")
done <- true
return
}
}
}()
}
// This sends 3 jobs to the worker over the `jobs`
// channel, then closes it.
j := 0
for {
j++
jobs <- j
fmt.Println("sent job", j)
}
close(jobs)
fmt.Println("sent all jobs")
// We await the worker using the
// [synchronization](channel-synchronization) approach
// we saw earlier.
<-done
}
https://play.golang.org/p/x28R_g8ftS
What I'm trying to do is get all the responses from a paginated url endpoint. jobs is a channel storing the page number. I have a function in if more{} checking for empty reponse and I have
done <- true
return
I thought this would close the go routine.
But, the page generator for{j++; jobs <- j} is causing it to get stuck in a loop. Any idea how this can be resolved?
By definition a for loop without conditions is an infinite loop. Unless you put some logic to break this infinite loop, you'll never get out of it.
In your playground your comment implies that you want to send 3 jobs. You should change your for loop accordingly:
for j := 0; j < 3; j++ {
jobs <- j
fmt.Println("sent job", j)
}
This is a simplified version of a worker.. Its not very useful for production level traffic, but should serve as a simple example, there are tons of them :-)
package main
import (
"log"
"sync"
)
type worker struct {
jobs chan int
wg *sync.WaitGroup
}
func main() {
w := worker{
jobs: make(chan int, 5), // I only want to work on 5 jobs at any given time
wg: new(sync.WaitGroup),
}
for i := 0; i < 3; i++ {
w.wg.Add(1)
go func(i int) {
defer w.wg.Done()
w.jobs <- i
}(i)
}
// wait in the background so that i can move to line 34 and start consuming my job queue
go func() {
w.wg.Wait()
close(w.jobs)
}()
for job := range w.jobs {
log.Println("Got job, I should do something with it", job)
}
}
This was I was looking for. I have a number generator in an infinite while loop. And the program exits on some condition, in this example, it is on the j value, but it can also be something else.
https://play.golang.org/p/Ud4etTjrmx
package main
import "fmt"
func jobs(job chan int) {
i := 1
for {
job <- i
i++
}
}
func main() {
jobsChan := make(chan int, 5)
done := false
j := 0
go jobs(jobsChan)
for !done {
j = <-jobsChan
if j < 20 {
fmt.Printf("job %d\n", j)
} else {
done = true
}
}
}