Is it possible in go:
Suppose, I have 3 concurrent routines which can send integers to each other. Now, suppose both concurrent routine 2 & 3 sends an integer to concurrent routine 1. Is it possible in go that routine 1 takes both two values and process it farther ? To make it clear, I have following code:
package main
import "rand"
func Routine1(command12 chan int, response12 chan int, command13 chan int, response13 chan int ) {
for i := 0; i < 10; i++ {
i := rand.Intn(100)
if i%2 == 0 {
command12 <- i
}
if i%2 != 0 {
command13 <- i
}
print(<-response13, " 1st\n");
}
close(command12)
}
func Routine2(command12 chan int, response12 chan int, command23 chan int, response23 chan int) {
for i := 0; ; i++ {
x, open := <-command12
if !open {
return;
}
print(x , " 2nd\n");
y := rand.Intn(100)
if i%2 == 0 {
command12 <- y
}
if i%2 != 0 {
command23 <- y
}
}
}
func Routine3(command13 chan int, response13 chan int, command23 chan int, response23 chan int) {
for i := 0; ; i++ {
x, open := <-command13
if !open {
return;
}
print(x , " 3nd\n");
y := rand.Intn(100)
response23 <- y
}
}
func main() {
command12 := make(chan int)
response12 := make(chan int)
command13 := make(chan int)
response13 := make(chan int)
command23 := make(chan int)
response23 := make(chan int)
go Routine1(command12, response12,command13, response13 )
Routine2(command12, response12,command23, response23)
Routine3(command13, response13,command23, response23 )
}
Here, in this example routine 1 can send an int to routine 2 or 3. I assume it is routine 3. Now suppose, routine 3 also send an int to routine 2. Is it possible for routine 2 to take those two values and process further (dynamic concurrent routines)? Can any body help to modify this program accordingly.
I hate abstract examples, anyway I will do my best to answer your question.
Is it possible in go that routine 1 takes both two values and process it farther?
What do you want to archive? Inside routine1 you can either do:
// Read exactly one command from routine2 as well as exactly
// one command from routine3
cmd1 := <-command12
cmd2 := <-command13
// Process the pair of the two commands here
OR
// Process a single command only, which was either sent by routine2
// or by routine3. If there are commands available on both channels
// (command12 and command13) the select statement chooses a branch
// fairly.
select {
case cmd1 := <-command12:
// process command from routine 2
case cmd2 := <-command13
// process command from routine 3
}
I hope that will answer your question. Also note that Go channels support multiple-writers (as well as mutliple readers) by default. So, it might be enough to use exactly one input channel per goroutine. For example, routine1 might only read commands from a channel called command1, but both routine2 and routine3 might use the same command1 channel to send messages to routine1.
Another common idiom in Go is to pass a reply channel as part of the message. For example:
type Command struct {
Cmd string
Reply chan-> int
}
func routine2() {
reply := make(chan int)
command1 <- Command{"doSomething", reply}
status := <-reply
}
func routine1() {
cmd <- command1;
// process cmd.Cmd
cmd.Reply <- 200 // SUCCESS (status code)
}
Depending on your actual problem, this might simplify your program drastically :)
It's not possible in GO, I mean creating concurrent channels.
Related
What I am trying to do is to essentially create a buffered channel effect using goroutines.
What I have is a function called Elem that looks as follows:
func Elem(in, out chan int) {
for {
buf := <-in
out <- buf
}
}
Originally the purpose of this function was to avoid blocking of a goroutine when it attempts to send an int value to another goroutine via a channel, and that goroutine is not receiving the value. So instead Elem acts as an intermediary which collects the value from the channel and blocks on behalf of the sending goroutine (assuming that no one is trying to receive the value).
Coming back to the point of the question, is there a way to chain an arbitrary number of these Elem goroutines together. So essentially what I am looking for is a function that takes 2 channels and an integer. The 2 channels represent the input channel for the 1st Elem goroutine and the output channel for the last Elem goroutine in the pipeline.
func ChainElems(in, out chan int, numOfElemsToChain int) {
for i:=0; i<numOfElemsToChain; i++ { //this doesn't have to be a loop I just thought that this is how you would go about it
//chaining logic
}
}
What I did is something like this:
package main
func Elem(in, out chan int) {
for {
buf := <-in
out <- buf
}
}
func main() {
in := make(chan int)
out1 := make(chan int)
out2 := make(chan int)
out3 := make(chan int)
go Elem(in, out1)
go Elem(out1, out2)
go Elem(out2, out3)
in <- 3
in <- 6
in <- 7
fmt.Println(<-out3)
fmt.Println(<-out3)
fmt.Println(<-out3)
}
However this is sort of "hardcoded" (only for 3 goroutines). I want to make this process a function for any number of Elems that I want to chain. Is this possible?
In the code hereunder, I don't understand why the "Worker" methods seem to exit instead of pulling values from the input channel "in" and processing them.
I had assumed they would only return after having consumed all input from the input channel "in" and processing them
package main
import (
"fmt"
"sync"
)
type ParallelCallback func(chan int, chan Result, int, *sync.WaitGroup)
type Result struct {
i int
val int
}
func Worker(in chan int, out chan Result, id int, wg *sync.WaitGroup) {
for item := range in {
item *= item // returns the square of the input value
fmt.Printf("=> %d: %d\n", id, item)
out <- Result{item, id}
}
wg.Done()
fmt.Printf("%d exiting ", id)
}
func Run_parallel(n_workers int, in chan int, out chan Result, Worker ParallelCallback) {
wg := sync.WaitGroup{}
for id := 0; id < n_workers; id++ {
fmt.Printf("Starting : %d\n", id)
wg.Add(1)
go Worker(in, out, id, &wg)
}
wg.Wait() // wait for all workers to complete their tasks
close(out) // close the output channel when all tasks are completed
}
const (
NW = 4
)
func main() {
in := make(chan int)
out := make(chan Result)
go func() {
for i := 0; i < 100; i++ {
in <- i
}
close(in)
}()
Run_parallel(NW, in, out, Worker)
for item := range out {
fmt.Printf("From out : %d: %d", item.i, item.val)
}
}
The output is
Starting : 0
Starting : 1
Starting : 2
Starting : 3
=> 3: 0
=> 0: 1
=> 1: 4
=> 2: 9
fatal error: all goroutines are asleep - deadlock!
fatal error: all goroutines are asleep - deadlock!
The full error shows where each goroutine is "stuck". If you run this in the playground, it will even show you the line number. That made it easy for me to diagnose.
Your Run_parallel runs in the main groutine, so before main can read from out, Run_parallel must return. Before Run_parallel can return, it must wg.Wait(). But before the workers call wg.Done(), they must write to out. That's what causes a deadlock.
One solution is simple: just run Run_parallel concurrently in its own Goroutine.
go Run_parallel(NW, in, out, Worker)
Now, main ranges over out, waiting on outs closure to signal completion. Run_parallel waits for the workers with wg.Wait(), and the workers will range over in. All the work will get done, and the program won't end until it's all done. (https://go.dev/play/p/oMrgH2U09tQ)
Solution :
Run_parallel has to run in it’s own goroutine:
package main
import (
"fmt"
"sync"
)
type ParallelCallback func(chan int, chan Result, int, *sync.WaitGroup)
type Result struct {
id int
val int
}
func Worker(in chan int, out chan Result, id int, wg *sync.WaitGroup) {
defer wg.Done()
for item := range in {
item *= 2 // returns the double of the input value (Bogus handling of data)
out <- Result{id, item}
}
}
func Run_parallel(n_workers int, in chan int, out chan Result, Worker ParallelCallback) {
wg := sync.WaitGroup{}
for id := 0; id < n_workers; id++ {
wg.Add(1)
go Worker(in, out, id, &wg)
}
wg.Wait() // wait for all workers to complete their tasks
close(out) // close the output channel when all tasks are completed
}
const (
NW = 8
)
func main() {
in := make(chan int)
out := make(chan Result)
go func() {
for i := 0; i < 10; i++ {
in <- i
}
close(in)
}()
go Run_parallel(NW, in, out, Worker)
for item := range out {
fmt.Printf("From out [%d]: %d\n", item.id, item.val)
}
println("- - - All done - - -")
}
Alternative formulation of the solution:
In that alternative formulation , it is not necessary to start Run_parallel as a goroutine (it triggers its own goroutine).
I prefer that second solution, because it automates the fact that Run_parallel() has to run parallel to the main function. Also, for the same reason it's safer, less error-prone (no need to remember to run Run_parallel with the go keyword).
package main
import (
"fmt"
"sync"
)
type ParallelCallback func(chan int, chan Result, int, *sync.WaitGroup)
type Result struct {
id int
val int
}
func Worker(in chan int, out chan Result, id int, wg *sync.WaitGroup) {
defer wg.Done()
for item := range in {
item *= 2 // returns the double of the input value (Bogus handling of data)
out <- Result{id, item}
}
}
func Run_parallel(n_workers int, in chan int, out chan Result, Worker ParallelCallback) {
go func() {
wg := sync.WaitGroup{}
defer close(out) // close the output channel when all tasks are completed
for id := 0; id < n_workers; id++ {
wg.Add(1)
go Worker(in, out, id, &wg)
}
wg.Wait() // wait for all workers to complete their tasks *and* trigger the -differed- close(out)
}()
}
const (
NW = 8
)
func main() {
in := make(chan int)
out := make(chan Result)
go func() {
defer close(in)
for i := 0; i < 10; i++ {
in <- i
}
}()
Run_parallel(NW, in, out, Worker)
for item := range out {
fmt.Printf("From out [%d]: %d\n", item.id, item.val)
}
println("- - - All done - - -")
}
I am new to go and I am trying to learn some basic use of signal functions in goroutines. I have an infinite for loop in go. Through this for loop, I pass values to a goroutine through a channel. I also have a threshold value after which I will like to stop sending values indefinitely to the goroutine (i.e. close the channel). When the threshold value is reached, I will like to break the for loop. Following is what I have tried so far.
In this particular example, thresholdValue = 10 and I would like to print values from 0 , ..., 9 and then stop.
I followed this post on medium and this post on stackoverflow. I picked elements from these posts which I could use.
This is what I have done at the present. In the main function of my code, I purposefully make the for loop an infinite loop. My main intention is to learn how to have the goroutine readValues() take the threshold value and then stop transmission of values indefinitely in the channel.
package main
import (
"fmt"
)
func main() {
ch := make(chan int)
quitCh := make(chan struct{}) // signal channel
thresholdValue := 10 //I want to stop the incoming data to readValues() after this value
go readValues(ch, quitCh, thresholdValue)
for i:=0; ; i++{
ch <- i
}
}
func readValues(ch chan int, quitCh chan struct{}, thresholdValue int) {
for value := range ch {
fmt.Println(value)
if (value == thresholdValue){
close(quitCh)
}
}
}
The goroutine in my code still misses the threshold. I will appreciate any direction as to how I should proceed from here.
to show good faith, this is the program rewritten.
package main
import (
"log"
"sync"
"time"
)
func main() {
ch := make(chan int, 5) // capacity increased for demonstration
thresholdValue := 10
var wg sync.WaitGroup
wg.Add(1)
go func() {
readValues(ch)
wg.Done()
}()
for i := 0; i < thresholdValue; i++ {
ch <- i
}
close(ch)
log.Println("sending done.")
wg.Wait()
}
func readValues(ch chan int) {
for value := range ch {
<-time.After(time.Second) // for demonstratin purposes.
log.Println(value)
}
}
In this version readValues exits because the for loop did exit and that main closed ch.
In other words a stop condition take effects and triggers the exit sequence (signal end of input then wait for the processing to finish)
I see lots of tutorials and examples on how to make Go wait for x number of goroutines to finish, but what I'm trying to do is have ensure there are always x number running, so a new goroutine is launched as soon as one ends.
Specifically I have a few hundred thousand 'things to do' which is processing some stuff that is coming out of MySQL. So it works like this:
db, err := sql.Open("mysql", connection_string)
checkErr(err)
defer db.Close()
rows,err := db.Query(`SELECT id FROM table`)
checkErr(err)
defer rows.Close()
var id uint
for rows.Next() {
err := rows.Scan(&id)
checkErr(err)
go processTheThing(id)
}
checkErr(err)
rows.Close()
Currently that will launch several hundred thousand threads of processTheThing(). What I need is that a maximum of x number (we'll call it 20) goroutines are launched. So it starts by launching 20 for the first 20 rows, and from then on it will launch a new goroutine for the next id the moment that one of the current goroutines has finished. So at any point in time there are always 20 running.
I'm sure this is quite simple/standard, but I can't seem to find a good explanation on any of the tutorials or examples or how this is done.
You may find Go Concurrency Patterns article interesting, especially Bounded parallelism section, it explains the exact pattern you need.
You can use channel of empty structs as a limiting guard to control number of concurrent worker goroutines:
package main
import "fmt"
func main() {
maxGoroutines := 10
guard := make(chan struct{}, maxGoroutines)
for i := 0; i < 30; i++ {
guard <- struct{}{} // would block if guard channel is already filled
go func(n int) {
worker(n)
<-guard
}(i)
}
}
func worker(i int) { fmt.Println("doing work on", i) }
Here I think something simple like this will work :
package main
import "fmt"
const MAX = 20
func main() {
sem := make(chan int, MAX)
for {
sem <- 1 // will block if there is MAX ints in sem
go func() {
fmt.Println("hello again, world")
<-sem // removes an int from sem, allowing another to proceed
}()
}
}
Thanks to everyone for helping me out with this. However, I don't feel that anyone really provided something that both worked and was simple/understandable, although you did all help me understand the technique.
What I have done in the end is I think much more understandable and practical as an answer to my specific question, so I will post it here in case anyone else has the same question.
Somehow this ended up looking a lot like what OneOfOne posted, which is great because now I understand that. But OneOfOne's code I found very difficult to understand at first because of the passing functions to functions made it quite confusing to understand what bit was for what. I think this way makes a lot more sense:
package main
import (
"fmt"
"sync"
)
const xthreads = 5 // Total number of threads to use, excluding the main() thread
func doSomething(a int) {
fmt.Println("My job is",a)
return
}
func main() {
var ch = make(chan int, 50) // This number 50 can be anything as long as it's larger than xthreads
var wg sync.WaitGroup
// This starts xthreads number of goroutines that wait for something to do
wg.Add(xthreads)
for i:=0; i<xthreads; i++ {
go func() {
for {
a, ok := <-ch
if !ok { // if there is nothing to do and the channel has been closed then end the goroutine
wg.Done()
return
}
doSomething(a) // do the thing
}
}()
}
// Now the jobs can be added to the channel, which is used as a queue
for i:=0; i<50; i++ {
ch <- i // add i to the queue
}
close(ch) // This tells the goroutines there's nothing else to do
wg.Wait() // Wait for the threads to finish
}
Create channel for passing data to goroutines.
Start 20 goroutines that processes the data from channel in a loop.
Send the data to the channel instead of starting a new goroutine.
Grzegorz Żur's answer is the most efficient way to do it, but for a newcomer it could be hard to implement without reading code, so here's a very simple implementation:
type idProcessor func(id uint)
func SpawnStuff(limit uint, proc idProcessor) chan<- uint {
ch := make(chan uint)
for i := uint(0); i < limit; i++ {
go func() {
for {
id, ok := <-ch
if !ok {
return
}
proc(id)
}
}()
}
return ch
}
func main() {
runtime.GOMAXPROCS(4)
var wg sync.WaitGroup //this is just for the demo, otherwise main will return
fn := func(id uint) {
fmt.Println(id)
wg.Done()
}
wg.Add(1000)
ch := SpawnStuff(10, fn)
for i := uint(0); i < 1000; i++ {
ch <- i
}
close(ch) //should do this to make all the goroutines exit gracefully
wg.Wait()
}
playground
This is a simple producer-consumer problem, which in Go can be easily solved using channels to buffer the paquets.
To put it simple: create a channel that accept your IDs. Run a number of routines which will read from the channel in a loop then process the ID. Then run your loop that will feed IDs to the channel.
Example:
func producer() {
var buffer = make(chan uint)
for i := 0; i < 20; i++ {
go consumer(buffer)
}
for _, id := range IDs {
buffer <- id
}
}
func consumer(buffer chan uint) {
for {
id := <- buffer
// Do your things here
}
}
Things to know:
Unbuffered channels are blocking: if the item wrote into the channel isn't accepted, the routine feeding the item will block until it is
My example lack a closing mechanism: you must find a way to make the producer to wait for all consumers to end their loop before returning. The simplest way to do this is with another channel. I let you think about it.
I've wrote a simple package to handle concurrency for Golang. This package will help you limit the number of goroutines that are allowed to run concurrently:
https://github.com/zenthangplus/goccm
Example:
package main
import (
"fmt"
"goccm"
"time"
)
func main() {
// Limit 3 goroutines to run concurrently.
c := goccm.New(3)
for i := 1; i <= 10; i++ {
// This function have to call before any goroutine
c.Wait()
go func(i int) {
fmt.Printf("Job %d is running\n", i)
time.Sleep(2 * time.Second)
// This function have to when a goroutine has finished
// Or you can use `defer c.Done()` at the top of goroutine.
c.Done()
}(i)
}
// This function have to call to ensure all goroutines have finished
// after close the main program.
c.WaitAllDone()
}
Also can take a look here: https://github.com/LiangfengChen/goutil/blob/main/concurrent.go
The example can refer the test case.
func TestParallelCall(t *testing.T) {
format := "test:%d"
data := make(map[int]bool)
mutex := sync.Mutex{}
val, err := ParallelCall(1000, 10, func(pos int) (interface{}, error) {
mutex.Lock()
defer mutex.Unlock()
data[pos] = true
return pos, errors.New(fmt.Sprintf(format, pos))
})
for i := 0; i < 1000; i++ {
if _, ok := data[i]; !ok {
t.Errorf("TestParallelCall pos not found: %d", i)
}
if val[i] != i {
t.Errorf("TestParallelCall return value is not right (%d,%v)", i, val[i])
}
if err[i].Error() != fmt.Sprintf(format, i) {
t.Errorf("TestParallelCall error msg is not correct (%d,%v)", i, err[i])
}
}
}
I try to write a simple code in GO, in which two go routine (Send and Receive) send each other integers. I give the code below. Can anybody help me why the output of this program is [no output]. Is there any silly mistake (sorry, I am new in GO) ?
package main
func Send (in1 <-chan int, out1 chan<- int) {
i := 2
out1 <- i
print(i, "\n")
}
func Receive (in <-chan int, out chan<- int) {
i := <-in
print(i, "\n")
out <- i
}
func main() {
for i := 0; i < 10; i++ {
ch1 := make(chan int)
ch := make(chan int)
go Send (ch1 , ch)
go Receive (ch , ch1)
ch = ch1
ch1 = ch
}
}
How about this:
package main
func Send (ch chan<- int) {
for i := 0; i < 10; i++ {
print(i, " sending\n")
ch <- i
}
}
func Receive (ch <-chan int) {
for i := 0; i < 10; i++ {
print(<-ch, " received\n")
}
}
func main() {
ch := make(chan int)
go Receive(ch)
Send(ch)
}
The output of this when I run it on golang.org is:
0 sending
0 received
1 sending
2 sending
1 received
2 received
3 sending
4 sending
3 received
4 received
5 sending
6 sending
5 received
6 received
7 sending
8 sending
7 received
8 received
9 sending
I'm not sure why the 9 was never received. There should be some way to sleep the main thread until the Receive goroutine has completed. Also, it's inelegant that the receiver and the sender both know that they are going to send 10 numbers. One of the goroutines should just shut off when the other has finished its work. I'm not sure how to do that.
EDIT1:
Here is an implementation with two channels, and the go routines send ints back and forth between eachother. One is designated as the responder, who only sends and int after it receives an int. The responder simply adds two to the int he receives, and then sends it back.
package main
func Commander(commands chan int, responses chan int) {
for i := 0; i < 10; i++ {
print(i, " command\n")
commands <- i
print(<-responses, " response\n");
}
close(commands)
}
func Responder(commands chan int, responses chan int) {
for {
x, open := <-commands
if !open {
return;
}
responses <- x + 2
}
}
func main() {
commands := make(chan int)
responses := make(chan int)
go Commander(commands, responses)
Responder(commands, responses)
}
The output when I run it on golang.org is:
0 command
2 response
1 command
3 response
2 command
4 response
3 command
5 response
4 command
6 response
5 command
7 response
6 command
8 response
7 command
9 response
8 command
10 response
9 command
11 response
David Grayson's fix works, but the explanation of why 9 is not always received wouldn't fit into a comment!
When the sending goroutine sends the value on the channel, it blocks until the receiving goroutine receives it. At that point, it's unblocked. The go scheduler might return from main almost immediately before it gives the receiving goroutine a chance to print anything. If you set GOMAXPROCS to something greater than 1, this happens less frequently because the receiving goroutine won't block while the main goroutine is active.
If you want to stop main from returning until all goroutines are finished, you can return a channel from Receive(), like this:
package main
import "fmt"
func Send(ch chan<- int) {
for i := 0; i < 10; i++ {
fmt.Println(i, " sending")
ch <- i
}
close(ch)
}
func Receive(ch <-chan int) (<-chan bool) {
done := make(chan bool)
go func() {
for {
i, ok := <-ch
if !ok {
break
}
fmt.Println(i, " received")
}
done <- true
}()
return done
}
func main() {
ch := make(chan int)
d := Receive(ch)
Send(ch)
_ = <-d
}
As for why your output was empty in your original example: you make two goroutines, and thus main is totally unblocked, so it just returns and the program exits! I've definitely made this mistake too, but the runtime only joins with the main goroutine, not with ALL goroutines, like you may expect.