How to start and stop a function - go

I have a go function processing that use two distinct goroutines. produce will push some data into a channel and consume will read these data. Here is an example:
type MyObject struct{
...
}
func processing() {
var wg sync.WaitGroup
dataChannel := make(chan MyObject, 5)
wg.Add(2)
go produce(wg, dataChannel)
go consume(wg, dataChannel)
wg.Wait()
}
func produce (wg *sync.WaitGroup, dataChannel chan MyObject){
for{
// Produce data in dataChannel
}
}
func consume (wg *sync.WaitGroup, dataChannel chan MyObject){
for{
// Consume data from dataChannel
}
}
I want my processing function to be started and stoped by an HTTP call. So I am looking to do something as follow:
func main() {
// echo instance
e := echo.New()
e.GET("/", startProcessing)
e.Logger.Fatal(e.Start(":8099"))
}
func startProcessing(c echo.Context) error{
command := c.QueryParam("command")
if(command == "start"){
processing()
}else if(command == "stop"){
if (/* ? processing is running ? */){
/* ? stop processing process? */
}
}
}
What is the correct way to do this with Go?

Here how to start and stop a function using context, try this:
package main
import (
"context"
"fmt"
"sync"
"time"
)
func main() {
ctx, cancel := context.WithCancel(context.Background())
var wg sync.WaitGroup
dataChannel := make(chan MyObject, 5)
wg.Add(2)
go produce(ctx, &wg, dataChannel)
go consume(&wg, dataChannel)
time.Sleep(1 * time.Second)
cancel() // cancel when we are finished consuming data
wg.Wait()
}
func produce(ctx context.Context, wg *sync.WaitGroup, dataChannel chan MyObject) {
defer wg.Done()
i := 1
for {
select {
case <-ctx.Done():
close(dataChannel)
return // returning not to leak the goroutine
case dataChannel <- MyObject{i}:
i++
time.Sleep(250 * time.Millisecond)
}
}
}
func consume(wg *sync.WaitGroup, dataChannel chan MyObject) {
defer wg.Done()
for v := range dataChannel {
fmt.Println(v)
}
}
type MyObject struct {
i int
}
For HTTP you need to do it yourself!
It needs to have some concurrent safe ID or map or something to keep track of how many functions you called and then call a cancel() to stop it.

Related

golang future with WaitGroup block forever [duplicate]

This question already has answers here:
fatal error: all goroutines are asleep - deadlock! when channel is not buffered
(1 answer)
Why is the order of channels receiving causing/resolving a deadlock in Golang?
(3 answers)
Closed 4 months ago.
I wrote the following code trying to implement future for a group of async jobs. and it works. but I don't understand why I have to wrap wg.Wait() in a goroutine w/o it main will block.
package main
import (
"fmt"
"net/http"
"sync"
)
var urls = []string{
"https://www.easyjet.com/",
"https://www.skyscanner.de/",
"https://www.ryanair.com",
"https://wizzair.com/",
"https://www.swiss.com/",
}
func checker(urls []string) func() chan urlStatus {
wg := &sync.WaitGroup{}
c := make(chan urlStatus)
for _, url := range urls {
wg.Add(1)
go checkUrl(url, c, wg)
}
return func() chan urlStatus {
go func() { // <--- why this goroutine is necessary. change it to sync will block
wg.Wait()
close(c)
}()
return c
}
}
func main() {
done := make(chan bool)
future := checker(urls)
go consumer(future(), done)
<-done
}
func consumer(c <-chan urlStatus, done chan bool) {
for r := range c {
if r.status {
fmt.Println(r.url, "is up.")
} else {
fmt.Println(r.url, "is down!!")
}
}
fmt.Println("end of consumer")
done <- true
}
//checks and prints a message if a website is up or down
func checkUrl(url string, c chan<- urlStatus, wg *sync.WaitGroup) {
defer wg.Done()
_, err := http.Get(url)
if err != nil {
// The website is down
c <- urlStatus{url, false}
} else {
// The website is up
c <- urlStatus{url, true}
}
}
type urlStatus struct {
url string
status bool
}
I think it is because of this line
go consumer(future(), done)
When future() is called, the main go routine is blocked by wg.Wait()

Sync between 2 goroutines

My task is to sync 2 goroutines so the output should look like that:
foobarfoobarfoobarfoobar
.The issue is that when I call them they come out completely randomized. This is my code:
package main
import (
"fmt"
"sync"
"time"
)
type ConcurrentPrinter struct {
sync.WaitGroup
sync.Mutex
}
func (cp *ConcurrentPrinter) printFoo(times int) {
cp.WaitGroup.Add(times)
go func() {
cp.Lock()
fmt.Print("foo")
cp.Unlock()
}()
}
func (cp *ConcurrentPrinter) printBar(times int) {
cp.WaitGroup.Add(times)
go func() {
cp.Lock()
fmt.Print("bar")
cp.Unlock()
}()
}
func main() {
times := 10
cp := &ConcurrentPrinter{}
for i := 0; i <= times; i++ {
cp.printFoo(i)
cp.printBar(i)
}
time.Sleep(10 * time.Millisecond)
}
As outlined in the comments, using goroutines may not be the best use case for what you are trying to achieve - and thus this may be an XY problem.
Having said that, if you want to ensure two independent goroutines interleave their work in an alternating sequence, you can implement a set of "ping-pong" mutexs:
var ping, pong sync.Mutex
pong.Lock() // ensure the 2nd goroutine waits & the 1st goes first
go func() {
for {
ping.Lock()
foo()
pong.Unlock()
}
}()
go func() {
for {
pong.Lock()
bar()
ping.Unlock()
}
}()
https://go.dev/play/p/VO2LoMJ8fek
Using channel:
func printFoo(i int, ch chan<- bool, wg *sync.WaitGroup) {
wg.Add(1)
go func() {
defer wg.Done()
fmt.Print("foo")
ch <- true
}()
}
func printBar(i int, ch chan<- bool, wg *sync.WaitGroup) {
wg.Add(1)
go func() {
defer wg.Done()
fmt.Print("bar")
ch <- true
}()
}
func main() {
times := 4
firstchan := make(chan bool)
secondchan := make(chan bool)
var wg sync.WaitGroup
for i := 0; i <= times; i++ {
printFoo(i, firstchan, &wg)
<-firstchan
printBar(i, secondchan, &wg)
<-secondchan
}
wg.Wait()
}
https://go.dev/play/p/MlZ9dHkUXGb

Multiple producers, single consumer: all goroutines are asleep - deadlock

I have been following a pattern of checking if there is anything in the channel before proceeding with work:
func consume(msg <-chan message) {
for {
if m, ok := <-msg; ok {
fmt.Println("More messages:", m)
} else {
break
}
}
}
that is based on this video. Here is my full code:
package main
import (
"fmt"
"strconv"
"strings"
"sync"
)
type message struct {
body string
code int
}
var markets []string = []string{"BTC", "ETH", "LTC"}
// produces messages into the chan
func produce(n int, market string, msg chan<- message, wg *sync.WaitGroup) {
// for i := 0; i < n; i++ {
var msgToSend = message{
body: strings.Join([]string{"market: ", market, ", #", strconv.Itoa(1)}, ""),
code: 1,
}
fmt.Println("Producing:", msgToSend)
msg <- msgToSend
// }
wg.Done()
}
func receive(msg <-chan message, wg *sync.WaitGroup) {
for {
if m, ok := <-msg; ok {
fmt.Println("Received:", m)
} else {
fmt.Println("Breaking from receiving")
break
}
}
wg.Done()
}
func main() {
wg := sync.WaitGroup{}
msgC := make(chan message, 100)
defer func() {
close(msgC)
}()
for ix, market := range markets {
wg.Add(1)
go produce(ix+1, market, msgC, &wg)
}
wg.Add(1)
go receive(msgC, &wg)
wg.Wait()
}
If you try to run it, we get the deadlock at the very end before we ever print a message that we are about to break. Which, tbh, makes sense, since the last time, when there is nothing else in the chan, we are trying to pull the value out, and so we get this error. But then this pattern isn't workable if m, ok := <- msg; ok. How do I make this code work & why do I get this deadlock error (presumably this pattern should work?).
Given that you do have multiple writers on a single channel, you have a bit of a challenge, because the easy way to do this in Go in general is to have a single writer on a single channel, and then have that single writer close the channel upon sending the last datum:
func produce(... args including channel) {
defer close(ch)
for stuff_to_produce {
ch <- item
}
}
This pattern has the nice property that no matter how you get out of produce, the channel gets closed, signalling the end of production.
You're not using this pattern—you deliver one channel to many goroutines, each of which can send one message—so you need to move the close (or, of course, use yet some other pattern). The simplest way to express the pattern you need is this:
func overall_produce(... args including channel ...) {
var pg sync.WaitGroup
defer close(ch)
for stuff_to_produce {
pg.Add(1)
go produceInParallel(ch, &pg) // add more args if appropriate
}
pg.Wait()
}
The pg counter accumulates active producers. Each must call pg.Done() to indicate that it is done using ch. The overall producer now waits for them all to be done, then it closes the channel on its way out.
(If you write the inner produceInParallel function as a closure, you don't need to pass ch and pg to it explicitly. You may also write overallProducer as a closure.)
Note that your single consumer's loop is probably best expressed using the for ... range construct:
func receive(msg <-chan message, wg *sync.WaitGroup) {
for m := range msg {
fmt.Println("Received:", m)
}
wg.Done()
}
(You mention an intent to add a select to the loop so that you can do some other computing if a message is not yet ready. If that code cannot be spun off into independent goroutines, you will in fact need the fancier m, ok := <-msg construct.)
Note also that the wg for receive—which may turn out to be unnecessary, depending on how you structure other things—is quite independent from the wait-group pg for the producers. While it's true that, as written, the consumer cannot be done until all the producers are done, we'd like to wait independently for the producers to be done, so that we can close the channel in the overall-producer wrapper.
Try this code, I have made few fixes that made it work:
package main
import (
"fmt"
"strconv"
"strings"
"sync"
)
type message struct {
body string
code int
}
var markets []string = []string{"BTC", "ETH", "LTC"}
// produces messages into the chan
func produce(n int, market string, msg chan<- message, wg *sync.WaitGroup) {
// for i := 0; i < n; i++ {
var msgToSend = message{
body: strings.Join([]string{"market: ", market, ", #", strconv.Itoa(1)}, ""),
code: 1,
}
fmt.Println("Producing:", msgToSend)
msg <- msgToSend
// }
}
func receive(msg <-chan message, wg *sync.WaitGroup) {
for {
if m, ok := <-msg; ok {
fmt.Println("Received:", m)
wg.Done()
}
}
}
func consume(msg <-chan message) {
for {
if m, ok := <-msg; ok {
fmt.Println("More messages:", m)
} else {
break
}
}
}
func main() {
wg := sync.WaitGroup{}
msgC := make(chan message, 100)
defer func() {
close(msgC)
}()
for ix, market := range markets {
wg.Add(1)
go produce(ix+1, market, msgC, &wg)
}
go receive(msgC, &wg)
wg.Wait()
fmt.Println("Breaking from receiving")
}
Only when main returns, you can close(msgC), but meanwhile receive is waiting for close signal, that's why DeadLock occurs. After producing messages, close the channel.
package main
import (
"fmt"
"strconv"
"strings"
"sync"
)
type message struct {
body string
code int
}
var markets []string = []string{"BTC", "ETH", "LTC"}
// produces messages into the chan
func produce(n int, market string, msg chan<- message, wg *sync.WaitGroup) {
// for i := 0; i < n; i++ {
var msgToSend = message{
body: strings.Join([]string{"market: ", market, ", #", strconv.Itoa(1)}, ""),
code: 1,
}
fmt.Println("Producing:", msgToSend)
msg <- msgToSend
// }
wg.Done()
}
func receive(msg <-chan message, wg *sync.WaitGroup) {
for {
if m, ok := <-msg; ok {
fmt.Println("Received:", m)
} else {
fmt.Println("Breaking from receiving")
break
}
}
wg.Done()
}
func main() {
wg := sync.WaitGroup{}
msgC := make(chan message, 100)
// defer func() {
// close(msgC)
// }()
for ix, market := range markets {
wg.Add(1)
go produce(ix+1, market, msgC, &wg)
}
wg.Wait() // wait for producer
close(msgC)
wg.Add(1)
go receive(msgC, &wg)
wg.Wait()
}

Trying to end goroutine using close(ch) but end up running infinitely

I am trying to end multiple goroutines once another goroutine closes a channel. However, I am ending up into infinite loop after close signal is received. I can't figure out why.
I know that it is possible using context.Context but I was trying out by closing channels.
Go Playground: https://play.golang.org/p/C6pcYgGLnG9
package main
import (
"fmt"
"time"
"sync"
)
func runner(id int, ch <-chan struct{}, wg *sync.WaitGroup) {
for {
select {
case <-time.Tick(time.Second):
fmt.Println("worker ", id)
case <- ch:
fmt.Println("closing worker ", id)
break
}
}
wg.Done()
}
func main() {
fmt.Println("Hello, playground")
ch := make(chan struct{})
var wg sync.WaitGroup
wg.Add(1)
go runner(1, ch, &wg)
wg.Add(1)
go runner(2, ch, &wg)
time.Sleep(5*time.Second)
close(ch)
wg.Wait()
}
The problem is the scope of your break:
func runner(id int, ch <-chan struct{}, wg *sync.WaitGroup) {
for {
select {
case <-time.Tick(time.Second):
fmt.Println("worker ", id)
case <- ch:
fmt.Println("closing worker ", id)
break
}
}
wg.Done()
}
You want to break out of the for loop, but you're actually only breaking out of the select. To remedy this, you have two choices:
Add a label to your for loop, and break from it explicitly:
func runner(id int, ch <-chan struct{}, wg *sync.WaitGroup) {
loop: // <---------- add a label
for {
select {
case <-time.Tick(time.Second):
fmt.Println("worker ", id)
case <-ch:
fmt.Println("closing worker ", id)
break loop // <---------- and break from it explicitly
}
}
wg.Done()
}
Probably a more idiomatic and robust solution, simply return when you're done. This means the wg.Done() call must be deferred.
func runner(id int, ch <-chan struct{}, wg *sync.WaitGroup) {
defer wg.Done() // <--- Defer the wg.Done() call, so it happens on return
for {
select {
case <-time.Tick(time.Second):
fmt.Println("worker ", id)
case <-ch:
fmt.Println("closing worker ", id)
return // <--- replace `break` with `return`
}
}
}

go routine deadlock with single channel

I started learning go recently and I am stuck on a problem.
I have a simple go routine which either returns or pushes value to a channel.
And my main fn delegates work to this routine till it meets condition or data is exhausted.
This code seem to deadlock on "found" channel. What am I doing wrong?
There are multiple workers
Item can be found in more than one worker at the same time
Once item is found, all workers should be stopped.
.
func workerRoutine(data Data, found chan bool, wg *sync.WaitGroup){
defer (*wg).Done()
// data processing
// return on false
// multiple routines can set this at the same time
found <-true
}
func main {
// ....
found:=make(chan bool)
var wg sync.WaitGroup
itemFound:=false
Loop:
for i:=0; i<limit; i++ {
select {
case <-found:
itemFound = true
break Loop
default:
if(some_check) {
wg.Add(1)
go workerRoutine(mdata,found,&wg)
}
}
}
wg.Wait()
// use itemFound
}
One possible solution is to avoid select statement and use separate goroutine for receiver (or sender, or both).
Example:
package main
import "sync"
func worker(res chan bool, wg *sync.WaitGroup) {
res <- true
wg.Done()
}
func receiver(res chan bool, wg *sync.WaitGroup) {
for range res {
}
wg.Done()
}
func main() {
var wg, wg2 sync.WaitGroup
wg.Add(1)
wg2.Add(10)
found := make(chan bool)
go receiver(found, &wg)
for i := 0; i < 10; i++ {
go worker(found, &wg2)
}
wg2.Wait()
close(found)
wg.Done()
}

Resources