I am trying to build a pipeline for ingesting data with Go. The 3 stages are "Download batches", "Transform each message", and "Enqueue messages into a queue".
The logic which seemed natural to me is to create 3 functions for each stage, and to tie these functions with unbuffered channels.
Somewhere in my code, I am not implementing channels correctly, or not using waitgroups? As only 1 message gets to the final stage and program seems to stop/block.
func (c *worker) startWork() {
// channel for messages to be sent to the queue
chMessagesToEnqueue := make(chan types.Foo)
// channel for messages to be transformed
chMessagesToTransform := make(chan []types.UpstreamFooType)
// start the goroutines with the channels
go c.startTransformer(chMessagesToTransform, chMessagesToEnqueue)
go c.startEnqueuer(chMessagesToEnqueue)
go c.startDownloader(chMessagesToTransform)
}
func (c *worker) startDownloader(out chan []types.UpstreamFooType) {
// https://github.com/SebastiaanKlippert/go-soda
// uses a library here to fetch data from upstream APIs, but the gist is:
var wg sync.WaitGroup
for i := 0; i < c.workerCount; i++ {
wg.Add(1)
go func() {
defer wg.Done()
for {
// i've cut out some meat out to make more concise
var results []types.UpstreamFooType
err = json.NewDecoder(resp.Body).Decode(&results)
out <- results
}
}()
}
wg.Wait()
}
func (c *worker) startTransformer(in <-chan []types.UpstreamFooType, out chan types.Foo) {
data := <-in
for _, record := range data {
msg := types.Foo{
name: record.fifa,
}
out <- msg
}
}
func (c *worker) startEnqueuer(in <-chan []types.Foo) {
data := <-in
c.logger.Infow("startEnqueuer", "data", data)
}
Related
I need your help to understand why my readFromWorker func lead to deadlock. When I comment out lines like below it works correctly (thus I know the problem is here).
The whole is here https://play.golang.org/p/-0mRDAeD2tr
I would really appreciate your help
func readFromWorker(inCh <-chan *data, wg *sync.WaitGroup) {
defer func() {
wg.Done()
}()
//stageIn1 := make(chan *data)
//stageOut1 := make(chan *data)
for v := range inCh {
fmt.Println("v", v)
//stageIn1 <- v
}
//go stage1(stageIn1, stageOut1)
//go stage2(stageOut1)
}
I've commented on the relevant parts where you were doing it wrong. Also, I'd recommend thinking of a better pattern.
Do remember that for range on channels doesn't stop looping unless close is called for the same channel it's looping on. Also, the rule of thumb of closing a channel is that the sender sending to the channel must also close it because sending to a closed channel causes panic.
Also, be very careful when using unbuffered and buffered channels. For unbuffered channels, the sender and receiver must be ready otherwise there would be a deadlock which happened in your case as well.
package main
import (
"fmt"
"sync"
)
type data struct {
id int
url string
field int
}
type job struct {
id int
url string
}
func sendToWorker(id int, inCh <-chan job, outCh chan<- *data, wg *sync.WaitGroup) {
// wg.Done() is itself a function call, no need to wrap it inside
// an anonymous function just to use defer.
defer wg.Done()
for v := range inCh {
// some pre process stuff and then pass to pipeline
outCh <- &data{id: v.id, url: v.url}
}
}
func readFromWorker(inCh <-chan *data, wg *sync.WaitGroup) {
// wg.Done() is itself a function call, no need to wrap it inside
// an anonymous function just to use defer.
defer wg.Done()
var (
stageIn1 = make(chan *data)
stageOut1 = make(chan *data)
)
// Spawn the goroutines so that there's no deadlock
// as the sender and receiver both should be ready
// when using unbuffered channels.
go stage1(stageIn1, stageOut1)
go stage2(stageOut1)
for v := range inCh {
fmt.Println("v", v)
stageIn1 <- v
}
close(stageIn1)
}
func stage1(in <-chan *data, out chan<- *data) {
for s := range in {
fmt.Println("stage1 = ", s)
out <- s
}
// Close the out channel
close(out)
}
func stage2(out <-chan *data) {
// Loop until close
for s := range out {
fmt.Println("stage2 = ", s)
}
}
func main() {
const chanBuffer = 1
var (
inputsCh = make(chan job, chanBuffer)
resultsCh = make(chan *data, chanBuffer)
wgInput sync.WaitGroup
wgResult sync.WaitGroup
)
for i := 1; i <= 4; i++ {
wgInput.Add(1)
go sendToWorker(i, inputsCh, resultsCh, &wgInput)
}
wgResult.Add(1)
go readFromWorker(resultsCh, &wgResult)
for j := 1; j <= 10; j++ {
inputsCh <- job{id: j, url: "google.com"}
}
close(inputsCh)
wgInput.Wait()
close(resultsCh)
wgResult.Wait()
}
I'm a bit of a loss why the following design is creating a memory leak. Able to repro the leak via a simple kubectl top pod command and watching memory hit the roof after a few seconds of execution.
I have a worker which watches a RabbitMQ message queue for messages. If I drop ~100K of small messages (each ~200B) into the exchange, I'm seeing the following GO app start to leak.
Before rewriting, I have the time and energy to deep dive and understand what exactly is causing the memory leak.
type workerMessage struct {
AmqpMessage amqp.Delivery
Data types.Data
}
func main() {
var wg sync.WaitGroup
for i := 0; i < opts.WorkerCount; i++ {
wg.Add(1)
go w.StartWork(&wg)
}
// create a wait group with 1 routine added
var wait sync.WaitGroup
wait.Add(1)
// capture 2 signals, and have them send to signalChan
stopChan := make(chan os.Signal, 1)
signal.Notify(stopChan, syscall.SIGINT, syscall.SIGTERM, os.Kill, os.Interrupt)
// wait for a receive on the signal channel, and decrement the wait group
// when a singal is received
go func() {
<-stopChan
wait.Done()
}()
// block until a signal is received
wait.Wait()
}
func (w *Worker) StartWork(wg *sync.WaitGroup) {
defer wg.Done()
// messages will be acked from AMQP here
chToBeAcked := make(chan workerMessage)
go w.watchForAMQPMessagesToAck(&iwg, chToBeAcked)
// data will be written to cassandra or object storage
chDataToStorage := make(chan workerMessage)
go w.writeDataToStorage(&iwg, chDataToStorage, chToBeAcked)
// call services, process data and decides what to do next
chProcessData := make(chan workerMessage)
go w.ProcessData(&iwg, chProcessData, chDataToStorage, chToBeAcked)
messageChannel, err := amqpChannel.Consume(
queue.Name,
....
)
stopChan := make(chan bool)
go func() {
for message := range messageChannel {
wm := workerMessage{
AmqpMessage: message,
}
chProcessData <- wm
}
}()
<-stopChan
}
func (w *Worker) ProcessData(
wg *sync.WaitGroup,
chProcessData <-chan workerMessage,
chDataToStorage chan workerMessage,
chMessagesWaitingForAck chan workerMessage,
){
defer wg.Done()
for msg := range chProcessData {
...
make some API calls
update some things in msg.Data
...
If fastExit {
chMessagesWaitingForAck <- msg
} else {
chDataToStorage <- msg
}
}
}
func (w *Worker) writeDataToStorage(
wg *sync.WaitGroup,
chDataToStorage <-chan workerMessage,
chMessagesWaitingForAck chan workerMessage,
){
defer wg.Done()
for msg := range chDataToStorage {
...
write data to cassandra or key/val object storage
...
chMessagesWaitingForAck <- msg
}
}
func (w *Worker) watchForAMQPMessagesToAck(
wg *sync.WaitGroup,
chMessagesWaitingForAck <-chan workerMessage,
){
defer wg.Done()
for msg := range chMessagesWaitingForAck {
err := msg.AmqpMessage.Ack(false)
w.stopOnError("failed to ack a message", msg, err)
}
}
The problem was creating the goroutines for every message! Pure oversight due to lack of sleep
Updating my question so nobody makes the same mistake
I have been following a pattern of checking if there is anything in the channel before proceeding with work:
func consume(msg <-chan message) {
for {
if m, ok := <-msg; ok {
fmt.Println("More messages:", m)
} else {
break
}
}
}
that is based on this video. Here is my full code:
package main
import (
"fmt"
"strconv"
"strings"
"sync"
)
type message struct {
body string
code int
}
var markets []string = []string{"BTC", "ETH", "LTC"}
// produces messages into the chan
func produce(n int, market string, msg chan<- message, wg *sync.WaitGroup) {
// for i := 0; i < n; i++ {
var msgToSend = message{
body: strings.Join([]string{"market: ", market, ", #", strconv.Itoa(1)}, ""),
code: 1,
}
fmt.Println("Producing:", msgToSend)
msg <- msgToSend
// }
wg.Done()
}
func receive(msg <-chan message, wg *sync.WaitGroup) {
for {
if m, ok := <-msg; ok {
fmt.Println("Received:", m)
} else {
fmt.Println("Breaking from receiving")
break
}
}
wg.Done()
}
func main() {
wg := sync.WaitGroup{}
msgC := make(chan message, 100)
defer func() {
close(msgC)
}()
for ix, market := range markets {
wg.Add(1)
go produce(ix+1, market, msgC, &wg)
}
wg.Add(1)
go receive(msgC, &wg)
wg.Wait()
}
If you try to run it, we get the deadlock at the very end before we ever print a message that we are about to break. Which, tbh, makes sense, since the last time, when there is nothing else in the chan, we are trying to pull the value out, and so we get this error. But then this pattern isn't workable if m, ok := <- msg; ok. How do I make this code work & why do I get this deadlock error (presumably this pattern should work?).
Given that you do have multiple writers on a single channel, you have a bit of a challenge, because the easy way to do this in Go in general is to have a single writer on a single channel, and then have that single writer close the channel upon sending the last datum:
func produce(... args including channel) {
defer close(ch)
for stuff_to_produce {
ch <- item
}
}
This pattern has the nice property that no matter how you get out of produce, the channel gets closed, signalling the end of production.
You're not using this pattern—you deliver one channel to many goroutines, each of which can send one message—so you need to move the close (or, of course, use yet some other pattern). The simplest way to express the pattern you need is this:
func overall_produce(... args including channel ...) {
var pg sync.WaitGroup
defer close(ch)
for stuff_to_produce {
pg.Add(1)
go produceInParallel(ch, &pg) // add more args if appropriate
}
pg.Wait()
}
The pg counter accumulates active producers. Each must call pg.Done() to indicate that it is done using ch. The overall producer now waits for them all to be done, then it closes the channel on its way out.
(If you write the inner produceInParallel function as a closure, you don't need to pass ch and pg to it explicitly. You may also write overallProducer as a closure.)
Note that your single consumer's loop is probably best expressed using the for ... range construct:
func receive(msg <-chan message, wg *sync.WaitGroup) {
for m := range msg {
fmt.Println("Received:", m)
}
wg.Done()
}
(You mention an intent to add a select to the loop so that you can do some other computing if a message is not yet ready. If that code cannot be spun off into independent goroutines, you will in fact need the fancier m, ok := <-msg construct.)
Note also that the wg for receive—which may turn out to be unnecessary, depending on how you structure other things—is quite independent from the wait-group pg for the producers. While it's true that, as written, the consumer cannot be done until all the producers are done, we'd like to wait independently for the producers to be done, so that we can close the channel in the overall-producer wrapper.
Try this code, I have made few fixes that made it work:
package main
import (
"fmt"
"strconv"
"strings"
"sync"
)
type message struct {
body string
code int
}
var markets []string = []string{"BTC", "ETH", "LTC"}
// produces messages into the chan
func produce(n int, market string, msg chan<- message, wg *sync.WaitGroup) {
// for i := 0; i < n; i++ {
var msgToSend = message{
body: strings.Join([]string{"market: ", market, ", #", strconv.Itoa(1)}, ""),
code: 1,
}
fmt.Println("Producing:", msgToSend)
msg <- msgToSend
// }
}
func receive(msg <-chan message, wg *sync.WaitGroup) {
for {
if m, ok := <-msg; ok {
fmt.Println("Received:", m)
wg.Done()
}
}
}
func consume(msg <-chan message) {
for {
if m, ok := <-msg; ok {
fmt.Println("More messages:", m)
} else {
break
}
}
}
func main() {
wg := sync.WaitGroup{}
msgC := make(chan message, 100)
defer func() {
close(msgC)
}()
for ix, market := range markets {
wg.Add(1)
go produce(ix+1, market, msgC, &wg)
}
go receive(msgC, &wg)
wg.Wait()
fmt.Println("Breaking from receiving")
}
Only when main returns, you can close(msgC), but meanwhile receive is waiting for close signal, that's why DeadLock occurs. After producing messages, close the channel.
package main
import (
"fmt"
"strconv"
"strings"
"sync"
)
type message struct {
body string
code int
}
var markets []string = []string{"BTC", "ETH", "LTC"}
// produces messages into the chan
func produce(n int, market string, msg chan<- message, wg *sync.WaitGroup) {
// for i := 0; i < n; i++ {
var msgToSend = message{
body: strings.Join([]string{"market: ", market, ", #", strconv.Itoa(1)}, ""),
code: 1,
}
fmt.Println("Producing:", msgToSend)
msg <- msgToSend
// }
wg.Done()
}
func receive(msg <-chan message, wg *sync.WaitGroup) {
for {
if m, ok := <-msg; ok {
fmt.Println("Received:", m)
} else {
fmt.Println("Breaking from receiving")
break
}
}
wg.Done()
}
func main() {
wg := sync.WaitGroup{}
msgC := make(chan message, 100)
// defer func() {
// close(msgC)
// }()
for ix, market := range markets {
wg.Add(1)
go produce(ix+1, market, msgC, &wg)
}
wg.Wait() // wait for producer
close(msgC)
wg.Add(1)
go receive(msgC, &wg)
wg.Wait()
}
I started learning go recently and I am stuck on a problem.
I have a simple go routine which either returns or pushes value to a channel.
And my main fn delegates work to this routine till it meets condition or data is exhausted.
This code seem to deadlock on "found" channel. What am I doing wrong?
There are multiple workers
Item can be found in more than one worker at the same time
Once item is found, all workers should be stopped.
.
func workerRoutine(data Data, found chan bool, wg *sync.WaitGroup){
defer (*wg).Done()
// data processing
// return on false
// multiple routines can set this at the same time
found <-true
}
func main {
// ....
found:=make(chan bool)
var wg sync.WaitGroup
itemFound:=false
Loop:
for i:=0; i<limit; i++ {
select {
case <-found:
itemFound = true
break Loop
default:
if(some_check) {
wg.Add(1)
go workerRoutine(mdata,found,&wg)
}
}
}
wg.Wait()
// use itemFound
}
One possible solution is to avoid select statement and use separate goroutine for receiver (or sender, or both).
Example:
package main
import "sync"
func worker(res chan bool, wg *sync.WaitGroup) {
res <- true
wg.Done()
}
func receiver(res chan bool, wg *sync.WaitGroup) {
for range res {
}
wg.Done()
}
func main() {
var wg, wg2 sync.WaitGroup
wg.Add(1)
wg2.Add(10)
found := make(chan bool)
go receiver(found, &wg)
for i := 0; i < 10; i++ {
go worker(found, &wg2)
}
wg2.Wait()
close(found)
wg.Done()
}
I'm having an issue trying to implement this (all goroutines asleep - deadlock!)
Here's the gist of the code:
var workers = runtime.NumCPU()
func main() {
jobs := make(chan *myStruct, workers)
done := make(chan *myStruct, workers)
go produceWork(file_with_jobs, jobs)
for i := 0; i < runtime.NumCPU(); i++ {
go Worker(jobs, done)
}
consumeWork(done)
}
func produceWork(vf string, jobs chan *utils.DigSigEntries) {
defer close(jobs)
// load file with jobs
file, err := ini.LoadFile(vf)
// get data for processing
for data, _ := range file {
// ...
jobs <- &myStruct{data1, data2, data3, false}
}
}
func Worker(in, out chan *myStruct) {
for {
item, open := <-in
if !open {
break
}
process(item)
out <- item
}
// close(out) --> tried closing the out channel, but then not all items are processed
// though no panics occur.
}
func process(item *myStruct) {
//...modify the item
item.status = true
}
func consumeWork(done chan *myStruct) {
for val := range done {
if !val.status {
fmt.Println(val)
}
}
}
I'm mainly trying to understand how to do this without using the sync/Wait stuff - just pure channels - is this possible? The goal of this routine is to have a single producer load up items that are processed by N workers - appreciate any pointers / help.
You can, as siritinga suggested, use a third signalling or counter channel, eg signal chan boolean, where the produceWork goroutine would add a value before each job entered into the jobs channel. So, an equal count of values will be passed to signal as to jobs:
func produceWork(vf string, jobs chan *utils.DigSigEntries, signal chan boolean) {
defer close(jobs)
// load file with jobs
file, err := ini.LoadFile(vf)
// get data for processing
for data, _ := range file {
// ...
signal <- true
jobs <- &myStruct{data1, data2, data3, false}
}
close(signal)
}
The consume would then start by reading from the signal channel. If there is a value, it can be certain there will be a value to read from the out channel (once a worker has passed it on). If the signal is closed, then all is done. We can close the remaining done channel:
func consumeWork(done chan *myStruct, signal chan boolean) {
for _ := range signal {
val <- done
if !val.status {
fmt.Println(val)
}
}
close(done)
}
While this is possible, I would not really recommend it. It doesn't make the code more clear than when using a sync.WaitGroup. After all, the signal channel would basically only work as a counter. A WaitGroup would have the same purpose and would do it cheaper.
But your question was not about how to solve the problem, but rather if it was possible to do it with pure channels.
Sorry, i didn't notice that you wanted to skip /sync :/
I'll leave the answer, maybe someone is looking for this.
import (
"sync"
)
func main() {
jobs := make(chan *myStruct, workers)
done := make(chan *myStruct, workers)
var workerWg sync.WaitGroup // waitGroup for workers
var consumeWg sync.WaitGroup // waitGroup for consumer
consumeWg.Add(1) // add one active Consumer
for i := 0; i < runtime.NumCPU(); i++ {
go Worker(&workerWg, jobs, done)
workerWg.Add(1)
}
go consumeWork(&consumeWg, done)
produceWork(file_with_jobs, jobs)
close(jobs)
workerWg.Wait()
close(done)
consumeWg.Wait()
}
func produceWork(vf string, jobs chan *utils.DigSigEntries) {
// load file with jobs
file, err := ini.LoadFile(vf)
// get data for processing
for data, _ := range file {
// ...
jobs <- &myStruct{data1, data2, data3, false}
}
}
func Worker(wg *sync.WaitGroup, done chan *myStruct) {
defer wg.Done()
for job := range jobs {
result := process(job)
out <- result
}
// close(out) --> tried closing the out channel, but then not all items are processed
// though no panics occur.
}
func process(item *myStruct) {
//...modify the item
item.status = true
}
func consumeWork(wg *sync.WaitGroup, done chan *myStruct) {
defer wg.Done()
for val := range done {
if !val.status {
fmt.Println(val)
}
}
}