I found an interesting piece of code when using rabbitMQ
forever := make(chan bool)
go func() {
for d := range msgs {
log.Printf("Received a message: %s", d.Body)
}
}()
log.Printf(" [*] Waiting for messages. To exit press CTRL+C")
<-forever
This is a block of code ，
In fact, in normal mode,
this would cause a deadlock error,
Like this
enter image description here
enter image description here
But when I import rabbitMQ package , this code does not cause an error
enter image description here
Why is that? I'm confused.
Thanks for answer!
Expect someone to explain
When you are using rabbitmq, the msgs variable is a receiving channel of type <-chan amqp.Delivery - see docs i.e a channel receive messages. The range loop is enters the body each time a message appears. This is a useful control sequence - block the main goroutine whilst the worker awaits and processes messages. There is no deadlock in this case as rabbitmq connection will send messages down the msgs chan when appropriate.
In earlier code, when you connect to the queue and instantiate the msgs channel, the amqp package çreates another goroutine in the background to send messages down the msgs channel.
msgs, err := ch.Consume(
q.Name, // queue
"", // consumer
true, // auto-ack
false, // exclusive
false, // no-local
false, // no-wait
nil, // args
)
This is unlike the deadlock examples provided, where there is no additional go-routines to send messages down the forever channel. The goroutine goes to sleep and there is no chance of it being awakened. It is a deadlock.
Related
In a project the program receives data via websocket. This data needs to be processed by n algorithms. The amount of algorithms can change dynamically.
My attempt is to create some pub/sub pattern where subscriptions can be started and canceled on the fly. Turns out that this is a bit more challenging than expected.
Here's what I came up with (which is based on https://eli.thegreenplace.net/2020/pubsub-using-channels-in-go/):
package pubsub
import (
"context"
"sync"
"time"
)
type Pubsub struct {
sync.RWMutex
subs []*Subsciption
closed bool
}
func New() *Pubsub {
ps := &Pubsub{}
ps.subs = []*Subsciption{}
return ps
}
func (ps *Pubsub) Publish(msg interface{}) {
ps.RLock()
defer ps.RUnlock()
if ps.closed {
return
}
for _, sub := range ps.subs {
// ISSUE1: These goroutines apparently do not exit properly...
go func(ch chan interface{}) {
ch <- msg
}(sub.Data)
}
}
func (ps *Pubsub) Subscribe() (context.Context, *Subsciption, error) {
ps.Lock()
defer ps.Unlock()
// prep channel
ctx, cancel := context.WithCancel(context.Background())
sub := &Subsciption{
Data: make(chan interface{}, 1),
cancel: cancel,
ps: ps,
}
// prep subsciption
ps.subs = append(ps.subs, sub)
return ctx, sub, nil
}
func (ps *Pubsub) unsubscribe(s *Subsciption) bool {
ps.Lock()
defer ps.Unlock()
found := false
index := 0
for i, sub := range ps.subs {
if sub == s {
index = i
found = true
}
}
if found {
s.cancel()
ps.subs[index] = ps.subs[len(ps.subs)-1]
ps.subs = ps.subs[:len(ps.subs)-1]
// ISSUE2: close the channel async with a delay to ensure
// nothing will be written to the channel anymore
// via a pending goroutine from Publish()
go func(ch chan interface{}) {
time.Sleep(500 * time.Millisecond)
close(ch)
}(s.Data)
}
return found
}
func (ps *Pubsub) Close() {
ps.Lock()
defer ps.Unlock()
if !ps.closed {
ps.closed = true
for _, sub := range ps.subs {
sub.cancel()
// ISSUE2: close the channel async with a delay to ensure
// nothing will be written to the channel anymore
// via a pending goroutine from Publish()
go func(ch chan interface{}) {
time.Sleep(500 * time.Millisecond)
close(ch)
}(sub.Data)
}
}
}
type Subsciption struct {
Data chan interface{}
cancel func()
ps *Pubsub
}
func (s *Subsciption) Unsubscribe() {
s.ps.unsubscribe(s)
}
As mentioned in the comments there are (at least) two issues with this:
ISSUE1:
After a while of running in implementation of this I get a few errors of this kind:
goroutine 120624 [runnable]:
bm/internal/pubsub.(*Pubsub).Publish.func1(0x8586c0, 0xc00b44e880, 0xc008617740)
/home/X/Projects/bm/internal/pubsub/pubsub.go:30
created by bookmaker/internal/pubsub.(*Pubsub).Publish
/home/X/Projects/bm/internal/pubsub/pubsub.go:30 +0xbb
Without really understanding this it appears to me that the goroutines created in Publish() do accumulate/leak. Is this correct and what am I doing wrong here?
ISSUE2:
When I end a subscription via Unsubscribe() it occurs that Publish() tried to write to a closed channel and panics. To mitigate this I have created a goroutine to close the channel with a delay. This feel really off-best-practice but I was not able to find a proper solution to this. What would be a deterministic way to do this?
Heres a little playground for you to test with: https://play.golang.org/p/K-L8vLjt7_9
Before diving into your solution and its issues, let me recommend again another Broker approach presented in this answer: How to broadcast message using channel
Now on to your solution.
Whenever you launch a goroutine, always think of how it will end and make sure it does if the goroutine is not ought to run for the lifetime of your app.
// ISSUE1: These goroutines apparently do not exit properly...
go func(ch chan interface{}) {
ch <- msg
}(sub.Data)
This goroutine tries to send a value on ch. This may be a blocking operation: it will block if ch's buffer is full and there is no ready receiver on ch. This is out of the control of the launched goroutine, and also out of the control of the pubsub package. This may be fine in some cases, but this already places a burden on the users of the package. Try to avoid these. Try to create APIs that are easy to use and hard to misuse.
Also, launching a goroutine just to send a value on a channel is a waste of resources (goroutines are cheap and light, but you shouldn't spam them whenever you can).
You do it because you don't want to get blocked. To avoid blocking, you may use a buffered channel with a "reasonable" high buffer. Yes, this doesn't solve the blocking issue, in only helps with "slow" clients receiving from the channel.
To "truly" avoid blocking without launching a goroutine, you may use non-blocking send:
select {
case ch <- msg:
default:
// ch's buffer is full, we cannot deliver now
}
If send on ch can proceed, it will happen. If not, the default branch is chosen immediately. You have to decide what to do then. Is it acceptable to "lose" a message? Is it acceptable to wait for some time until "giving up"? Or is it acceptable to launch a goroutine to do this (but then you'll be back at what we're trying to fix here)? Or is it acceptable to get blocked until the client can receive from the channel...
Choosing a reasonable high buffer, if you encounter a situation when it still gets full, it may be acceptable to block until the client can advance and receive from the message. If it can't, then your whole app might be in an unacceptable state, and it might be acceptable to "hang" or "crash".
// ISSUE2: close the channel async with a delay to ensure
// nothing will be written to the channel anymore
// via a pending goroutine from Publish()
go func(ch chan interface{}) {
time.Sleep(500 * time.Millisecond)
close(ch)
}(s.Data)
Closing a channel is a signal to the receiver(s) that no more values will be sent on the channel. So always it should be the sender's job (and responsibility) to close the channel. Launching a goroutine to close the channel, you "hand" that job and responsibility to another "entity" (a goroutine) that will not be synchronized to the sender. This may easily end up in a panic (sending on a closed channel is a runtime panic, for other axioms see How does a non initialized channel behave?). Don't do that.
Yes, this was necessary because you launched goroutines to send. If you don't do that, then you may close "in-place", without launching a goroutine, because then the sender and closer will be the same entity: the Pubsub itself, whose sending and closing operations are protected by a mutex. So solving the first issue solves the second naturally.
In general if there are multiple senders for a channel, then closing the channel must be coordinated. There must be a single entity (often not any of the senders) that waits for all senders to finish, practically using a sync.WaitGroup, and then that single entity can close the channel, safely. See Closing channel of unknown length.
I was following the chat client/server example for the gorilla websocket library.
https://github.com/gorilla/websocket/blob/master/examples/chat/hub.go#L36
I tried modifying the code to notify other clients when a new client connects, like so:
for {
select {
case client := <-h.register:
h.clients[client] = true
// My addition. Hangs after this (no further register/unregister events are processed):
h.broadcast <- []byte("Another client connected!")
case client := <-h.unregister:
if _, ok := h.clients[client]; ok {
delete(h.clients, client)
close(client.send)
}
case message := <-h.broadcast:
for client := range h.clients {
select {
case client.send <- message:
default:
close(client.send)
delete(h.clients, client)
}
}
}
}
}
My understanding is on the next iteration of the outer for loop, the broadcast channel should receive that data and follow the logic in the message case, but it just hangs.
Why? I can't spot any reason. No further channel events are processed (nothing on register/unregister or broadcast), which makes me think it's some kind of unbuffered channel mechanism it's stuck on, but I don't see how?
The hub's broadcast channel is unbuffered. Communication on an unbuffered channel waits for a ready sender and a ready receiver. The hub goroutine blocks because the goroutine cannot be ready to send and receive at the same time.
Changing the channel from an unbuffered channel to a buffered channel does not fix the problem. Consider the case where the buffer capacity is one:
return &Hub{
broadcast: make(chan []byte, 1),
...
}
with this timeline:
1 clientA: client.hub.register <- client
2 clientB: c.hub.broadcast <- message
3 hub: case client := <-h.register:
4 hub: h.broadcast <- []byte("Another client connected!")
The hub blocks at #4 because the channel was filled to capacity at #2. Increasing the channel capacity to two or more does not fix the problem because any number of clients can broadcast a message while another client is registering.
To fix the problem, move the broadcast code to a function and call that function from both cases in the select:
// sendAll sends message to all registered clients.
// This method must only be called by Hub.run.
func (h *Hub) sendAll(message []byte) {
for client := range h.clients {
select {
case client.send <- message:
default:
close(client.send)
delete(h.clients, client)
}
}
}
func (h *Hub) run() {
for {
select {
case client := <-h.register:
h.clients[client] = true
h.sendAll([]byte("Another client connected!"))
case client := <-h.unregister:
if _, ok := h.clients[client]; ok {
delete(h.clients, client)
close(client.send)
}
case message := <-h.broadcast:
h.sendAll(message)
}
}
}
Your channels are unbuffered, this means that each read/write blocks until an other goroutine performs the opposite operation on the same channel.
When you try to write to h.broadcast the goroutine stops, waiting for a reader. But the same goroutine is supposed to act as a reader of this channel, which never happens because the goroutine is blocked by the write. Thus the program deadlock.
Yea, this won't work. You cannot send/receive on the same unbuffered channel in the same go routine.
The line h.broadcast <- []byte("Another client connected!") blocks until another go routine pops off that queue. An easy solution is to make the broadcast channel have a length 1 buffer. broadcast := make(chan []byte, 1)
You can see it in this playground example
// c := make(chan int) <- This will hang
c := make(chan int, 1)
c <- 1
fmt.Println(<-c)
Remove the length 1 buffer and the whole system deadlocks. One issue you can run into is if 2 clients register at the same time, then you can have the case where 2 items are trying to be stuffed into the broadcast channel, and we're back to the same problem with the unbuffered channel. You can avoid this and keep 1 go routine like so:
for {
select {
case message := <-h.broadcast:
// ...
default:
}
select { // This select statement can only add 1 item to broadcast at most
case client := <-h.register:
// ...
h.broadcast <- []byte("Another client connected!")
}
}
}
However, this will still break if another go routine is also adding to the broadcast channel. So I'd go with Cerise Limon's solution, or buffer the channel enough that other go routines won't ever fill the buffer.
I'm a bit confused about the difference between buffered and unbuffered channels in Go. For example, the below code executes fine:
package main
import "fmt"
func main() {
messages := make(chan string)
go func() { messages <- "ping" }()
msg := <-messages
fmt.Println(msg)
}
On the other hand, when I pass "ping" to messages in a regular function, there is a deadlock.
package main
import "fmt"
func main() {
messages := make(chan string)
func() { messages <- "ping" }()
msg := <-messages
fmt.Println(msg)
}
Finally, this is fixed when I use a buffered channel, like so
package main
import "fmt"
func main() {
messages := make(chan string, 1)
func() { messages <- "ping" }()
msg := <-messages
fmt.Println(msg)
}
I'm confused why the second case failed. Go By Example says that
By default channels are unbuffered, meaning that they will only accept sends (chan <-) if there is a corresponding receive (<- chan) ready to receive the sent value. Buffered channels accept a limited number of values without a corresponding receiver for those values.
In all three cases, isn't msg the sink for messages?
Unbuffered channels block if they are not read from. Buffered channels will not block until they hit capacity.
Your first example actually starts a separate go routine which executes the function that tries to write "ping" to the messages channel. It will block until the statement to read from the messages channel executes. The statement to read from the messages channel is able to be hit since the function is on a separate goroutine.
Your second example declares and calls a function which attempts to write to the messages channel, but that channel will never be ready to be written to since you are executing on the same main execution thread. The statement to read from the messages channel never hits since you are blocked on writing to the channel.
The third example, the channel is buffered and can be written to since it can accept 1 value before it blocks.
In the first example, the nested function is called from another goroutine. The function starts running and blocks waiting to write the channel. The main goroutine also runs, and reads from the channel, releasing the block in the second goroutine.
In the second example, the nested function is called by main, and main waits for it to return. Since the channel is not buffered, the write operation blocks, which means the main goroutine is blocked and there are no others, so deadlock.
In the third example, the channel is buffered, so the first write does not block.
You may have skimmed over a key word there: "if there is a corresponding receive (<- chan) ready to receive the sent value".
In your first example, the receiver is running concurrently to the sender, so when the sender sends, the receiver is ready to receive at that time. In the second example, they are not running concurrently, so when the sender sends, nothing is ready to receive (because the receive operation won't run until the anonymous function returns), and the send blocks.
I have some code that is a job dispatcher and is collating a large amount of data from lots of TCP sockets. This code is a result of an approach to Large number of transient objects - avoiding contention and it largely works with CPU usage down a huge amount and locking not an issue now either.
From time to time my application locks up and the "Channel length" log is the only thing that keeps repeating as data is still coming in from my sockets. However the count remains at 5000 and no downstream processing is taking place.
I think the issue might be a race condition and the line it is possibly getting hung up on is channel <- msg within the select of the jobDispatcher. Trouble is I can't work out how to verify this.
I suspect that as select can take items at random the goroutine is returning and the shutdownChan doesn't have a chance to process. Then data hits inboundFromTCP and it blocks!
Someone might spot something really obviously wrong here. And offer a solution hopefully!?
var MessageQueue = make(chan *trackingPacket_v1, 5000)
func init() {
go jobDispatcher(MessageQueue)
}
func addMessage(trackingPacket *trackingPacket_v1) {
// Send the packet to the buffered queue!
log.Println("Channel length:", len(MessageQueue))
MessageQueue <- trackingPacket
}
func jobDispatcher(inboundFromTCP chan *trackingPacket_v1) {
var channelMap = make(map[string]chan *trackingPacket_v1)
// Channel that listens for the strings that want to exit
shutdownChan := make(chan string)
for {
select {
case msg := <-inboundFromTCP:
log.Println("Got packet", msg.Avr)
channel, ok := channelMap[msg.Avr]
if !ok {
packetChan := make(chan *trackingPacket_v1)
channelMap[msg.Avr] = packetChan
go processPackets(packetChan, shutdownChan, msg.Avr)
packetChan <- msg
continue
}
channel <- msg
case shutdownString := <-shutdownChan:
log.Println("Shutting down:", shutdownString)
channel, ok := channelMap[shutdownString]
if ok {
delete(channelMap, shutdownString)
close(channel)
}
}
}
}
func processPackets(ch chan *trackingPacket_v1, shutdown chan string, id string) {
var messages = []*trackingPacket_v1{}
tickChan := time.NewTicker(time.Second * 1)
defer tickChan.Stop()
hasCheckedData := false
for {
select {
case msg := <-ch:
log.Println("Got a messages for", id)
messages = append(messages, msg)
hasCheckedData = false
case <-tickChan.C:
messages = cullChanMessages(messages)
if len(messages) == 0 {
messages = nil
shutdown <- id
return
}
// No point running checking when packets have not changed!!
if hasCheckedData == false {
processMLATCandidatesFromChan(messages)
hasCheckedData = true
}
case <-time.After(time.Duration(time.Second * 60)):
log.Println("This channel has been around for 60 seconds which is too much, kill it")
messages = nil
shutdown <- id
return
}
}
}
Update 01/20/16
I tried to rework with the channelMap as a global with some mutex locking but it ended up deadlocking still.
Slightly tweaked the code, still locks but I don't see how this one does!!
https://play.golang.org/p/PGpISU4XBJ
Update 01/21/17
After some recommendations I put this into a standalone working example so people can see. https://play.golang.org/p/88zT7hBLeD
It is a long running process so will need running locally on a machine as the playground kills it. Hopefully this will help get to the bottom of it!
I'm guessing that your problem is getting stuck doing this channel <- msg at the same time as the other goroutine is doing shutdown <- id.
Since neither the channel nor the shutdown channels are buffered, they block waiting for a receiver. And they can deadlock waiting for the other side to become available.
There are a couple of ways to fix it. You could declare both of those channels with a buffer of 1.
Or instead of signalling by sending a shutdown message, you could do what Google's context package does and send a shutdown signal by closing the shutdown channel. Look at https://golang.org/pkg/context/ especially WithCancel, WithDeadline and the Done functions.
You might be able to use context to remove your own shutdown channel and timeout code.
And JimB has a point about shutting down the goroutine while it might still be receiving on the channel. What you should do is send the shutdown message (or close, or cancel the context) and continue to process messages until your ch channel is closed (detect that with case msg, ok := <-ch:), which would happen after the shutdown is received by the sender.
That way you get all of the messages that were incoming until the shutdown actually happened, and should avoid a second deadlock.
I'm new to Go but in this code here
case msg := <-inboundFromTCP:
log.Println("Got packet", msg.Avr)
channel, ok := channelMap[msg.Avr]
if !ok {
packetChan := make(chan *trackingPacket_v1)
channelMap[msg.Avr] = packetChan
go processPackets(packetChan, shutdownChan, msg.Avr)
packetChan <- msg
continue
}
channel <- msg
Aren't you putting something in channel (unbuffered?) here
channel, ok := channelMap[msg.Avr]
So wouldn't you need to empty out that channel before you can add the msg here?
channel <- msg
Like I said, I'm new to Go so I hope I'm not being goofy. :)
I am trying to write a RabbitMQ Consumer in Go. Which is suppose to take the 5 objects at a time from the queue and process them. Moreover, it is suppose to acknowledge if successfully processed else send to the dead-letter queue for 5 times and then discard, it should be running infinitely and handling the cancellation event of the consumer.
I have few questions :
Is there any concept of BasicConsumer vs EventingBasicConsumer in RabbitMq-go Reference?
What is Model in RabbitMQ and is it there in RabbitMq-go?
How to send the objects when failed to dead-letter queue and again re-queue them after ttl
What is the significance of consumerTag argument in the ch.Consume function in the below code
Should we use the channel.Get() or channel.Consume() for this scenario?
What are the changes i need to make in the below code to meet above requirement. I am asking this because i couldn't find decent documentation of RabbitMq-Go.
func main() {
consumer()
}
func consumer() {
objConsumerConn := &rabbitMQConn{queueName: "EventCaptureData", conn: nil}
initializeConn(&objConsumerConn.conn)
ch, err := objConsumerConn.conn.Channel()
failOnError(err, "Failed to open a channel")
defer ch.Close()
msgs, err := ch.Consume(
objConsumerConn.queueName, // queue
"demo1", // consumerTag
false, // auto-ack
false, // exclusive
false, // no-local
false, // no-wait
nil, // args
)
failOnError(err, "Failed to register a consumer")
forever := make(chan bool)
go func() {
for d := range msgs {
k := new(EventCaptureData)
b := bytes.Buffer{}
b.Write(d.Body)
dec := gob.NewDecoder(&b)
err := dec.Decode(&k)
d.Ack(true)
if err != nil { fmt.Println("failed to fetch the data from consumer", err); }
fmt.Println(k)
}
}()
log.Printf(" Waiting for Messages to process. To exit press CTRL+C ")
<-forever
}
Edited question:
I have delayed the processing of the messages as suggested in the links link1 link2. But the problem is messages are getting back to their original queue from dead-lettered queue even after ttl. I am using RabbitMQ 3.0.0. Can anyone point out what is the problem?
Is there any concept of BasicConsumer vs EventingBasicConsumer in
RabbitMq-go Reference?
Not exactly, but the Channel.Get and Channel.Consume calls serve a similar concept. With Channel.Get you have a non-blocking call that gets the first message if there's any available, or returns ok=false. With Channel.Consume the queued messages are delivered to a channel.
What is Model in RabbitMQ and is it there in RabbitMq-go?
If you're referring to the IModel and Connection.CreateModel in C# RabbitMQ, that's something from the C# lib, not from RabbitMQ itself. It was just an attempt to abstract away from the RabbitMQ "Channel" terminology, but it never caught on.
How to send the objects when failed to dead-letter queue and again
re-queue them after ttl
Use the delivery.Nack method with requeue=false.
What is the significance of consumerTag argument in the ch.Consume
function in the below code
The ConsumerTag is just a consumer identifier. It can be used to cancel the channel with channel.Cancel, and to identify the consumer responsible for the delivery. All messages delivered with the channel.Consume will have the ConsumerTag field set.
Should we use the channel.Get() or channel.Consume() for this scenario?
I think channel.Get() is almost never preferable over channel.Consume(). With channel.Get you'll be polling the queue and wasting CPU doing nothing, which doesn't make sense in Go.
What are the changes i need to make in the below code to meet above
requirement.
Since you're batch processing 5 at a time, you can have a goroutine that receives from the consumer channel and once it gets the 5 deliveries you call another function to process them.
To acknowledge or send to the dead-letter queue you'll use the delivery.Ack or delivery.Nack functions. You can use multiple=true and call it once for the batch. Once the message goes to the dead letter queue, you have to check the delivery.Headers["x-death"] header for how many times its been dead-lettered and call delivery.Reject when its been retried for 5 times already.
Use channel.NotifyCancel to handle the cancellation event.