I need kafka consumer logs for debug. I do the following:
chanLogs := make(chan confluentkafka.LogEvent)
go func() {
for {
logEv := <-chanLogs
logger.Debug("KAFKA: " + logEv.String())
}
}()
configMap["go.logs.channel.enable"] = true
configMap["go.logs.channel"] = chanLogs
consumer, err := confluentkafka.NewConsumer(&configMap)
err := consumer.SubscribeTopics(Topics, nil)
And I never get a line. I tried it with kafka chan (consumer.Logs()) with the same result. What I do wrong?
UPD
In initial post I wrongfully set parameter name. The correct one is go.logs.channel.enable. But sometimes this still don't work.
As described in the doc, you should enable that feature:
go.logs.channel.enable (bool, false) - Forward log to Logs() channel.
go.logs.channel (chan kafka.LogEvent, nil) - Forward logs to application-provided channel instead of Logs(). Requires go.logs.channel.enable=true.
So change your code like:
configMap["go.logs.channel"] = chanLogs
configMap["go.logs.channel.enable"] = true
consumer, err := confluentkafka.NewConsumer(&configMap)
See also in the doc or in the sample on the code repo here
The solution was to add
configMap["debug"] = "all"
I found it here
Related
I am connecting to a websocket that is stream live stock trades.
I have to read the prices, perform calculations on the fly and based on these calculations make another API call e.g. buy or sell.
I want to ensure my calculations/processing doesn't slow down my ability to stream in all the live data.
What is a good design pattern to follow for this type of problem?
Is there a way to log/warn in my system to know if I am falling behind?
Falling behind means: the websocket is sending price data, and I am not able to process that data as it comes in and it is lagging behind.
While doing the c.ReadJSON and then passing the message to my channel, there might be a delay in deserializing into JSON
When inside my channel and processing, calculating formulas and sending another API request to buy/sell, this will add delays
How can I prevent lags/delays and also monitor if indeed there is a delay?
func main() {
c, _, err := websocket.DefaultDialer.Dial("wss://socket.example.com/stocks", nil)
if err != nil {
panic(err)
}
defer c.Close()
// Buffered channel to account for bursts or spikes in data:
chanMessages := make(chan interface{}, 10000)
// Read messages off the buffered queue:
go func() {
for msgBytes := range chanMessages {
logrus.Info("Message Bytes: ", msgBytes)
}
}()
// As little logic as possible in the reader loop:
for {
var msg interface{}
err := c.ReadJSON(&msg)
if err != nil {
panic(err)
}
chanMessages <- msg
}
}
You can read bytes, pass them to the channel, and use other goroutines to do conversion.
I worked on a similar crypto market bot. Instead of creating large buffured channel i created buffered channel with cap of 1 and used select statement for sending socket data to channel.
Here is the example
var wg sync.WaitGroup
msg := make(chan []byte, 1)
wg.Add(1)
go func() {
defer wg.Done()
for data := range msg {
// decode and process data
}
}()
for {
_, data, err := c.ReadMessage()
if err != nil {
log.Println("read error: ", err)
return
}
select {
case msg <- data: // in case channel is free
default: // if not, next time will try again with latest data
}
}
This will insure that you'll get the latest data when you are ready to process.
I'm integrating Binance API into an existing system and while most parts a straight forward, the data streaming API hits my limited understanding of go-routines. I don't believe there is anything special in the golang SDK for Binance, but essentially I only need two functions, one that starts the data stream and processes events with the event handler given as a parameter and a second one that ends the data stream without actually shutting down the client as it would close all other connections. On a previous project, there were two message types for this, but the binance SDK uses an implementation that returns two go channels, one for errors and an another one, I guess from the name, for stopping the data stram.
The code I wrote for starting the data stream looks like this:
func startDataStream(symbol, interval string, wsKlineHandler futures.WsKlineHandler, errHandler futures.ErrHandler) (err error){
doneC, stopC, err := futures.WsKlineServe(symbol, interval, wsKlineHandler, errHandler)
if err != nil {
fmt.Println(err)
return err
}
return nil
}
This works as expected and streams data. A simple test verifies it:
func runWSDataTest() {
symbol := "BTCUSDT"
interval := "15m"
errHandler := func(err error) {fmt.Println(err)}
wsKlineHandler := func(event *futures.WsKlineEvent) {fmt.Println(event)}
_ = startDataStream(symbol, interval, wsKlineHandler, errHandler)
}
The thing that is not so clear to me, mainly due to incomplete understanding, really is how do I stop the stream. I think the returned stopC channel can be used to somehow issue a end singnal similar to, say, a sigterm on system level and then the stream should end.
Say, I have a stopDataStream function that takes a symbol as an argument
func stopDataStream(symbol){
}
Let's suppose I start 5 data streams for five symbols and now I want to stop just one of the streams. That begs the question of:
How do I track all those stopC channels?
Can I use a collection keyed with the symbol, pull the stopC channel, and then just issue a signal to end just that data stream?
How do I actually write into the stopC channel from the stop function?
Again, I don't think this is particularly hard, it's just I could not figure it out yet from the docs so any help would be appreciated.
Thank you
(Answer originally written by #Marvin.Hansen)
Turned out, just saving & closing the channel solved it all. I was really surprised how easy this is, but here is the code of the updated functions:
func startDataStream(symbol, interval string, wsKlineHandler futures.WsKlineHandler, errHandler futures.ErrHandler) (err error) {
_, stopC, err := futures.WsKlineServe(symbol, interval, wsKlineHandler, errHandler)
if err != nil {
fmt.Println(err)
return err
}
// just save the stop channel
chanMap[symbol] = stopC
return nil
}
And then, the stop function really becomes embarrassing trivial:
func stopDataStream(symbol string) {
stopC := chanMap[symbol] // load the stop channel for the symbol
close(stopC) // just close it.
}
Finally, testing it all out:
var (
chanMap map[string]chan struct{}
)
func runWSDataTest() {
chanMap = make(map[string]chan struct{})
symbol := "BTCUSDT"
interval := "15m"
errHandler := func(err error) { fmt.Println(err) }
wsKlineHandler := getKLineHandler()
println("Start stream")
_ = startDataStream(symbol, interval, wsKlineHandler, errHandler)
time.Sleep(3 * time.Second)
println("Stop stream")
stopDataStream(symbol)
time.Sleep(1 * time.Second)
}
This is it.
I'm new to Go lang and I could use some suggestions on how to refactor the code. All I got to do is depending on the success or error from Sarama (Apache Kafka thing in go) I need to log and forward it further. So far my code looks like this
go func() {
for err := range producer.Errors() {
batchID := err.Msg.Metadata.(ackMeta).batchID # notice the struct here
statusChan := err.Msg.Metadata.(ackMeta).statusChan
statusChan <- false
close(statusChan)
logs.Debug(appName, "Signalled failure on statusChan for batch ", batchID)
logs.Error(appName, "Failed to publish data to analyzer for batchID: ", batchID, err)
}
}()
go func() {
for succ := range producer.Successes() {
batchID := succ.Metadata.(ackMeta).batchID # notice the struct here
statusChan := succ.Metadata.(ackMeta).statusChan
statusChan <- true
close(statusChan)
logs.Debug(appName, "Signalled success on statusChan for batch ", batchID)
logs.Debug(appName, "Successfully published data to analyzer:", succ.Topic, succ.Key, succ.Partition, succ.Offset, succ.Metadata)
}
I think I can do a better job and wrap the whole thing in a single function but so far I can't think of any other apart from using the switch case as shown here
func checkSuccessOrFailAck(msg interface{}) {
switch msgType := msg.(type) {
case producer.Errors:
batchID := msg.Msg.Metadata.(ackMeta).batchID
statusChan := msg.Msg.Metadata.(ackMeta).statusChan
statusChan <- false
close(statusChan)
logs.Debug(appName, "Signalled failure on statusChan for batch ", batchID)
logs.Error(appName, "Failed to publish data to analyzer for batchID: ", batchID, msg)
case producer.Successes:
batchID := msg.Metadata.(ackMeta).batchID
statusChan := msg.Metadata.(ackMeta).statusChan
statusChan <- true
close(statusChan)
logs.Debug(appName, "Signalled success on statusChan for batch ", batchID)
logs.Debug(appName, "Successfully published data to analyzer:", succ.Topic, succ.Key, succ.Partition, succ.Offset, succ.Metadata)
}
}
The types of messages are different and so is the way I extract the attributes from it. But I'm not happy with this approach as the statements are more than the previous one. Could there be a better way to think to write ?
First, it's not clear if the second code works at all. Since Errors() and Successes() return channels, you'd need to select to read from both of them simultaneously without blocking either.
Since succ and err have different types in the code, I'm not sure you can do it much shorter than your original sample. Go doesn't have generics yet, so writing code that would unify the two is challenging. If you really need it I'd try to find the closest points where the type is the same - it should be easier to unify code from that point on. for example err.Msg and succ are the same type?
Moreover, even if Go had generics already, it's not clear unifying these would be a win. Go's philosophy is that a little code repetition is better than over-complicated abstractions. The code isn't the shortest it could be in theory, but it's explicit and clear.
You could do them both in a single goroutine using a select:
go func() {
errsOpen := true
succsOpen := true
for errsOpen && succsOpen {
select {
case err,errsOpen := <- producer.Errors()
batchID := err.Msg.Metadata.(ackMeta).batchID # notice the struct here
statusChan := err.Msg.Metadata.(ackMeta).statusChan
statusChan <- false
close(statusChan)
logs.Debug(appName, "Signalled failure on statusChan for batch ", batchID)
logs.Error(appName, "Failed to publish data to analyzer for batchID: ", batchID, err)
case succ,succsOpen := <- producer.Successes()
batchID := succ.Metadata.(ackMeta).batchID # notice the struct here
statusChan := succ.Metadata.(ackMeta).statusChan
statusChan <- true
close(statusChan)
logs.Debug(appName, "Signalled success on statusChan for batch ", batchID)
logs.Debug(appName, "Successfully published data to analyzer:", succ.Topic, succ.Key, succ.Partition, succ.Offset, succ.Metadata)
}
}
}()
This will loop until both channels have been closed, and at each loop iteration, handle a value from whichever channel has a value waiting to be received (or if both have values waiting, choose one at random).
I am using Kafka 10.0 and https://github.com/Shopify/sarama.
I am trying to get the offset of the latest message that a consumer processed.
To do so I've found the method NewOffsetManagerFromClient(group string, client Client) which require the group name.
How do I get consumer group name?
offsets := make(map[int32]int64)
config := sarama.NewConfig()
config.Consumer.Offsets.CommitInterval = 200 * time.Millisecond
config.Version = sarama.V0_10_0_0
// config.Consumer.Offsets.Initial = sarama.OffsetNewest
cli, _ := sarama.NewClient(kafkaHost, config)
defer cli.Close()
offsetManager, _ := sarama.NewOffsetManagerFromClient(group, cli)
for _, partition := range partitions {
partitionOffsetManager, _ := offsetManager.ManagePartition(topic, partition)
offset, _ := partitionOffsetManager.NextOffset()
offsets[partition] = offset
}
return offsets
I created a consumer with
consumer := sarama.NewConsumer(connections, config)
but I do not know how to create a consumer group and get its group name.
You are attempting to create your own offset manager to find current offsets:
offsetManager, _ := sarama.NewOffsetManagerFromClient(group, cli)
Similarly, the consumer that was consuming your topic's messages would have to use the same offset manager and they would have used a specific group id. Use that group id.
I think you can use any string as groupId. Please look at the example from sarama GoDoc
// Start a new consumer group
group, err := NewConsumerGroupFromClient("my-group", client)
if err != nil {
panic(err)
}
defer func() { _ = group.Close() }()
Maybe you can give it any string. And you should make sure the other consumers can get the same groupId for joining the group.
So I am trying to use Kafka for my application which has a producer logging actions into the Kafka MQ and the consumer which reads it off the MQ.Since my application is in Go, I am using the Shopify Sarama to make this possible.
Right now, I'm able to read off the MQ and print the message contents using a
fmt.Printf
Howeveer, I would really like the error handling to be better than console printing and I am willing to go the extra mile.
Code right now for consumer connection:
mqCfg := sarama.NewConfig()
master, err := sarama.NewConsumer([]string{brokerConnect}, mqCfg)
if err != nil {
panic(err) // Don't want to panic when error occurs, instead handle it
}
And the processing of messages:
go func() {
defer wg.Done()
for message := range consumer.Messages() {
var msgContent Message
_ = json.Unmarshal(message.Value, &msgContent)
fmt.Printf("Reading message of type %s with id : %d\n", msgContent.Type, msgContent.ContentId) //Don't want to print it
}
}()
My questions (I am new to testing Kafka and new to kafka in general):
Where could errors occur in the above program, so that I can handle them? Any sample code will be great for me to start with. The error conditions I could think of are when the msgContent doesn't really contain any Type of ContentId fields in the JSON.
In kafka, are there situations when the consumer is trying to read at the current offset, but for some reason was not able to (even when the JSON is well formed)? Is it possible for my consumer to backtrack to say x steps above the failed offset read and re-process the offsets? Or is there a better way to do this? again, what could these situations be?
I'm open to reading and trying things.
Regarding 1) Check where I log error messages below. This is more or less what I would do.
Regarding 2) I don't know about trying to walk backwards in a topic. Its very much possible by just creating a consumer over and over, with its starting offset minus one each time. But I wouldn't advise it, as most likely you'll end up replaying the same message over and over. I do advice saving your offset often so you can recover if things go south.
Below is code that I believe addresses most of your questions. I haven't tried compiling this. And sarama api has been changing lately, so the api may currently differ a bit.
func StartKafkaReader(wg *sync.WaitGroup, lastgoodoff int64, out chan<- *Message) (error) {
wg.Add(1)
go func(){
defer wg.Done()
//to track the last known good offset we processed, which is
// updated after each successfully processed event.
saveprogress := func(off int64){
//Save the offset somewhere...a file...
//Ive also used kafka to store progress
//using a special topic as a WAL
}
defer saveprogress(lastgoodoffset)
client, err := sarama.NewClient("clientId", brokers, sarama.NewClientConfig())
if err != nil {
log.Error(err)
return
}
defer client.Close()
sarama.NewConsumerConfig()
consumerConfig.OffsetMethod = sarama.OffsetMethodManual
consumerConfig.OffsetValue = int64(lastgoodoff)
consumer, err := sarama.NewConsumer(client, topic, partition, "consumerId", consumerConfig)
if err != nil {
log.Error(err)
return
}
defer consumer.Close()
for {
select {
case event := <-consumer.Events():
if event.Err != nil {
log.Error(event.Err)
return
}
msgContent := &Message{}
err = json.Unmarshal(message.Value, msgContent)
if err != nil {
log.Error(err)
continue //continue to skip this message or return to stop without updating the offset.
}
// Send the message on to be processed.
out <- msgContent
lastgoodoff = event.Offset
}
}
}()
}