I'm new to this Apache Kafka topic and I was writing some basic producer-consumer code, and I'm facing some issues with consumer code, after starting the zookeeper and Kafka, I created a topic name "firsttopic", and I'm entering some events using CLI commands as a producer and to retrieve those events as a consumer I've written a go code using Kafka-go that I'm attaching below and the error that I'm facing too. For Kafka I'm using "github.com/segmentio/kafka-go".
func Startkafka() {
conf := kafka.ReaderConfig{
Brokers: []string{"localhost:9092"},
Topic: "firsttopic",
GroupID: "g1",
MaxBytes: 10,
}
reader := kafka.NewReader(conf)
for {
m, err := reader.ReadMessage((context.Background()))
if err != nil {
fmt.Println("Some error occured", err)
continue
}
fmt.Println("Message is : ", string(m.Value))
}}
func main() {
go Startkafka()
fmt.Println("Kafka has been started...")
}
Error:
Kafka has been started...
Some error occured read tcp 127.0.0.1:34858->127.0.1.1:9092: i/o timeout
Based on the error, you should remove 127.0.1.1 from /etc/hosts mapping to localhost as it ought to remain as 127.0.0.1 during the client connection to localhost
Alternatively, connect to 127.0.0.1:9092 instead
MaxBytes: 10 is very low, You should increase that as that may be causing an issue. The rest of the code seems to be correct. you should first try connecting using Kafka-consumer shipped with Kafka installation whether it is up which will also verify the connection string that you are using.
In you main function, that StartKafka() calling in a go routine. Then it is start in another stack and run concurrently while main routine end the execution.
Remove that go routine and run the application. it will blocking the application and consume from Kafka topic.
If you need to build proper graceful shutdown process for the application, you have to use channels or something. But following code simply works for consume the topic content.
func main() {
fmt.Println("starting kafka...")
Startkafka()
}
Related
I am working on an application that tries to send some data to a remote server. Once I get the hostname, I get a connection by resolving the hostname using net.Dialer.DialContext. Once, I resolve the hostname, I keep on using conn.Write method to write data to the connection.
conn, err := d.DialContext(ctx, string(transport), addr)
_, err := client.conn.Write([]byte(msg))
Error faced: I observed that due to some issues, I was not able to ping my server. Surprisingly, conn obtained from DialContext did not complain while doing conn.Write and it kept writing to the same connection.
Can someone help me in how to modify my writing methods in order to get an error in case the destination server is not reachable?
From this UDP connection example
the best a "connected" UDP socket can do to simulate a send failure is to save the ICMP response, and return it as an error on the next write.
So try and (for testing) make a second conn.Write, to confirm that you would indeed get an error this time.
I have a grpc server and a client (in my blog project). when I run the server, It seems everything is ok, when I run the client, I face this error, and both server and client close.
rpc error: code = Unavailable desc = transport is closing
I think error is related to this piece of code:
func newPost(c proto_blog.BlogServiceClient) {
fmt.Println("Starting to do a Unary RPC")
req := &proto_blog.ReqNewPost{
Title: "How can we make an gRPC server?",
Content: "First You have to.....\nAt the end, you have to....",
Author: "Arsham Ahora",
Date: fmt.Sprint(time.Now()),
}
res, err := c.NewPost(context.Background(), req)
if err != nil {
log.Fatalf("Error calling greet server: %v", err)
}
log.Printf("Response from Greet: %v", res.Id)
}
** I noticed this error is not related to whether you have Unary or Streaming.
I want to list some possible reasons cause code = Unavailable desc = transport is closing on gRPC per gRPC faq, in case of someone meets those reasons.
This error means the connection the RPC is using was closed, and there are many possible reasons, including:
mis-configured transport credentials, connection failed on handshaking
bytes disrupted, possibly by a proxy in between
server shutdown
Keepalive parameters caused connection shutdown, for example if you have configured your server to terminate connections regularly to trigger DNS lookups. If this is the case, you may want to increase your MaxConnectionAgeGrace, to allow longer RPC calls to finish.
It can be tricky to debug this because the error happens on the client side but the root cause of the connection being closed is on the server side. Turn on logging on both client and server, and see if there are any transport errors.
The default logger is controlled by environment variables. Turn everything on like this:
$ export GRPC_GO_LOG_VERBOSITY_LEVEL=99
$ export GRPC_GO_LOG_SEVERITY_LEVEL=info
I found that in server code I have a code like this before returning the response:
log.Fatalf("Error happend: %v", e)
And I changed my code like this:
if e != nil {
log.Fatalf("Error happend: %v", e)
}
That error has not occurred but log.Fatalf() has broken my app.
For more details, it was not an error directly from the grpc part, it was because my app broke before returning any response to the gRPC client.
I think you sent wrong piece of code, anyway, As the error was said: "transport is closing" your connection is closed, You have to find where in your server you are exiting your server and handle that.
I've installed IBM MQ (8.0.0.4) on a local Windows VM and followed the instructions in the docs to enable AMQP (https://www.ibm.com/support/knowledgecenter/en/SSFKSJ_8.0.0/com.ibm.mq.con.doc/tamqp_creating.htm), and also disabled authentication since this is just a local development environment. With that done, I am able to connect from my host OS using the node.js sample in IBM's mqlight package:
>npm install mqlight
...
>node node_modules/mqlight/samples/recv.js -s amqp://windows-10:5672
Connected to amqp://windows-10:5672 using client-id recv_126117c
Subscribed to pattern: public
However, when attempting to connect from a golang app using vcabbage/amqp (version 0.12.5), it returns an EOF error on the attempt to dial the host. Minimal example:
package main
import (
"fmt"
"pack.ag/amqp"
)
func main() {
_, err := amqp.Dial("amqp://windows-10:5672")
fmt.Println(err) // EOF
}
Nothing is appearing in the IBM MQ error logs.
Unfortunately Google does not turn up any relevant results for connecting to IBM MQ via AMQP 1.0 in golang, so I'm stuck. Does anyone have any ideas?
So the solution is apparently to use SASL Anonymous mode; this allows the client to connect.
package main
import (
"fmt"
"pack.ag/amqp"
)
func main() {
_, err := amqp.Dial("amqp://windows-10:5672", amqp.ConnSASLAnonymous())
fmt.Println(err) // nil
}
If anybody wants to try to make it work in "normal" mode, it appears that IBM MQ was closing the channel as soon as the initial header packet was sent. The EOF was bubbling up from the receiving goroutine in any case.
I am using the http://github.com/streadway/amqp package in my application in order to handle connections to a remote RabbitMQ server. Everything is ok and works fine, but I have a question.
The current name for a connection is "ip:port", so when there are multiple connections from the same IP+port, they become hardly distinguishable. It would be great if you can specify a name for each connection.
Is there any way to set a distinct friendly name for each connection?
RabbitMQ 3.6.5 added the facility for the connecting client to report a friendly name string value to identify a connection for management purposes. This is strictly an identifier and, as it is client-reported, it cannot be relied upon for anything other than weak identification of connections. The release notes state:
Clients now can provide a human-readable connection name that will be displayed in the management UI... In order to use this feature, set the connection_name key in client properties. Note that this name doesn’t have to be unique and cannot be used as a connection identifier, for example, in HTTP API requests.
Solution
Provided you are using a sufficiently new version of RabbitMQ, you can set this parameter when making connections using streadway/amqp by passing an instance of amqp.Config when making the initial connection. The Properties field allows custom properties of the connection to be specified.
The example program below opens a connection using the AMQP URL provided in the environment variable AMQP_URL, identified using the connection name passed as the first command line argument to the invocation.
package main
import (
"log"
"os"
"github.com/streadway/amqp"
)
func main() {
amqpUrl := os.Getenv("AMQP_URL")
cfg := amqp.Config{
Properties: amqp.Table{
"connection_name": os.Args[1],
},
}
conn, err := amqp.DialConfig(amqpUrl, cfg)
if err != nil {
log.Fatal(err)
}
defer conn.Close()
<-(chan struct{})(nil)
}
Starting multiple instances to connect to a local RabbitMQ instance using the following command line:
AMQP_URL=amqp://admin:password#localhost:5672 go run ./main.go connX
where a numeral is substituted for X yields the following output in the "Connections" page of the RabbitMQ Management web UI:
and the individual connection detail pages shows the value under the "Client-provided name" detail value:
I'm using gin to build a simple api server. right now, after the server started a few minutes later, all of incoming requests were stuck. After checking connections state , I got a message like below :
I already have tried to set c.Request.Close=true or c.Header("Connection","close") or both of them, but it was not working. I was wondering if anyone can help me fix this issue.
UPDATE-1 the way to start server
runtime.GOMAXPROCS(runtime.NumCPU())
//start serving
r := gin.New()
r.Use(gin.LoggerWithWriter(logFile))
r.Use(gin.RecoveryWithWriter(logFile))
r.Use(handler.SetResponseHeader())
controller.Router.RegisterRouter(r, cfg)
//r.Run(cfg.SvcHost)
s := &http.Server{
Addr: cfg.SvcHost,
Handler: r,
}
s.ListenAndServe()
UPDATE-2
after temporarily disabling the global gorm.DB instance and creating a new db connection for each request, this issue didn't occur any more.
TCP is waiting for you to close those connections. Probably you have read end of stream from them and have ignored it.