I’m having what seems to be a timeout issue with the go client. Essentially I have a service running in AWS, an endpoint for a heartbeat (that does not connect to edgedb), and an endpoint to list all of a particular object in the DB (“select Foo { ... };“).
When the service is started both of these endpoints function as expected, but over the course of the day the list endpoint will start to hang forever (I’m assuming because the client is timing out connecting to the DB?) The heartbeat route continues to function normally so AWS doesn’t mark the service as unhealthy. Is there maybe a connection option I’m missing?
Here’s how I create the client:
log.Println("Connecting to EdgeDB...")
ctx := context.Background()
var opts edgedb.Options
opts = edgedb.Options{
Database: os.Getenv("DB_DBNAME"),
User: os.Getenv("DB_USER"),
Password: edgedb.NewOptionalStr(os.Getenv("DB_PASSWORD")),
Host: os.Getenv("DB_HOST"),
Port: 5656,
Concurrency: uint(4),
TLSOptions: edgedb.TLSOptions{
SecurityMode: "insecure",
},
}
db, dbErr := edgedb.CreateClient(ctx, opts)
return projectDb{DB: db}, db, dbErr
This timeout behavior seems to happen with a local edgedb eventually as well, but no logged errors
Related
I'm trying to access my ETCD database from a K8s controller, but getting rpc error/EOF when trying to open ETCD client.
My setup:
ETCD service is deployed in my K8s cluster and included in my Istio service mesh (its DNS record: my-etcd-cluster.my-etcd-namespace.svc.cluster.local)
I have a custom K8s controller developed with use of Kubebuilder framework and deployed in the same cluster, different namespace, but configured to be a part of the same Istio service mesh
I'm trying to connect to ETCD database from the controller, using Go client SDK library for ETCD
Here's my affected Go code:
cli, err := clientv3.New(clientv3.Config{
Endpoints: []string{"http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379"},
DialTimeout: 5 * time.Second,
Username: username,
Password: password,
})
if err != nil {
return nil, fmt.Errorf("opening ETCD client failed: %v", err)
}
And here's an error I'm getting when clientv3.New(...) gets executed:
{"level":"warn","ts":"2022-03-16T23:37:42.174Z","logger":"etcd-client","caller":"v3#v3.5.0/retry_interceptor.go:62","msg":"retrying of unary invoker failed",
"target":"etcd-endpoints://0xc00057f500/#initially=[http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379]","attempt":0,
"error":"rpc error: code = Unavailable desc = error reading from server: EOF"}
...
1.647473862175209e+09 INFO controller.etcdclient Finish reconcile loop for some-service/test-svc-client {"reconciler group": "my-controller.something.io", "reconciler kind": "ETCDClient", "name": "test-svc-client", "namespace": "some-service", "reconcile-etcd-client": "some-service/test-svc-client"}
1.6474738621752858e+09 ERROR controller.etcdclient Reconciler error {"reconciler group": "my-controller.something.io", "reconciler kind": "ETCDClient", "name": "test-svc-client", "namespace": "some-service", "error": "opening ETCD client failed: rpc error: code = Unavailable desc = error reading from server: EOF"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime#v0.11.0/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime#v0.11.0/pkg/internal/controller/controller.go:227
The same error happens when I'm passing some dummy, invalid credentials.
However, when I tried to access the database in a HTTP API manner:
postBody, _ := json.Marshal(map[string]string{
"name": username,
"password": password,
})
responseBody := bytes.NewBuffer(postBody)
resp, err := http.Post("http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379/v3/auth/authenticate", "application/json", responseBody)
if err != nil {
return ctrl.Result{}, fmt.Errorf("an error occured %w", err)
}
l.Info(fmt.Sprintf("code: %d", resp.StatusCode))
defer resp.Body.Close()
...I got 200 OK and a proper token (which is expected), so I believe my Istio configuration is ok and my controller should be able to see the ETCD db service. I have no clue why this doesn't work when following the client SDK approach.
When I'm using port-forwarding of the ETCD service and accessing it locally, clientv3.New() and other client SDK methods work like a charm. What am I missing? I'd really appreciate any suggestions.
EDIT:
I've also added a simple pod to try accessing my etcd db via etcdctl:
apiVersion: v1
kind: Pod
metadata:
name: test-pod
namespace: my-controller-namespace
spec:
containers:
- name: etcdctl
image: bitnami/etcd
command:
- sleep
- infinity
When logged into the container via kubectl exec, I was able to access my db:
$ etcdctl --endpoints=my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379 --user="user" --password="password" put foo bob
OK
I guess the problem is somewhere in the SDK?
Turned out to be version mismatch - my ETCD db is v3.5.2 and the clientv3 library that I used was v3.5.0.
As seen in ETCD changelog (https://github.com/etcd-io/etcd/blob/main/CHANGELOG/CHANGELOG-3.5.md):
I'm using a Go API and I need to connect it to my Cassandra DB on a Google Cloud server.
This is my Go connection code:
func SetupDBConnection() {
cluster := gocql.NewCluster("XX.XXX.XXX.XXX")
cluster.Keyspace = "afterstr"
cluster.Consistency = gocql.Quorum
s, err := cluster.CreateSession()
if err != nil {
fmt.Println(err)
}
}
But, when I run this code, I have this error:
gocql: unable to create session: control: unable to connect to initial hosts: dial tcp xx.xxx.xxx.xxx:9042: i/o timeout
This is my Google Cloud configuration:
I also modified the cassandra.yaml file in my server to change the listen address and the RPC address to put my IP with the port I use to run my Go program that I opened.
(It is opened 3 times because I was just testing something)
Try:
cluster.Authenticator = gocql.PasswordAuthenticator{Username: username, Password: password}
I have a server-side streaming RPC hosted on Google Cloud Run.
With the following proto definition:
syntax = "proto3";
package test.v1;
service MyService {
// Subscribe to a stream of events.
rpc Subscribe (SubscribeRequest) returns (stream SubscribeResponse) {}
}
message SubscribeRequest {
}
message SubscribeResponse {
}
Using BloomRPC/grpcurl, when I stop the method I get a stream.Context().Done() event which I can use to gracefully stop certain tasks. Here is an example of the Suscribe method:
func (s *myService) Subscribe(req *pb.SubscribeRequest, stream pb.Instruments_SubscribeServer) error {
// Create a channel for this client.
ch := make(chan *pb.SubscribeResponse)
// Add the channel object 'ch' to a Global list of channels where we have a 'broadcaster' sending
// messages to all connected clients.
// TODO: pass to broadcaster client list.
for {
select {
case <-stream.Context().Done():
close(ch)
fmt.Println("Removed client from global list of channels")
return nil
case res := <-ch:
_ = stream.Send(res)
}
}
}
On the client side, when I test the service locally (i.e. running a local gRPC server in Golang), using BloomRPC/grpcurl I get a message on the stream.Context().Done() channel whenever I stop the BloomRPC/grpcurl connection. This is the expected behaviour.
However, running the exact same code on Cloud Run in the same way (via BloomRPC/grpcurl), I don't get a stream.Context().Done() message - any reason why this would be different on Google Cloud Run? Looking at the Cloud Run logs, a call to the Subscribe method essentially 'hangs' until the request reaches its timeout.
I needed to enable HTTP/2 Connections in Cloud Run for this to work.
I'm using gin framework to build an API server. In General, I'm build 2 projects. Project 'API' and Project 'SOCKET'. Project 'API' is the main REST API that will used in Android, developed using gin framework (golang). And Project 'SOCKET' is the socket server for client that will use socket connection , using node.js (Socket.IO)
The process begin like this :
User A : as the requester ; A connect to "API"
User B : as the responder ; B connect to "SOCKET"
User A call API requestData from android, the request will handled by "API"'s project. And Project "API" will record the request, and publish on redis
as new_request using pubsub
this is the code for example :
client := redis.NewClient(&redis.Options{
Addr: "localhost:6379",
Password: "", // no password set
DB: 0, // use default DB
})
pong, err := client.Ping().Result()
fmt.Println(pong, err)
if err !=nil {
fmt.Println("err",err);
}
pubsub, err := client.Subscribe("responseclient")
if err !=nil {
panic(err)
}
defer pubsub.Close()
err = client.Publish("new_request", "Example New Request").Err()
if err !=nil {
panic(err)
}
msg, err :=pubsub.ReceiveMessage()
if err != nil {
panic(err)
}
fmt.Println(msg.Channel, msg.Payload)
}
In Project "SOCKET" there is a subscriber that will listen every publish that occured, and publish new message to channel responseclient
this is for the example code :
ioApp.on ('connection' , function(socket) {
redisSub.on('new_request', function (channel, message) {
console.log(channel + ':' + message);
redisPub.publish("responseclient", JSON.stringify(res));
});
})
This work smoothly, if User B is Connected to Socket.IO. But if User B was offline, or not connected to socket.io, this will waiting for long, until we kill manually or until User B is online
What i am asking for , are :
Can we create something like a callback on redis pub/sub ? If the subscriber doesn't accept the message, due to off line, or something else , we close the connection. Is this possible ?
In Node.Js i know i can use timeout function, that will close the subscribe or emit any event if on certain time, there was no message received, how to do this on golang ? I need to inform the User A if User B is active or offline, so he can wait for a another time to create request.
If nothing can, what is your suggestion for me to do this ?
I hope my question , understandable, and can answered well.
*Some code maybe, missing variable.
** I'm using this library for golang redis : go-redis
1) There are no callbacks in Redis.
2) The usual way to implement a timeout in Go is to use channels and select - where one is a channel where you do the blocking and another channel receives a message on timeout. Examples of that can be found here and here for the docs
Now for (3), you have some options on methods. The first is to use a list, pushing from one side (publishing) and popping from another (subscribing). For the receiver you wild use BRPOP of BLPOP - blocking pop from right or left respectively. You can combine the two to have persistent messaging.
Now part of PUBSUB also depends on what you are publishing to. If you are publishing to a channel that would have a subscriber if and only if there is a user connected to receive it (and thus one and only one subscriber to that channel), you can check the response from your publish command. It will tell you how many clients it was published to. If the channel is only subscribed to by an online receiver you would get a '1' back, and a '0' if the user was offline.
A third example is to store the messages in a sorted set, with the timestamp as the score. This would allow the receiver to connect and get messages from the last time it was connected - but that assumes some persistence of that somewhere - usually the client. You would also need some cleanup activity on the sorted sets.
Some other things to consider in this scenario is whether you eventually use replication, in which case you have to explicitly account for failovers - though really in the scenario you describe you'd want to account for disconnects and reconnects. There are specific examples of this at my post on reliable PUBSUB.
package main
import (
"context"
"fmt"
"time"
"github.com/go-redis/redis/v8"
)
var ctx = context.Background()
func main() {
rdb := redis.NewClient(&redis.Options{
Addr: "localhost:6379",
Password: "", // no password set
DB: 0, // use default DB
})
subscribe := rdb.Subscribe(ctx, "hello")
subscriptions := subscribe.ChannelWithSubscriptions(ctx, 1)
go func() {
var sentCount = 0
for {
rdb.Publish(ctx,"hello",time.Now().UnixNano())
sentCount++
if sentCount >300{
break
}
}
}()
for {
select {
case sub:=<-subscriptions:
fmt.Println(sub)
}
}
}
I am using eclipse paho golang library to create new MQTT client for a particular client id:
func CreateMQTTClient(clientID string) (client MQTT.Client) {
username := viper.GetString("messaging.rabbitmq.username")
password := viper.GetString("messaging.rabbitmq.password")
host := viper.GetString("messaging.rabbitmq.host")
mqqtPort := viper.GetString("messaging.rabbitmq.mqqtPort")
rabbitMqMQQTURL := "tcp://" + host + ":" + mqqtPort
opts := MQTT.NewClientOptions().AddBroker(rabbitMqMQQTURL)
opts.SetClientID(clientID)
opts.Username = username
opts.Password = password
opts.SetCleanSession(false)
cli := MQTT.NewClient(opts)
if (!cli.IsConnected()) {
log.Println("I came here for cli:", clientID)
if token := cli.Connect(); token.Wait() && token.Error() != nil {
log.Print(token.Error())
}
}
return cli
}
I am not sure how do I get this client back using clientId. If I call CreateMQTTClient again, all existing subscriptions are lost.
There is, unfortunately, no way to query an MQTT server to find out what subscriptions it has active for your client id. When you connect with the same client ID as a previous session the server assumes you have the same state as last time you were connected, but there is no way to pre connect a MessageHandler with a topic in the Go client, the only way to add and remove them is with Subscribe/Unsubscribe.
Assuming the server is working correctly, if you connect as above reusing a client id the server will continue to send you messages according to your previous subscriptions but the Go client doesn't know what to do with them so will invoke the default message handler. The best way to currently resolve this would be to call Subscribe() in the OnConnectHandler, assuming the topics you want to subscribe to are predetermined rather than dynamic.