I have a gRPC server exposed by an ingress.
Here are the nginx annotations I use :
nginx.ingress.kubernetes.io/ssl-redirect: "true"
nginx.ingress.kubernetes.io/backend-protocol: "GRPC"
I am able to request it using grpcurl :
grpcurl -d '<arg>:<value>' <address>:443 api.Service/Request
But when I use a basic golang program :
conn,_ := grpcconn.Dial("<address>:443",grpccon.WithInsecure())
api := apiService.NewServiceClient(conn)
res,err := api.Request(<parameters>)
if err!=nil {
log.Fatalf("Request failed : %v",err)
}
log.Println(res)
I get the error :
rpc error : code = Unavailable desc = connection closed
Related
I'm trying to access my ETCD database from a K8s controller, but getting rpc error/EOF when trying to open ETCD client.
My setup:
ETCD service is deployed in my K8s cluster and included in my Istio service mesh (its DNS record: my-etcd-cluster.my-etcd-namespace.svc.cluster.local)
I have a custom K8s controller developed with use of Kubebuilder framework and deployed in the same cluster, different namespace, but configured to be a part of the same Istio service mesh
I'm trying to connect to ETCD database from the controller, using Go client SDK library for ETCD
Here's my affected Go code:
cli, err := clientv3.New(clientv3.Config{
Endpoints: []string{"http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379"},
DialTimeout: 5 * time.Second,
Username: username,
Password: password,
})
if err != nil {
return nil, fmt.Errorf("opening ETCD client failed: %v", err)
}
And here's an error I'm getting when clientv3.New(...) gets executed:
{"level":"warn","ts":"2022-03-16T23:37:42.174Z","logger":"etcd-client","caller":"v3#v3.5.0/retry_interceptor.go:62","msg":"retrying of unary invoker failed",
"target":"etcd-endpoints://0xc00057f500/#initially=[http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379]","attempt":0,
"error":"rpc error: code = Unavailable desc = error reading from server: EOF"}
...
1.647473862175209e+09 INFO controller.etcdclient Finish reconcile loop for some-service/test-svc-client {"reconciler group": "my-controller.something.io", "reconciler kind": "ETCDClient", "name": "test-svc-client", "namespace": "some-service", "reconcile-etcd-client": "some-service/test-svc-client"}
1.6474738621752858e+09 ERROR controller.etcdclient Reconciler error {"reconciler group": "my-controller.something.io", "reconciler kind": "ETCDClient", "name": "test-svc-client", "namespace": "some-service", "error": "opening ETCD client failed: rpc error: code = Unavailable desc = error reading from server: EOF"}
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
/go/pkg/mod/sigs.k8s.io/controller-runtime#v0.11.0/pkg/internal/controller/controller.go:266
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
/go/pkg/mod/sigs.k8s.io/controller-runtime#v0.11.0/pkg/internal/controller/controller.go:227
The same error happens when I'm passing some dummy, invalid credentials.
However, when I tried to access the database in a HTTP API manner:
postBody, _ := json.Marshal(map[string]string{
"name": username,
"password": password,
})
responseBody := bytes.NewBuffer(postBody)
resp, err := http.Post("http://my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379/v3/auth/authenticate", "application/json", responseBody)
if err != nil {
return ctrl.Result{}, fmt.Errorf("an error occured %w", err)
}
l.Info(fmt.Sprintf("code: %d", resp.StatusCode))
defer resp.Body.Close()
...I got 200 OK and a proper token (which is expected), so I believe my Istio configuration is ok and my controller should be able to see the ETCD db service. I have no clue why this doesn't work when following the client SDK approach.
When I'm using port-forwarding of the ETCD service and accessing it locally, clientv3.New() and other client SDK methods work like a charm. What am I missing? I'd really appreciate any suggestions.
EDIT:
I've also added a simple pod to try accessing my etcd db via etcdctl:
apiVersion: v1
kind: Pod
metadata:
name: test-pod
namespace: my-controller-namespace
spec:
containers:
- name: etcdctl
image: bitnami/etcd
command:
- sleep
- infinity
When logged into the container via kubectl exec, I was able to access my db:
$ etcdctl --endpoints=my-etcd-cluster.my-etcd-namespace.svc.cluster.local:2379 --user="user" --password="password" put foo bob
OK
I guess the problem is somewhere in the SDK?
Turned out to be version mismatch - my ETCD db is v3.5.2 and the clientv3 library that I used was v3.5.0.
As seen in ETCD changelog (https://github.com/etcd-io/etcd/blob/main/CHANGELOG/CHANGELOG-3.5.md):
I'm using a Go API and I need to connect it to my Cassandra DB on a Google Cloud server.
This is my Go connection code:
func SetupDBConnection() {
cluster := gocql.NewCluster("XX.XXX.XXX.XXX")
cluster.Keyspace = "afterstr"
cluster.Consistency = gocql.Quorum
s, err := cluster.CreateSession()
if err != nil {
fmt.Println(err)
}
}
But, when I run this code, I have this error:
gocql: unable to create session: control: unable to connect to initial hosts: dial tcp xx.xxx.xxx.xxx:9042: i/o timeout
This is my Google Cloud configuration:
I also modified the cassandra.yaml file in my server to change the listen address and the RPC address to put my IP with the port I use to run my Go program that I opened.
(It is opened 3 times because I was just testing something)
Try:
cluster.Authenticator = gocql.PasswordAuthenticator{Username: username, Password: password}
I have created a basic service with grpc-gateway. This is what my .proto file looks like
syntax = "proto3";
package health;
option go_package = "github.com/userl/brw/api/health";
import "google/api/annotations.proto";
service Health {
// Sends a greeting
rpc Ping (HealthRequest) returns (HealthReply) {
option (google.api.http) = {
get: "/ping"
};
}
}
// The request message containing the user's name
message HealthRequest {
}
// The response message containing the greetings
message HealthReply {
string message = 1;
}
I generate and start the service with the following code
mux := runtime.NewServeMux()
opts := []grpc.DialOption{grpc.WithInsecure()}
err := pHealth.RegisterHealthHandlerFromEndpoint(ctx, mux, ":3002"), opts)
if err != nil {
log.Fatal(err)
}
http.ListenAndServe(":3001"), mux)
Now when I try to curl command for example on
curl http://localhost:3001/ping I get the following error
curl: (7) Failed to connect to localhost port 3003: Connection refused
When I tried grpcurl
grpcurl --plaintext localhost:3002 list I get the following error
failed to dial target host "localhost:3002": dial tcp 127.0.0.1:3002: connect: connection refused
Not sure what is going wrong here or what I need to do.
I am trying to implement peer discovery logic through DNS in go using SRV records in cluster. I have headless service and statefulset pods ready and I am able to list all SRV records by using
kubectl run -it srvlookup --image=tutum/dnsutils --rm --restart=Never -- dig SRV demoapp.default.svc.cluster.local
but the following code does not work in cluster:
func pingdns() (url string) {
log.Println("start ping demoapp.default.svc.cluster.local.")
_, addrs, err := net.LookupSRV("dns-tcp", "tcp", "demoapp.default.svc.cluster.local")
if err != nil {
log.Println(err.Error())
return "dns wrong"
}
fmt.Println(addrs)
return "dns done."
}
error output:
lookup _dns-tcp._tcp.demoapp.default.svc.cluster.local on 10.96.0.10:53: no such host
I found example in this k8s-in-action book but it is written in NodeJS. How to do it in Golang ?
const dns = require('dns');
const dataFile = "/var/data/kubia.txt";
const serviceName = "kubia.default.svc.cluster.local";
const port = 8080;
...
var handler = function(request, response) {
if (request.method == 'POST') {
...
} else {
response.writeHead(200);
if (request.url == '/data') {
var data = fileExists(dataFile)
? fs.readFileSync(dataFile, 'utf8')
: "No data posted yet";
response.end(data);
} else {
response.write("You've hit " + os.hostname() + "\n");
response.write("Data stored in the cluster:\n");
dns.resolveSrv(serviceName, function (err, addresses) {
The app performs a DNS lookup to obtain SRV records.
if (err) {
response.end("Could not look up DNS SRV records: " + err);
return;
}
var numResponses = 0;
if (addresses.length == 0) {
response.end("No peers discovered.");
} else {
addresses.forEach(function (item) { ...
Thanks Shubham ! I read the medium post and found using grpc to make connection to SRV should browse all IPs as a RR fashion. But I am still looking for get all IPs.
Medium article:
https://medium.com/google-cloud/loadbalancing-grpc-for-kubernetes-cluster-services-3ba9a8d8fc03
Gitrepo:
https://github.com/jtattermusch/grpc-loadbalancing-kubernetes-examples#example-1-round-robin-loadbalancing-with-grpcs-built-in-loadbalancing-policy
import (
"google.golang.org/grpc/balancer/roundrobin"
"google.golang.org/grpc/credentials"
)
conn, err := grpc.Dial("dns:///be-srv-lb.default.svc.cluster.local", grpc.WithTransportCredentials(ce), grpc.WithBalancerName(roundrobin.Name))
c := echo.NewEchoServerClient(conn)
It makes calls on a list of IPs one by one. RR
Creating channel with target greeter-server.default.svc.cluster.local:8000
Greeting: Hello you (Backend IP: 10.0.2.95)
Greeting: Hello you (Backend IP: 10.0.0.74)
Greeting: Hello you (Backend IP: 10.0.1.51)
I have found my main issue is related to this problem.
Why golang Lookup*** function can't provide a server parameter?
It looks like you are looking for headless service.
Look for an example implementation: https://medium.com/google-cloud/loadbalancing-grpc-for-kubernetes-cluster-services-3ba9a8d8fc03
Kubernetes documentation: https://kubernetes.io/docs/concepts/services-networking/service/#headless-services for your reference.
To explain this: Kubernetes has a few implementations for services basically Endpoint based service and Headless service. Endpoint based services could be either with or without selectors. LoadBalancers are generally provided by Cloud providers.
Headless service are designed for client-side load balancing implementation. It looks like you are trying to implement your own DNS driven client-side load balancer.
I'm trying to send a custom Proto in Error Response of my grpc calls. As per this, it should be possible.
Here is the code.
st := status.Newf(codes.NotFound, "user %s doesn't exist", req.Name)
desc := "The username Doesn't exist, please give a valid username"
customProtoError := &data.Error{
Message: "Check username",
Code: 1404,
Type: "Service",
DetailedMessage: desc,
}
st, err := st.WithDetails(customProtoError)
if err != nil {
panic(fmt.Sprintf("Unexpected error attaching metadata: %v", err))
}
I've both grpc & http Handlers for these GRPC APIs.
The Marshaler used in HTTP server is github.com/gogo/gateway
mux := runtime.NewServeMux(
runtime.WithMarshalerOption(JSONContentType, &gateway.JSONPb{}))
The configuration for HTTP and GRPC server is available here.
When I try to access the API using HTTP calls, here is what I get in response.
HTTP Error:
{
"error": "user kishore1 doesn't exist",
"code": 5,
"message": "user kishore1 doesn't exist",
"details": [
{
"type_url": "type.googleapis.com/data.Error",
"value": "Cg5DaGVjayB1c2VybmFtZRD8ChoHU2t5ZmxvdyI4VGhlIHVzZXJuYW1lIERvZXNuJ3QgZXhpc3QsIHBsZWFzZSBnaXZlIGEgdmFsaWQgdXNlcm5hbWU="
}
]
}
The type_url is coming with google, as its hardcoded in the golang proto code. But this says Any in JSON will be deserialized to embedded messages.
Grpc Client Code:
resp, err := client.GetUser(ctx, req)
if err != nil {
st := status.Convert(err)
for _, detail := range st.Details() {
switch t := detail.(type) {
case *pb.Error:
fmt.Println("Oops! Your request was rejected by the server.")
...
default:
fmt.Println("Error Received - ", detail, reflect.TypeOf(detail))
}
}
Grpc Error:
Error Received - proto: not found *errors.prefixError
Any idea what could be causing this mess? HTTP doesn't deserialize as an embedded message and GRPC gives proto.NotFound
Could this be due to gogo proto being the gateway? I even tried the same with googleapis proto errdetails.BadRequest but still the same.
After spending about a day, I just figured out the issue was due to incompatible versions.
So here is what worked for me, when using gogo proto.
For the GRPC errors, I had to upgrade my grpc-gateway version and use gogo/status instead of grpc/status
github.com/gogo/status v1.1.0
github.com/grpc-ecosystem/grpc-gateway v1.14.5
For HTTP errors, I had to downgrade my golang/protobuf
github.com/golang/protobuf v1.3.2