How to confirm gRPC traffic from Go client is TLS encrypted - go

I wrote a sample gRPC client a server in Go, both configured for server-authenticated TLS.
The client gRPC call succeeds, giving me the impression the TLS is configured properly, otherwise if the TLS handshake had failed, I would expect the client to fail and not make the gRPC request (i.e. not default to plaintext).
Yet I am puzzled by a result I obtain when I attach Wireshark to that network to sniff TCP packets. I do not see any packet with TLS, for e.g. I do not see the TLS CLIENT HELLO packet.
So is this because I'm misinterpreting what I see in Wireshark, or is my gRPC client actually doing plaintext gRPC?
The client code looks like this, note the grpc.withTransportCredentials which I think means it will use TLS or fail, but never plaintext:
// block the dial until connection is successful or 3 sec timeout
dialOptions := []grpc.DialOption{
grpc.WithBlock(),
grpc.WithTimeout(3 * time.Second),
}
// Load TLS Configuration
tlsCredentials, err := LoadTLSCredentials()
if err != nil {
log.Fatalf("Failed to load TLS credentials: %v", err)
}
dialOptions = append(dialOptions, grpc.WithTransportCredentials(tlsCredentials))
// Dial the gRPC server
log.Printf("Dialing %v", *address)
conn, err := grpc.Dial(*address, dialOptions...)
if err != nil {
log.Fatalf("Failed to connect to the server: %v", err)
}
defer conn.Close()
// then this application sets up a gRPC request, and logs the response to stdout,
// in my testing stdout shows the expected gRPC response, so I'd assume TLS is working.
func LoadTLSCredentials() (credentials.TransportCredentials, error) {
rootCA, err := ioutil.ReadFile("ca.cert")
if err != nil {
return nil, err
}
certPool := x509.NewCertPool()
if !certPool.AppendCertsFromPEM(rootCA) {
return nil, fmt.Errorf("Failed to add rootCA to x509 certificate pool")
}
config := &tls.Config{
MinVersion: tls.VersionTLS12,
RootCAs: certPool,
}
return credentials.NewTLS(config), nil
}
And here's a screenshot of Wireshark showing no TLS packet
whereas I would expect something similar to the following which clearly shows some TLS activity (not my app, image is from the web for illustration purposes)
I'm running Wireshark v2.6.10 on Ubuntu 16.04. The source and destination IPs match my gRPC client and server IPs (both are docker containers on the same docker network).
Not that it really matters, but as can be seen in my client code, I'm sharing a root CA certificate on the client (self signed). I can do this because I deploy both the client and the server.

As #steffanUllrich explained in the comments, this was a case of Wireshark can be better configured to show TLS. I confirmed the gRPC exchange is indeed TLS protected.

You should right click the packet list, and select 'decode as..' menu item, then select 'tls' to force wireshark dissect traffic in this tcp port as TLS.

Related

Is gRPC client an API gateway for the gRPC servers?

My goal is to understand what is gRPC Client is for.
My questions:
Is gRPC client created by both the backend engineer (Go) and another engineer (Java) to communicate with the gRPC services?
i.e. the company hired a new backend engineer and he specialized in Java. He created a new service (i.e. service-garage, written in Java) and also created the garage.proto file. He need the service-user (written in Go), so he do protoc --java_out=. --grpc--java_out=. user.proto. By doing so, he can call the service-user method in Java inside the service-garage.
Is gRPC client an API gateway?
i.e. There are 2 services, service-user (written in Go) and service-garage (written in Java). There should be one backend engineer that write the gRPC client, so the frontend engineer can use this API gateway to send request to.
My current thought:
microservices use multiple ports i.e. 7000 and 9000
i.e. a service-user and a service-garage
services/service-user/main.go on port 7000
func main() {
srv := grpc.NewServer()
var garageSrv GaragesServer
model.RegisterGaragesServer(srv, garageSrv)
log.Println("Starting RPC server at", config.SERVICE_GARAGE_PORT)
l, err := net.Listen("tcp", config.SERVICE_GARAGE_PORT)
if err != nil {
log.Fatalf("could not listen to %s: %v", config.SERVICE_GARAGE_PORT, err)
}
log.Fatal(srv.Serve(l))
}
services/service-garage/main.go on port 9000
func main() {
srv := grpc.NewServer()
var userSrv UsersServer
model.RegisterUsersServer(srv, userSrv)
log.Println("Starting RPC server at", config.SERVICE_USER_PORT)
l, err := net.Listen("tcp", config.SERVICE_USER_PORT)
if err != nil {
log.Fatalf("could not listen to %s: %v", config.SERVICE_USER_PORT, err)
}
log.Fatal(srv.Serve(l))
}
microservices use multiple repositories
repository for gRPC protocol buffers (accessed by gRPC client and gRPC servers)
repositories for services (ie service-uer and service-garage`
repository for gRPC client with port 80 and 443. The frontend web developer or frontend mobile developer can send request to port 80 and 443 ie localhost:80/user/register and localhost:80/user/list
gRPC client is the API endpoint that the frontend web developer or frontend mobile developer send request to.
client-2/main.go
func main() {
mux := http.NewServeMux()
mux.HandleFunc("/user/register", user.Register)
mux.HandleFunc("/user/list", user.List)
mux.HandleFunc("/garage/add", garage.Add)
mux.HandleFunc("/garage/list", garage.List)
err := http.ListenAndServe("localhost:8080", mux)
if err != nil {
log.Panicln(err)
}
}

I can't get `golang.org/x/crypto/acme/autocert` to work (for gRPC), I get an `acme_account+key` file and no X509 cert

UPDATE After much debugging, I uncovered Get "https://acme-v02.api.letsencrypt.org/directory": x509: certificate signed by unknown authority and suspect (!?) this results from the recent expiration of Let's Encrypt's root cert.
I accept that "This package is a work in progress and makes no API stability promises." but, if it no longer works (and it's much more likely that my code|deployment is at issue), then perhaps the repo can be marked e.g. Here be dragons.
The code results in an acme_account+key (EC PRIVATE KEY) but no certs I'm challenged to get autocert to disclose (log) its magic in order to understand where I'm going wrong.
The code is essentially the repo's Manager example with input from this answer. I assume that GetCertificate blocks on the completion of the ACME flow.
Code:
package main
import (
"crypto/tls"
"flag"
"fmt"
"log"
"net"
"net/http"
"golang.org/x/crypto/acme/autocert"
"google.golang.org/grpc"
"google.golang.org/grpc/credentials"
"google.golang.org/grpc/health"
healthpb "google.golang.org/grpc/health/grpc_health_v1"
)
const (
email string = "my#email.com"
var (
host = flag.String("host", "foo.example.org", "Fully-qualified domain name")
port = flag.Uint("port", 443, "gRPC service port")
path = flag.String("path", "", "Folder location for certificate")
)
func main() {
flag.Parse()
if *host == "" {
log.Fatal("Flag --host is required")
}
log.Printf("Host: %s", *host)
log.Printf("Port: %d", *port)
if *path == "" {
log.Fatal("Flag --path is required")
}
log.Printf("Path: %s", *path)
addr := fmt.Sprintf(":%d", *port)
lis, err := net.Listen("tcp", addr)
if err != nil {
log.Fatalf("failed to listen: %s", err)
}
m := &autocert.Manager{
Cache: autocert.DirCache(*path),
Prompt: autocert.AcceptTOS,
HostPolicy: autocert.HostWhitelist(*host),
Email: email,
}
go func() {
log.Println("Starting HTTP server w/ autocert handler")
if err := http.ListenAndServe(":http", m.HTTPHandler(nil)); err != nil {
log.Fatalf("HTTP failure\n%s", err)
}
}()
tlsConfig := &tls.Config{
ClientAuth: tls.RequireAndVerifyClientCert,
GetCertificate: func(hello *tls.ClientHelloInfo) (*tls.Certificate, error) {
cert, err := m.GetCertificate(hello)
if err != nil {
log.Fatalf("GetCertificate\n%s", err)
}
return cert, err
},
}
opts := grpc.Creds(credentials.NewTLS(tlsConfig))
server := grpc.NewServer(opts)
healthcheck := health.NewServer()
healthpb.RegisterHealthServer(server, healthcheck)
log.Println("Starting gRPC server")
if err := server.Serve(lis); err != nil {
log.Fatalf("gRPC failure\n%s", err)
}
}
I'm deploying to a (Google Compute Engine) Container VM, the equivalent docker command is:
docker run \
--name=autocert \
--detach \
--net=host \
--volume=/tmp/certs:/certs \
${IMAGE} \
--host=${HOST} \
--port=${PORT} \
--path=/certs
And container logs:
2021/11/25 17:30:00 Host: [HOST]
2021/11/25 17:30:00 Port: 443
2021/11/25 17:30:00 Path: /certs
2021/11/25 17:30:00 Starting gRPC server
2021/11/25 17:30:00 Starting HTTP server
The host's /tmp/certs directory receives acme_account+key (which I've struggled to find explained by Google) but suspect (!?) is the initial phase of Domain Validation. It contains a private key (BEGIN EC PRIVATE KEY).
Even after some time with the server running, no further files are persisted.
I receive no emails from Let's Encrypt at the configured email address.
Unfortunately, while easy to use, autocert produces little logging and I've been unable to determine whether I can log the ACME flow that's (hopefully) taking place.
Since adding the anonymous function for GetCertificate, acme_account+key is no longer created (I removed the previous file to check whether it's recreated) and so I'm unable to gather any logging from it but the function is never invoked. Either this is because my anonymous function is incorrect or because I've exceeded requests against the ACME endpoint. Removing the function and reverting to m.GetCertificate does not result in recreation of acme_account+key so I'm at a loss.
The autocert Manager type documents an *acme.Client field which I'm not setting. The comment describes "if the Client.Key is nil, a new ECDSA P-256 key is generated" which is perhaps what I'm experiencing but it doesn't explain what I should do about it. Should I set this value to the content of acme_account+key?:
UPDATE I tried decoding the private key, creating a crypto.Signer and passing this in &acme.Client{Key: key} but it made no evident difference
// Client is used to perform low-level operations, such as account registration
// and requesting new certificates.
//
// If Client is nil, a zero-value acme.Client is used with DefaultACMEDirectory
// as the directory endpoint.
// If the Client.Key is nil, a new ECDSA P-256 key is generated and,
// if Cache is not nil, stored in cache.
//
// Mutating the field after the first call of GetCertificate method will have no effect.
Client *acme.Client
Evidently, I'm using this incorrectly. I'm not receiving a cert from Let's Encrypt and so I'm unable to get a cert from the endpoint and unable to invoke the gRPC endpoint:
openssl s_client -showcerts -connect ${HOST}:${PORT}
grpcurl \
-proto health.proto \
${HOST}:${PORT} \
grpc.health.v1.Health/Check
Failed to dial target host "${HOST}:${PORT}": remote error: tls: internal error
Guidance would be appreciated.
🤦‍♂️
Ugh :-(
I'd started using SCRATCH and hadn't copied the CA certificates
Once the container had CA certs, everything worked almost flawlessly.
I continue to have problems trying to use:
tlsConfig := &tls.Config{
ClientAuth: tls.RequireAndVerifyClientCert,
GetCertificate: m.GetCertificate
}
And am using m.TLSConfig()
So, autocert works like a treat (though it's difficult to debug self-inflicted errors 😊)

tls: handshake failure when enabling tls for RabbitMQ with streadway/amqp

I'm attempting to connect to RabbitMQ with amqps:// in Go using streadway/amqp. I can connect successfully with amqp://. When enabling TLS and using amqps:// I get the following error:
panic: remote error: tls: handshake failure
RabbitMQ is running in docker with the following environment variables and settings:
environment:
RABBITMQ_SSL_CACERTFILE: /ca_certificate.pem
RABBITMQ_SSL_CERTFILE: /server_certificate.pem
RABBITMQ_SSL_KEYFILE: /server_key.pem
ports:
- 5671:5671 # Note that 5671 is for tls and 5672 is non-tls
volumes:
- ./ca_certificate.pem:/ca_certificate.pem:ro
- ./server_certificate.pem:/server_certificate.pem:ro
- ./server_key.pem:/server_key.pem:ro
I've tried the following with amqp/streadway:
err := amqp.DialTLS(amqps://guest:guest#localhost:5671", nil)
if err != nil {
panic(err)
}
I've also tried reading the cert files, creating a key pair, and appending the certificate authority to the cert pool and using it that way in a tls.Config{} with the following functions:
tls.LoadX509KeyPair()
x509.NewCertPool().AppendCertsFromPEM()
I generate the certs with mkcert for 127.0.0.1, localhost, rabbitmq.
According to some answers that aren't related to RabbitMQ, some people suggest the ciphers could be wrong. So I took a look at what ciphers rabbitmq is using:
$ openssl s_client -connect localhost:5671 -tls1
Protocol : TLSv1
Cipher : ECDHE-RSA-AES256-SHA
<etc etc...>
Verify return code: 0 (ok)
There are also one or two errors when I run the above command, but I'm guessing it's because I'm not providing the CA certificate in this command (I'm using MacOS). Maybe related, maybe not, as I don't have this issue with postgres, for example:
verify error:num=19:self signed certificate in certificate chain
verify return:0
4644699756:error:1401E410:SSL routines:CONNECT_CR_FINISHED:sslv3 alert handshake failure:/AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libressl-47.100.4/libressl-2.8/ssl/ssl_pkt.c:1200:SSL alert number 40
Then I use the following tls.Config settings in golang:
tlsConfig := &tls.Config{
Certificates: []tls.Certificate{cert}, // from tls.LoadX509KeyPair
RootCAs: caCertPool,
CipherSuites: []uint16{
tls.TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384, // these look like they match the Cipher above
tls.TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA,
},
CurvePreferences: []tls.CurveID{tls.CurveP521, tls.CurveP384, tls.CurveP256},
PreferServerCipherSuites: true,
InsecureSkipVerify: true,
MinVersion: tls.VersionTLS10,
}
I still have the same issue. I highly doubt it's the library, it must be something I'm doing wrong, but what is it?
I reproduced your setup. It doesn't work because you need to configure the AMQP connection with the client certs.
Using mkcert: mkcert -client rabbitmq.test localhost 127.0.0.1 ::1 (note the -client flag).
After this, you just need to pass the client certs into your AMQP tlsConfig with tls.LoadX509KeyPair, and it should just work:
cert, err := tls.LoadX509KeyPair("./rabbitmq.test+3-client.pem", "./rabbitmq.test+3-client-key.pem")
// Load CA cert
caCert, err := ioutil.ReadFile("./rootCA.pem") // The same you configured in your MQ server
if err != nil {
log.Fatal(err)
}
caCertPool := x509.NewCertPool()
caCertPool.AppendCertsFromPEM(caCert)
tlsConfig := &tls.Config{
Certificates: []tls.Certificate{cert}, // from tls.LoadX509KeyPair
RootCAs: caCertPool,
// ...other options are just the same as yours
}
conn, err := amqp.DialTLS("amqps://test:secret#127.0.0.1:5671", tlsConfig)
if err != nil {
panic(err) // does not panic!
}
// ... application code
PS: in my setup I used some different names (user/password/container) than yours, but those should be irrelevant

WebSocket over TLS: Golang / Gorilla

I have been trying to set up a WebSocket connection over TLS (so with encryption). I use Golang with Gorilla. A WebSocket connection is implemented as an initial HTTP connection that gets upgraded to the WebSocket protocol connection. The code is like this:
func wsEndpoint(w http.ResponseWriter, r *http.Request) {
// upgrade
ws, err := upgrader.Upgrade(w, r, nil)
if err != nil {
//...
}
log.Println("Client Connected")
err = ws.WriteMessage(1, []byte("Hi Client!"))
if err != nil {
//...
}
// listen indefinitely for new messages coming
}
Then we set up the routing:
func main() {
//...
setupRoutes()
log.Fatal(http.ListenAndServe(":8080", nil))
}
Does it suffice to change the last line to:
...http.ListenAndServeTLS(...)
so in other words to use TLS to establish the first connection?
Does this approach suffice to secure the entire communication over WebSocket from start till the end? Should I be certain that all packet transmission within the connection duration is also protected by TLS? If not, how to set it up in Golang / Gorilla framework?
Use http.ListenAndServeTLS to encrypt the underlying network connections used for the HTTP protocol and the WebSocket protocol.
The approach secures the entire communication on the underlying network connection including all WebSocket traffic.
The Gorilla server code uses the network connection provided by the net/http server. The Gorilla server code does create new network connections.
http.ListenAndServeTLS is a helper function that calls lower-level functions and methods. It also works to call those lower-level functions and methods directly.
One approach to this problem would be to set up a reverse proxy like nginx with certbot to generate certificates.
Here's how it would work
[ Client ] ----------> [ nginx ] --------------> [ golang server ]
Encrypted Not encrypted

Getting Unimplemented desc = unknown service error gRPC

In one of my services that happens to be my loadbalancer, I am getting the following error when calling the server method in one of my deployed services:
rpc error: code = Unimplemented desc = unknown service
fooService.FooService
I have a few other services set up with gRPC and they work fine. It just seems to be this one and I am wondering if that is because it is the loadbalancer?
func GetResult(w http.ResponseWriter, r *http.Request) {
conn, errOne := grpc.Dial("redis-gateway:10006", grpc.WithInsecure())
defer conn.Close()
rClient := rs.NewRedisGatewayClient(conn)
result , errTwo := rClient.GetData(context.Background(), &rs.KeyRequest{Key: "trump", Value: "trumpVal"}, grpc.WithInsecure())
fmt.Fprintf(w, "print result: %s \n", result) //prints nil
fmt.Fprintf(w, "print error one: %v \n", errOne) // prints nil
fmt.Fprintf(w, "print error two: %s \n", errTwo) // prints error
}
The error says there is no service called fooService.FooService which is true because the dns name for the service I am calling is called foo-service. However it is the exact same setup for my other services that use gRPC and work fine. Also my proto files are correctly configured so that is not an issue.
server I am calling:
func main() {
lis, err := net.Listen("tcp", ":10006")
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
grpcServer := grpc.NewServer()
newServer := &RedisGatewayServer{}
rs.RegisterRedisGatewayServer(grpcServer, newServer)
if err := grpcServer.Serve(lis); err != nil {
log.Fatalf("failed to serve: %v", err)
}
}
The function I am trying to access from client:
func (s *RedisGatewayServer) GetData(ctx context.Context, in *rs.KeyRequest)(*rs.KeyRequest, error) {
return in, nil
}
My docker and yaml files are all correct also with the right naming and ports.
I had this exact problem, and it was because of a very simple mistake: I had put the call to the service registration after the server start. The code looked like this:
err = s.Serve(listener)
if err != nil {
log.Fatalf("[x] serve: %v", err)
}
primepb.RegisterPrimeServiceServer(s, &server{})
Obviously, the registration should have been called before the server was ran:
primepb.RegisterPrimeServiceServer(s, &server{})
err = s.Serve(listener)
if err != nil {
log.Fatalf("[x] serve: %v", err)
}
thanks #dolan's for the comment, it solved the problem.
Basically we have to make sure, the method value should be same in both server and client (you can even copy the method name from the pb.go file generated from server side)
func (cc *ClientConn) Invoke(ctx context.Context, method string, args, reply interface{}, opts ...CallOption) error {
this invoke function will be there inside all the methods which you have implemented in gRPC service.
for me, my client was connecting to the wrong server port.
My server listens to localhost:50052
My client connects to localhost:50051
and hence I see this error.
I had the same problem - I was able to perform rpc call from Postman to the service whereas apigateway was able to connect to the service but on method call It gave error code 12 unknown service and the reason was in my proto files client was under package X whereas server was under package Y.
Quite silly but yeah making the package for both proto under apigateway and service solved my problem.
There are many scenarios in which this can happen. The underlying issue seems to be the same - the GRPC server is reachable but cannot service client requests.
The two scenarios I faced that are not documented in previous answers are:
1. Client and server are not running the same contract version
A breaking change introduced in the contract can cause this error.
In my case, the server was running the older version of the contract while the client was running the latest one.
A breaking change meant that the server could not resolve the service my client was asking for thus returning the unimplemented error.
2. The client is connecting to the wrong GRPC server
The client reached the incorrect server that doesn't implement the contract.
Consider this scenario if you're running multiple different GRPC services. You might be mistakingly dialing the wrong one.

Resources