https load balancer with fiber is not working - go

I am trying to make simple load balancer using fiber in Go. In my computer it's working fine using http.
import (
"crypto/tls"
"log"
"os"
store "intraGo/stores/session_store"
"github.com/gofiber/fiber/v2"
"github.com/gofiber/fiber/v2/middleware/proxy"
"github.com/gofiber/template/html"
)
...
if os.Getenv("CERT_FILE") == "" || os.Getenv("KEY_FILE") == "" {
// this part is working fine
app.Use(proxy.Balancer(proxy.Config{Servers: []string{"http://localhost:9000"}}))
log.Fatal(app.Listen(":" + os.Getenv("SERVER_PORT")))
} else {
app.Use(proxy.Balancer(proxy.Config{Servers: []string{"https://servername.com"}}))
cer, err := tls.LoadX509KeyPair(os.Getenv("CERT_FILE"), os.Getenv("KEY_FILE"))
if err != nil {
log.Fatal(err)
}
config := &tls.Config{Certificates: []tls.Certificate{cer}}
ln, err := tls.Listen("tcp", ":"+os.Getenv("SERVER_PORT"), config)
if err != nil {
panic(err)
}
log.Fatal(app.Listener(ln))
}
In the production envirment using https it will give me a http 500 result with the follwing text for every request. The request will never reach the server.
HostClient can't follow redirects to a different protocol, please use Client instead
I checked Go / fiber documentation but did not find anything that could help to solve this issue. I also checked if the https part (like certificates, https server) is working fine by adding this to my app:
app.get("/", (req, res) => { res.send("Hello World!"); });
This worked as expected, so I guess the problem is in the loadbalancer itself.
Found this link while trying to find the solution on the net, I am not sure, but this might be related to the problem:
https://github.com/valyala/fasthttp/issues/841

Related

Why Google Logging client libraries not logging inside Google cloud functions?

I'm trying to implement a google cloud function to test Google Logging client library. below is my code
// Package p contains an HTTP Cloud Function.
package loggingclient
import (
"cloud.google.com/go/logging"
"net/http"
"context"
"fmt"
)
// HelloWorld prints the JSON encoded "message" field in the body
// of the request or "Hello, World!" if there isn't one.
func HelloWorld(w http.ResponseWriter, r *http.Request) {
label := map[string]string{"priority": "High"}
var projectName = "my-project-id"
ctx := context.Background()
client, err := logging.NewClient(ctx, projectName)
if err != nil {
fmt.Printf("client not created: %v", err)
}
lg := client.Logger("MY-LOGGER")
lg.Log(logging.Entry{
Payload: "Hello, This is error!!",
Severity: logging.Error,
Labels: label,
})
client.Close()
}
Here, I'm expecting a log entry with a message:"Hello, This is error!!" and with a lable:"priority": "High" and severirty "ERROR"
But actually, when I trigger this Cloud Function, I didn't get any new log entries. Therefore don't client logging libraries work inside cloud functions?, How to resolve this?
Thanks
It works on cloud functions. I have done the exact same thing in a cloud function before. You can use google's official documenation with cloud function logging here
Also ensure that the service account have necessary permissions for logging
https://cloud.google.com/logging/docs/access-control

Creating V4 Signed URLs in CloudRun

I'd like to create Signed URLs to Google Cloud Storage resources from an app deployed using CloudRun.
I set up CloudRun with a custom Service Account with the GCS role following this guide.
My intent was to use V4 Signing to create Signed URLs from CloudRun. There is a guide for this use-case where a file service_account.json is used to generate JWT config. This works for me on localhost when I download the file from google's IAM. I'd like to avoid having this file committed in the repository use the one that I provided in CloudRun UI.
I was hoping that CloudRun injects this service account file to the app container and makes it accessible in GOOGLE_APPLICATION_CREDENTIALS variable but that's not the case.
Do you have a recommendation on how to do this? Thank you.
As you say, Golang Storage Client Libraries require a service account json file to sign urls.
There is currently a feature request open in GitHub for this but you should be able to work this around with this sample that I found here:
import (
"context"
"fmt"
"time"
"cloud.google.com/go/storage"
"cloud.google.com/go/iam/credentials/apiv1"
credentialspb "google.golang.org/genproto/googleapis/iam/credentials/v1"
)
const (
bucketName = "bucket-name"
objectName = "object"
serviceAccount = "[PROJECTNUMBER]-compute#developer.gserviceaccount.com"
)
func main() {
ctx := context.Background()
c, err := credentials.NewIamCredentialsClient(ctx)
if err != nil {
panic(err)
}
opts := &storage.SignedURLOptions{
Method: "GET",
GoogleAccessID: serviceAccount,
SignBytes: func(b []byte) ([]byte, error) {
req := &credentialspb.SignBlobRequest{
Payload: b,
Name: serviceAccount,
}
resp, err := c.SignBlob(ctx, req)
if err != nil {
panic(err)
}
return resp.SignedBlob, err
},
Expires: time.Now().Add(15*time.Minute),
}
u, err := storage.SignedURL(bucketName, objectName, opts)
if err != nil {
panic(err)
}
fmt.Printf("\"%v\"", u)
}
Cloud Run (and other compute platforms) does not inject a service account key file. Instead, they make access_tokens available on the instance metadata service. You can then exchange this access token with a JWT.
However, often times, Google’s client libraries and gcloud works out of the box on GCP’s compute platforms without explicitly needing to authenticate. So if you use the instructions on the page you linked (gcloud or code samples) it should be working out-of-the-box.

Getting Unimplemented desc = unknown service error gRPC

In one of my services that happens to be my loadbalancer, I am getting the following error when calling the server method in one of my deployed services:
rpc error: code = Unimplemented desc = unknown service
fooService.FooService
I have a few other services set up with gRPC and they work fine. It just seems to be this one and I am wondering if that is because it is the loadbalancer?
func GetResult(w http.ResponseWriter, r *http.Request) {
conn, errOne := grpc.Dial("redis-gateway:10006", grpc.WithInsecure())
defer conn.Close()
rClient := rs.NewRedisGatewayClient(conn)
result , errTwo := rClient.GetData(context.Background(), &rs.KeyRequest{Key: "trump", Value: "trumpVal"}, grpc.WithInsecure())
fmt.Fprintf(w, "print result: %s \n", result) //prints nil
fmt.Fprintf(w, "print error one: %v \n", errOne) // prints nil
fmt.Fprintf(w, "print error two: %s \n", errTwo) // prints error
}
The error says there is no service called fooService.FooService which is true because the dns name for the service I am calling is called foo-service. However it is the exact same setup for my other services that use gRPC and work fine. Also my proto files are correctly configured so that is not an issue.
server I am calling:
func main() {
lis, err := net.Listen("tcp", ":10006")
if err != nil {
log.Fatalf("failed to listen: %v", err)
}
grpcServer := grpc.NewServer()
newServer := &RedisGatewayServer{}
rs.RegisterRedisGatewayServer(grpcServer, newServer)
if err := grpcServer.Serve(lis); err != nil {
log.Fatalf("failed to serve: %v", err)
}
}
The function I am trying to access from client:
func (s *RedisGatewayServer) GetData(ctx context.Context, in *rs.KeyRequest)(*rs.KeyRequest, error) {
return in, nil
}
My docker and yaml files are all correct also with the right naming and ports.
I had this exact problem, and it was because of a very simple mistake: I had put the call to the service registration after the server start. The code looked like this:
err = s.Serve(listener)
if err != nil {
log.Fatalf("[x] serve: %v", err)
}
primepb.RegisterPrimeServiceServer(s, &server{})
Obviously, the registration should have been called before the server was ran:
primepb.RegisterPrimeServiceServer(s, &server{})
err = s.Serve(listener)
if err != nil {
log.Fatalf("[x] serve: %v", err)
}
thanks #dolan's for the comment, it solved the problem.
Basically we have to make sure, the method value should be same in both server and client (you can even copy the method name from the pb.go file generated from server side)
func (cc *ClientConn) Invoke(ctx context.Context, method string, args, reply interface{}, opts ...CallOption) error {
this invoke function will be there inside all the methods which you have implemented in gRPC service.
for me, my client was connecting to the wrong server port.
My server listens to localhost:50052
My client connects to localhost:50051
and hence I see this error.
I had the same problem - I was able to perform rpc call from Postman to the service whereas apigateway was able to connect to the service but on method call It gave error code 12 unknown service and the reason was in my proto files client was under package X whereas server was under package Y.
Quite silly but yeah making the package for both proto under apigateway and service solved my problem.
There are many scenarios in which this can happen. The underlying issue seems to be the same - the GRPC server is reachable but cannot service client requests.
The two scenarios I faced that are not documented in previous answers are:
1. Client and server are not running the same contract version
A breaking change introduced in the contract can cause this error.
In my case, the server was running the older version of the contract while the client was running the latest one.
A breaking change meant that the server could not resolve the service my client was asking for thus returning the unimplemented error.
2. The client is connecting to the wrong GRPC server
The client reached the incorrect server that doesn't implement the contract.
Consider this scenario if you're running multiple different GRPC services. You might be mistakingly dialing the wrong one.

How to Make My Web Server written in Golang to support HTTP/2 Server Push?

My Web Server is Coded in Golang and supports HTTPS. I wish to leverage HTTP/2 Server Push features in the Web Server. The following Link explains how to convert HTTP Server to Support HTTP/2 :-
https://www.ianlewis.org/en/http2-and-go
However, it is not clear how to implement the Server Push notifications in Golang.
- How should I add the Server Push functionality ?
- How do I control, or manage, the documents and files to be Pushed ?
Go 1.7 and older do not support HTTP/2 server push in the standard library. Support for server push will be added in the upcoming 1.8 release (see the release notes, expected release is February).
With Go 1.8 you can use the new http.Pusher interface, which is implemented by net/http's default ResponseWriter. Pushers Push method returns ErrNotSupported, if server push is not supported (HTTP/1) or not allowed (the client has disabled server push).
Example:
package main
import (
"io"
"log"
"net/http"
)
func main() {
http.HandleFunc("/pushed", func(w http.ResponseWriter, r *http.Request) {
io.WriteString(w, "hello server push")
})
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
if pusher, ok := w.(http.Pusher); ok {
if err := pusher.Push("/pushed", nil); err != nil {
log.Println("push failed")
}
}
io.WriteString(w, "hello world")
})
http.ListenAndServeTLS(":443", "server.crt", "server.key", nil)
}
If you want to use server push with Go 1.7 or older use can use the golang.org/x/net/http2 and write the frames directly.
As mentioned in other answers, you can make use of Go 1.8 feature (cast the writer to http.Pusher and then use the Push method).
That comes with a caveat: you must be serving the HTTP2 traffic right from your server.
If you're behind a proxy like NGINX, this might not work. If you want to consider that scenario, you can make use of the Link header to advertise the URLs to be pushed.
// In the case of HTTP1.1 we make use of the `Link` header
// to indicate that the client (in our case, NGINX) should
// retrieve a certain URL.
//
// See more at https://www.w3.org/TR/preload/#server-push-http-2.
func handleIndex(w http.ResponseWriter, r *http.Request) {
var err error
if *http2 {
pusher, ok := w.(http.Pusher)
if ok {
must(pusher.Push("/image.svg", nil))
}
} else {
// This ends up taking the effect of a server push
// when interacting directly with NGINX.
w.Header().Add("Link",
"</image.svg>; rel=preload; as=image")
}
w.Header().Add("Content-Type", "text/html")
_, err = w.Write(assets.Index)
must(err)
}
ps.: I wrote more about this here https://ops.tips/blog/nginx-http2-server-push/ if you're interested.

Selectively Follow Redirects in Go

I'm trying to write a twitter reader that resolves the final URLs of link shorteners etc, but gives me a URL along the way for a list of manually defined host patterns. The reason to do this is that i don't want to end up with the paywall URL but the one before.
As far as i can tell the way to do this is write my own client based on the default RoundTripper because returning an error from a custom CheckRedirect function aborts the client without yielding a response.
Is there a way to use the default client and record a list of URLs/specific URL from a custom checkRedirect function?
The client request will actually still return the last valid Response in cases where your custom CheckResponse yields an error (As mentioned in the comments).
http://golang.org/pkg/net/http/#Client
If CheckRedirect returns an error, the Client's Get method returns both the previous Response and CheckRedirect's error (wrapped in a url.Error) instead of issuing the Request req.
If you maintain a list of "known" paywall-urls, you can abort the paywall-redirect in your CheckResponse with a custom error type (Paywalled in the example below).
Your error handling code later has to consider that error type as a special (non-erroneous) case.
Example:
package main
import (
"errors"
"fmt"
"net/http"
"net/url"
)
var Paywalled = errors.New("next redirect would hit a paywall")
var badHosts = map[string]error{
"registration.ft.com": Paywalled,
}
var client = &http.Client{
CheckRedirect: func(req *http.Request, via []*http.Request) error {
// N.B.: when used in production, also check for redirect loops
return badHosts[req.URL.Host]
},
}
func main() {
resp, err := client.Get("http://on.ft.com/14pQBYE")
// ignore non-nil err if it's a `Paywalled` wrapped in url.Error
if e, ok := err.(*url.Error); (ok && e.Err != Paywalled) || (!ok && err != nil) {
fmt.Println("error: ", err)
return
}
resp.Body.Close()
fmt.Println(resp.Request.URL)
}

Resources