I need to write a simple web server in Go. It accepts requests, maps the request to avro object and sends it to Kafka. The requirement is that it answers immediately to keep the latency low for the users. Mapping to avro object and sending to Kafka can happen asynchronously. I came up with the following design, but I wonder if it is using Go structures in the intended way or if it can be optimized using channels for example. I'm omitting private methods and initializing structures. The problem is that the server can handle up to 10500 requests a second and then the response time goes up dramatically. So I was wondering if there is a way to optimize it.
func main() {
runtime.GOMAXPROCS(runtime.NumCPU()) // not needed in Go 1.5.0
server := &Server{
Producer: newProducer(brokerList),
}
defer func() {
if err := server.Close(); err != nil {
Error.Println("Failed to close server", err)
}
}()
Error.Fatal(server.Run(*addr))
}
func (s *Server) Run(addr string) error {
httpServer := &http.Server{
Addr: addr,
Handler: s.Handler(),
}
return httpServer.ListenAndServe()
}
func (s *Server) Handler() http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
defer r.Body.Close()
req, err := ParseRequest(r.Body)
if err != nil {
Warning.Println("Failed to parse request", err.Error())
} else {
go handleRequest(s, req)
}
w.WriteHeader(204) // respond with 'no bid'
)
}
func handleRequest(s *Server, req *openrtb.BidRequest) {
req.Validate()
var avroObject, err = createAvro(req)
if err != nil {
Warning.Printf(err.Error())
}
if avroObject != nil {
sendToKafka(avroObject, s)
}
}
}
Related
I have a function in go which is handling connections which are coming through tcp and handled via ssh. I am trying to set an idle timeout by creating struct in the connection function.
Use case - a customer should be able to make a connection and upload/download multiple files
Reference - IdleTimeout in tcp server
Function code:
type Conn struct {
net.Conn
idleTimeout time.Duration
}
func HandleConn(conn net.Conn) {
var err error
rAddr := conn.RemoteAddr()
session := shortuuid.New()
config := LoadSSHServerConfig(session)
blocklistItem := blocklist.GetBlockListItem(rAddr)
if blocklistItem.IsBlocked() {
conn.Close()
atomic.AddInt64(&stats.Stats.BlockedConnections, 1)
return
}
func (c *Conn) Read(b []byte) (int, error) {
err := c.Conn.SetReadDeadline(time.Now().Add(c.idleTimeout))
if err != nil {
return 0, err
}
return c.Conn.Read(b)
}
sConn, chans, reqs, err := ssh.NewServerConn(conn, config)
if err != nil {
if err == io.EOF {
log.Errorw("SSH: Handshaking was terminated", log.Fields{
"address": rAddr,
"error": err,
"session": session})
} else {
log.Errorw("SSH: Error on handshaking", log.Fields{
"address": rAddr,
"error": err,
"session": session})
}
atomic.AddInt64(&stats.Stats.AuthorizationFailed, 1)
return
}
log.Infow("connection accepted", log.Fields{
"user": sConn.User(),
})
if user, ok := users[session]; ok {
log.Infow("SSH: Connection accepted", log.Fields{
"user": user.LogFields(),
"clientVersion": string(sConn.ClientVersion())})
atomic.AddInt64(&stats.Stats.AuthorizationSucceeded, 1)
// The incoming Request channel must be serviced.
go ssh.DiscardRequests(reqs)
// Key ID: sConn.Permissions.Extensions["key-id"]
handleServerConn(user, chans)
log.Infow("connection finished", log.Fields{"user": user.LogFields()})
log.Infow("checking connections", log.Fields{
//"cc": Stats.AcceptedConnections,
"cc2": &stats.Stats.AcceptedConnections})
// Remove connection from local cache
delete(users, session)
} else {
log.Infow("user not found from memory", log.Fields{"username": sConn.User()})
}
}
This code is coming from the Listen function:
func Listen() {
listener, err := net.Listen("tcp", sshListen)
if err != nil {
panic(err)
}
if useProxyProtocol {
listener = &proxyproto.Listener{
Listener: listener,
ProxyHeaderTimeout: time.Second * 10,
}
}
for {
// Once a ServerConfig has been configured, connections can be accepted.
conn, err := listener.Accept()
if err != nil {
log.Errorw("SSH: Error accepting incoming connection", log.Fields{"error": err})
atomic.AddInt64(&stats.Stats.FailedConnections, 1)
continue
}
// Before use, a handshake must be performed on the incoming net.Conn.
// It must be handled in a separate goroutine,
// otherwise one user could easily block entire loop.
// For example, user could be asked to trust server key fingerprint and hangs.
go HandleConn(conn)
}
}
Is that even possible to set a deadline for only the connections which have been idle for 20 secinds (no upload/downloads).
EDIT 1 : Following #LiamKelly's suggestions, I have made the changes in the code. Now the code is like
type SshProxyConn struct {
net.Conn
idleTimeout time.Duration
}
func (c *SshProxyConn) Read(b []byte) (int, error) {
err := c.Conn.SetReadDeadline(time.Now().Add(c.idleTimeout))
if err != nil {
return 0, err
}
return c.Conn.Read(b)
}
func HandleConn(conn net.Conn) {
//lines of code as above
sshproxyconn := &SshProxyConn{nil, time.Second * 20}
Conn, chans, reqs, err := ssh.NewServerConn(sshproxyconn, config)
//lines of code
}
But now the issue is that SSH is not happening. I am getting the error "Connection closed" when I try to do ssh. Is it still waiting for "conn" variable in the function call?
Is that even possible to set a deadline for only the connections which have been idle for 20 [seconds]
Ok so first a general disclaimer, I am going to assume go-protoproxy implements the Conn interface as we would expected. Also as you hinted at before, I don't think you can put a a struct method inside another function (I also recommend renaming it something unique to prevent Conn vs net.Conn confusion).
type SshProxyConn struct {
net.Conn
idleTimeout time.Duration
}
func (c *SshProxyConn) Read(b []byte) (int, error) {
err := c.Conn.SetReadDeadline(time.Now().Add(c.idleTimeout))
if err != nil {
return 0, err
}
return c.Conn.Read(b)
}
func HandleConn(conn net.Conn) {
This makes is more clear what your primary issue is, which you passed the normal net.Conn to your SSH server, not your wrapper class. So
sConn, chans, reqs, err := ssh.NewServerConn(conn, config)
should be EDIT
sshproxyconn := &SshProxyConn{conn, time.Second * 20}
Conn, chans, reqs, err := ssh.NewServerConn(sshproxyconn , config)
I'm implementing a TCP server application that accepts incoming TCP connections in an infinite loop.
I'm trying to use Context throughout the application to allow shutting down, which is generally working great.
The one thing I'm struggling with is cancelling a net.Listener that is waiting on Accept(). I'm using a ListenConfig which, I believe, has the advantage of taking a Context when then creating a Listener. However, cancelling this Context does not have the intended effect of aborting the Accept call.
Here's a small app that demonstrates the same problem:
package main
import (
"context"
"fmt"
"net"
"time"
)
func main() {
lc := net.ListenConfig{}
ctx, cancel := context.WithCancel(context.Background())
go func() {
time.Sleep(2*time.Second)
fmt.Println("cancelling context...")
cancel()
}()
ln, err := lc.Listen(ctx, "tcp", ":9801")
if err != nil {
fmt.Println("error creating listener:", err)
} else {
fmt.Println("listen returned without error")
defer ln.Close()
}
conn, err := ln.Accept()
if err != nil {
fmt.Println("accept returned error:", err)
} else {
fmt.Println("accept returned without error")
defer conn.Close()
}
}
I expect that, if no clients connect, when the Context is cancelled 2 seconds after startup, the Accept() should abort. However, it just sits there until you Ctrl-C out.
Is my expectation wrong? If so, what is the point of the Context passed to ListenConfig.Listen()?
Is there another way to achieve the same goal?
I believe you should be closing the listener when your timeout runs out. Then, when Accept returns an error, check that it's intentional (e.g. the timeout elapsed).
This blog post shows how to do a safe shutdown of a TCP server without a context. The interesting part of the code is:
type Server struct {
listener net.Listener
quit chan interface{}
wg sync.WaitGroup
}
func NewServer(addr string) *Server {
s := &Server{
quit: make(chan interface{}),
}
l, err := net.Listen("tcp", addr)
if err != nil {
log.Fatal(err)
}
s.listener = l
s.wg.Add(1)
go s.serve()
return s
}
func (s *Server) Stop() {
close(s.quit)
s.listener.Close()
s.wg.Wait()
}
func (s *Server) serve() {
defer s.wg.Done()
for {
conn, err := s.listener.Accept()
if err != nil {
select {
case <-s.quit:
return
default:
log.Println("accept error", err)
}
} else {
s.wg.Add(1)
go func() {
s.handleConection(conn)
s.wg.Done()
}()
}
}
}
func (s *Server) handleConection(conn net.Conn) {
defer conn.Close()
buf := make([]byte, 2048)
for {
n, err := conn.Read(buf)
if err != nil && err != io.EOF {
log.Println("read error", err)
return
}
if n == 0 {
return
}
log.Printf("received from %v: %s", conn.RemoteAddr(), string(buf[:n]))
}
}
In your case you should call Stop when the context runs out.
If you look at the source code of TCPConn.Accept, you'll see it basically calls the underlying socket accept, and the context is not piped through there. But Accept is simple to cancel by closing the listener, so piping the context all the way isn't strictly necessary.
I have found a strange behaviour when receiving message from GCP Pub/Sub.
Following codes are how I register the subscriptions using pubsub client
gcp.go
package gcp
import (
"context"
"path"
"runtime"
"google.golang.org/api/option"
"cloud.google.com/go/pubsub"
)
// PubsubClient is the GCP pubsub service client.
var PubsubClient *pubsub.Client
// Initialize initializes GCP client service using the environment.
func Initialize(env, projectName string) error {
var err error
ctx := context.Background()
credentialOpt := option.WithCredentialsFile(getFilePathByEnv(env))
PubsubClient, err = pubsub.NewClient(ctx, projectName, credentialOpt)
return err
}
// GetTopic returns the specified topic in GCP pub/sub service and create it if it not exist.
func GetTopic(topicName string) (*pubsub.Topic, error) {
topic := PubsubClient.Topic(topicName)
ctx := context.Background()
isTopicExist, err := topic.Exists(ctx)
if err != nil {
return topic, err
}
if !isTopicExist {
ctx = context.Background()
topic, err = PubsubClient.CreateTopic(ctx, topicName)
}
return topic, err
}
// GetSubscription returns the specified subscription in GCP pub/sub service and creates it if it not exist.
func GetSubscription(subName string, topic *pubsub.Topic) (*pubsub.Subscription, error) {
sub := PubsubClient.Subscription(subName)
ctx := context.Background()
isSubExist, err := sub.Exists(ctx)
if err != nil {
return sub, err
}
if !isSubExist {
ctx = context.Background()
sub, err = PubsubClient.CreateSubscription(ctx, subName, pubsub.SubscriptionConfig{Topic: topic})
}
return sub, err
}
func getFilePathByEnv(env string) string {
_, filename, _, _ := runtime.Caller(1)
switch env {
case "local":
return path.Join(path.Dir(filename), "local.json")
case "development":
return path.Join(path.Dir(filename), "development.json")
case "staging":
return path.Join(path.Dir(filename), "staging.json")
case "production":
return path.Join(path.Dir(filename), "production.json")
default:
return path.Join(path.Dir(filename), "local.json")
}
}
main.go
package main
import (
"context"
"fmt"
"log"
"net/http"
"runtime"
"runtime/debug"
"runtime/pprof"
"time"
"rpriambudi/pubsub-receiver/gcp"
"cloud.google.com/go/pubsub"
"github.com/go-chi/chi"
)
func main() {
log.Fatal(http.ListenAndServe(":4001", Route()))
}
func Route() *chi.Mux {
InitializeSubscription()
chiRoute := chi.NewRouter()
chiRoute.Route("/api", func(r chi.Router) {
r.Get("/_count", func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "Number of goroutines: %v", runtime.NumGoroutine())
})
r.Get("/_stack", getStackTraceHandler)
})
return chiRoute
}
func InitializeSubscription() {
gcp.Initialize("local", "fifth-bonbon-277102")
go pubsubHandler("test-topic-1", "test-topic-1-subs")
go pubsubHandler("test-topic-2", "test-topic-2-subs")
go pubsubHandler("test-topic-3", "test-topic-3-subs")
// ....
return
}
func getStackTraceHandler(w http.ResponseWriter, r *http.Request) {
stack := debug.Stack()
w.Write(stack)
pprof.Lookup("goroutine").WriteTo(w, 2)
}
func pubsubHandler(topicID string, subscriptionID string) {
topic, err := gcp.GetTopic(topicID)
fmt.Println("topic: ", topic)
if err != nil {
fmt.Println("Failed get topic: ", err)
return
}
sub, err := gcp.GetSubscription(subscriptionID, topic)
fmt.Println("subscription: ", sub)
if err != nil {
fmt.Println("Get subscription err: ", err)
return
}
err = sub.Receive(context.Background(), func(ctx context.Context, msg *pubsub.Message) {
messageHandler(subscriptionID, ctx, msg)
})
if err != nil {
fmt.Println("receive error: ", err)
}
}
func messageHandler(subscriptionID string, ctx context.Context, msg *pubsub.Message) {
defer func() {
if r := recover(); r != nil {
fmt.Println("recovered from panic.")
msg.Ack()
}
}()
fmt.Println("message of subscription: ", subscriptionID)
fmt.Println("Message ID: ", string(msg.ID))
fmt.Println("Message received: ", string(msg.Data))
msg.Ack()
time.Sleep(10 * time.Second)
}
It works great when i just have a few of pubsubHandler inside the InitializeSubscription. But when I adding more pubsubHandler inside the initialize function (approx 10 or more handler), things starting got interesting. The ack never reach the pubsub server, making the message is simply not ack-ed (I have checked the AcknowledgeRequest in metrics explorer, and no ack request coming). Thus, the message is keep coming back to the subscriber. Also, when i restart the application, sometimes it won't receive any message, neither new or an un-acked ones.
I seems to find a workaround by set the NumGoroutines to 1 for each subscription object in the pubsubHandler function.
func pubsubHandler(topicID string, subscriptionID string) {
....
sub, err := gcp.GetSubscription(subscriptionID, topic)
....
sub.ReceiverSettings.NumGoroutines = 1
err = sub.Receive(context.Background(), func(ctx context.Context, msg *pubsub.Message) {
messageHandler(subscriptionID, ctx, msg)
})
....
}
My question is, is this an intended behaviour? What is the root cause that may lead to those unexpected behaviours? Or my implementations is simply wrong, to achieve the intended results? (multiple subscription inside one application). Or is there any best practices to follow when creating a subscription handler?
In my understanding, the Receive function from pubsub.Subscription is a blocking code natively. Hence, when I tried to run it inside a goroutines, it may lead to an unexpected side effects, especially if we're not limiting the number of goroutines that may handle the messages. Is my reasoning a valid one?
Thank you for your answers, and have a good day!
Edit 1: Updating the example to a full code, since the pubsub client is not directly imported in the main.go before.
I believe the issue might be the rate at which you are handling messages (currently 10 seconds per message). If you receive too many messages at once, your client might be overwhelmed, which will lead to a buildup of a backlog of messages.
I recommend playing around with flow control settings and increasing ReceiveSettings.NumGoroutines to something higher than the default of 10. If your publish rate is high, you could also increase MaxOutstandingMessages, or completely disable the limit by setting it to -1. This tells the client to hold onto more messages at once, a limit that is shared per Receive call.
I was performing some RPC tests, and stumbled across a problem I can't seem to solve. In my testing I create three separate RPC servers, all of which I try to close and shutdown. However upon performing my last test (TestRpcCodecServerClientComm), it seems my client connection is connecting to the first RPC server I started (I know this because I at some point attached IDs to the RPCHandlers), even though I attempted everything I could to make sure it was shutdown. Though the code is not there I have attempted to inspect every single error I could, but that did not bring about anything.
rpc.go
package rbot
import (
"io"
"net"
"net/rpc"
"net/rpc/jsonrpc"
)
func RpcCodecClientWithPort(port string) (rpc.ClientCodec, error) {
conn, err := net.Dial("tcp", "localhost:"+port)
if err != nil {
return nil, err
}
return jsonrpc.NewClientCodec(conn), nil
}
func RpcCodecServer(conn io.ReadWriteCloser) rpc.ServerCodec {
return jsonrpc.NewServerCodec(conn)
}
rpc_test.go
package rbot
import (
"errors"
"fmt"
"net"
"net/rpc"
"testing"
)
type RPCHandler struct {
RPCServer net.Listener
conn rpc.ServerCodec
done chan bool
TestPort string
stop bool
GotRPC bool
}
func (r *RPCHandler) SetupTest() {
r.stop = false
r.GotRPC = false
r.done = make(chan bool)
r.TestPort = "5556"
}
// TODO: Create separate function to handle erroring
func (r *RPCHandler) CreateRPCServer() error {
rpc.RegisterName("TestMaster", TestAPI{r})
var err error
r.RPCServer, err = net.Listen("tcp", ":"+r.TestPort)
if err != nil {
return err
}
go func() {
for {
conn, err := r.RPCServer.Accept()
if err != nil || r.stop {
r.done <- true
return
}
r.conn = RpcCodecServer(conn)
rpc.ServeCodec(r.conn)
}
}()
return nil
}
func (r *RPCHandler) CloseRPCServer() error {
r.stop = true
if r.conn != nil {
err := r.conn.Close()
if err != nil {
fmt.Println(err)
}
}
err := r.RPCServer.Close()
<-r.done
return err
}
type TestAPI struct {
t *RPCHandler
}
func (tapi TestAPI) Send(msg string, result *string) error {
if msg == "Got RPC?" {
tapi.t.GotRPC = true
return nil
}
return errors.New("Didn't receive right message")
}
// Check if we can create and close an RPC server successfully using the RPC server codec.
func TestRpcCodecServer(t *testing.T) {
r := RPCHandler{}
r.SetupTest()
err := r.CreateRPCServer()
if err != nil {
t.Fatalf("Could not create rpc server! %s:", err.Error())
}
err = r.CloseRPCServer()
if err != nil {
t.Fatalf("Could not close RPC server! %s:", err.Error())
}
}
// Check if we can create a client without erroring.
func TestRpcCodecClientWithPortt(t *testing.T) {
r := RPCHandler{}
r.SetupTest()
r.CreateRPCServer()
defer r.CloseRPCServer()
RPCClient, err := RpcCodecClientWithPort(r.TestPort)
defer RPCClient.Close()
if err != nil {
t.Fatalf("Could not create an RPC client! %s:", err.Error())
}
}
// Let's double check and make sure our server and client can speak to each other
func TestRpcCodecServerClientComm(t *testing.T) {
r := RPCHandler{}
r.SetupTest()
r.CreateRPCServer()
defer r.CloseRPCServer()
RPCCodec, _ := RpcCodecClientWithPort(r.TestPort)
RPCClient := rpc.NewClientWithCodec(RPCCodec)
defer RPCClient.Close()
var result string
err := RPCClient.Call("TestMaster.Send", "Got RPC?", &result)
if err != nil {
t.Fatalf("Error while trying to send RPC message: %s", err.Error())
}
if !r.GotRPC {
t.Fatalf("Could not send correct message over RPC")
}
}
Not sure if I'm just mishandling the connection or something of the like, any help would be much appreciated.
For the Record The RPC api does receive the correct string message
While not the source of your problems, your test configuration has a few race conditions which you should take care of before they cause problems. Always check for issues with the -race option. You should also let the OS allocate the port so you don't run into conflicts. See for example how httptest.Server works.
Your failure here is that you're not creating a new rpc.Server for each test, you're reusing the rpc.DefaultServer. The first call to CreateRPCServer registers a TestAPI under the name TestMaster. Each subsequent call uses the already registered instance.
If you create a new rpc.Server each time you setup the test and register a new TestAPI, the final test will pass.
srv := rpc.NewServer()
srv.RegisterName("TestMaster", testAPI)
...
// and then use srv to handle the new connection
srv.ServeCodec(RpcCodecServer(conn))
Bolt obtains a file lock on the data file so multiple processes cannot open the same database at the same time. Opening an already open Bolt database will cause it to hang until the other process closes it.
As this is the case,is there any connection pooling concept like various clients connecting and accessing the database at the same time.? Is this possible in boltdb?Like there are various connections reading and writing in the database at the same time.How it can be implemented?
A Bolt database is usually embedded into a larger program and is not used over the network like you would with shared databases (think SQLite vs MySQL). Using Bolt is a bit like having a persistent map[[]byte][]byte if that were possible. Depending on what you are doing, you might want to just use something like Redis.
That said, if you need to use Bolt this way, it is not very difficult to wrap with a simple server. Here is an example that writes/reads keys from a Bolt DB over HTTP. You can use Keep-Alive for connection pooling.
Code at: https://github.com/skyec/boltdb-server
package main
import (
"flag"
"fmt"
"io/ioutil"
"log"
"net/http"
"time"
"github.com/boltdb/bolt"
"github.com/gorilla/mux"
)
type server struct {
db *bolt.DB
}
func newServer(filename string) (s *server, err error) {
s = &server{}
s.db, err = bolt.Open(filename, 0600, &bolt.Options{Timeout: 1 * time.Second})
return
}
func (s *server) Put(bucket, key, contentType string, val []byte) error {
return s.db.Update(func(tx *bolt.Tx) error {
b, err := tx.CreateBucketIfNotExists([]byte(bucket))
if err != nil {
return err
}
if err = b.Put([]byte(key), val); err != nil {
return err
}
return b.Put([]byte(fmt.Sprintf("%s-ContentType", key)), []byte(contentType))
})
}
func (s *server) Get(bucket, key string) (ct string, data []byte, err error) {
s.db.View(func(tx *bolt.Tx) error {
b := tx.Bucket([]byte(bucket))
r := b.Get([]byte(key))
if r != nil {
data = make([]byte, len(r))
copy(data, r)
}
r = b.Get([]byte(fmt.Sprintf("%s-ContentType", key)))
ct = string(r)
return nil
})
return
}
func (s *server) ServeHTTP(w http.ResponseWriter, r *http.Request) {
vars := mux.Vars(r)
if vars["bucket"] == "" || vars["key"] == "" {
http.Error(w, "Missing bucket or key", http.StatusBadRequest)
return
}
switch r.Method {
case "POST", "PUT":
data, err := ioutil.ReadAll(r.Body)
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
err = s.Put(vars["bucket"], vars["key"], r.Header.Get("Content-Type"), data)
w.WriteHeader(http.StatusOK)
case "GET":
ct, data, err := s.Get(vars["bucket"], vars["key"])
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
return
}
w.Header().Add("Content-Type", ct)
w.Write(data)
}
}
func main() {
var (
addr string
dbfile string
)
flag.StringVar(&addr, "l", ":9988", "Address to listen on")
flag.StringVar(&dbfile, "db", "/var/data/bolt.db", "Bolt DB file")
flag.Parse()
log.Println("Using Bolt DB file:", dbfile)
log.Println("Listening on:", addr)
server, err := newServer(dbfile)
if err != nil {
log.Fatalf("Error: %s", err)
}
router := mux.NewRouter()
router.Handle("/v1/buckets/{bucket}/keys/{key}", server)
http.Handle("/", router)
log.Fatal(http.ListenAndServe(addr, nil))
}
There is no connection pooling concept in boltdb, because there is no connection. It is not a client/server database, it is an embedded database (like sqlite or Berkeley-DB).
Boltdb is designed so that multiple goroutines of the same process can access the database at the same time (using different transactions). The model is single writer, multiple readers. Boltdb is not designed to support accesses from multiple processes.
If you need a Go program to use an embedded database supporting access from multiple processes at the same time, you may want to have a look at the wrappers over LMDB, such as:
https://github.com/szferi/gomdb
https://github.com/armon/gomdb