Go webserver performance drastically drops as number of requests increases - go

Question

I'm benchmarking a simple webserver written in Go using wrk. The server is running on a machine with 4GB RAM. At the beginning of the test, the performance is very good with the code serving up to 2000 requests/second. But as time goes on, memory utilized by the process creeps up and once it reaches 85% (I'm checking this using top), the throughput drops to ~100 requests/second. The throughput again increases to optimal amounts once I restart the server.
Is the degradation of performance due to memory issues? Why isn't Go releasing this memory? My Go server looks like this:
func main() {
defer func() {
// Wait for all messages to drain out before closing the producer
p.Flush(1000)
p.Close()
}()
http.HandleFunc("/endpoint", handler)
log.Fatal(http.ListenAndServe(":8080", nil))
}
In the handler, I convert the incoming Protobuf message to a Json and write it to Kafka using confluent Kafka Go library.
var p, err = kafka.NewProducer(&kafka.ConfigMap{
"bootstrap.servers": "abc-0.com:6667,abc-1.com:6667",
"message.timeout.ms": "30000",
"sasl.kerberos.keytab": "/opt/certs/TEST.KEYTAB",
"sasl.kerberos.principal": "TEST#TEST.ABC.COM",
"sasl.kerberos.service.name": "kafka",
"security.protocol": "SASL_PLAINTEXT",
})
var topic = "test"
func handler(w http.ResponseWriter, r *http.Request) {
body, _ := ioutil.ReadAll(r.Body)
// Deserialize byte[] to Protobuf message
protoMessage := &tutorial.REALTIMEGPS{}
_ := proto.Unmarshal(body, protoMessage)
// Convert Protobuf to Json
realTimeJson, _ := convertProtoToJson(protoMessage)
_, err := fmt.Fprintf(w, "")
if err != nil {
log.Fatal(responseErr)
}
// Send to Kafka
produceMessage([]byte(realTimeJson))
}
func produceMessage(message []byte) {
// Delivery report
go func() {
for e := range p.Events() {
switch ev := e.(type) {
case *kafka.Message:
if ev.TopicPartition.Error != nil {
log.Println("Delivery failed: ", ev.TopicPartition)
} else {
log.Println("Delivered message to ", ev.TopicPartition)
}
}
}
}()
// Send message
_ := p.Produce(&kafka.Message{
TopicPartition: kafka.TopicPartition{Topic: &topic, Partition: kafka.PartitionAny},
Value: message,
}, nil)
}
func convertProtoToJson(pb proto.Message) (string, error) {
marshaler := jsonpb.Marshaler{}
json, err := marshaler.MarshalToString(pb)
return json, err
}

Answer 1

The problem is that at the end of each of your request you call produceMessage(), which sends a message to kafka, and launches a goroutine to receive events for checking errors.
Your code launches goroutines non-stop as incoming requests come in, and they don't end until something goes wrong with your kafka client. This requires more and more memory, and likely more and more CPU as more goroutines are to be scheduled.
Don't do this. A single goroutine is sufficient for delivery report purposes. Launch a single goroutine when your p variable is set, and you're good to go.
For example:
var p *kafka.Producer
func init() {
var err error
p, err = kafka.NewProducer(&kafka.ConfigMap{
// ...
}
if err != nil {
// Handle error
}
// Delivery report
go func() {
for e := range p.Events() {
switch ev := e.(type) {
case *kafka.Message:
if ev.TopicPartition.Error != nil {
log.Println("Delivery failed: ", ev.TopicPartition)
} else {
log.Println("Delivered message to ", ev.TopicPartition)
}
}
}
}()
}