GoLang net/http memory keeps increasing on contineous requests - go

Question

I have the following code in GoLang
package main
import (
"bytes"
"encoding/json"
"io/ioutil"
"log"
"net/http"
"time"
)
func httpClient() *http.Client {
var transport http.RoundTripper = &http.Transport{
DisableKeepAlives: false,
}
client := &http.Client{Timeout: 60 * time.Second, Transport: transport}
return client
}
func sendRequest(client *http.Client, method string) []byte {
endpoint := "https://httpbin.org/post"
values := map[string]string{"foo": "baz"}
jsonData, err := json.Marshal(values)
req, err := http.NewRequest(method, endpoint, bytes.NewBuffer(jsonData))
if err != nil {
log.Fatalf("Error Occurred. %+v", err)
}
resp, err:= client.Do(req)
if err != nil {
defer resp.Body.Close()
log.Fatalf("Error sending request to API endpoint. %+v", err)
}
// Close the connection to reuse it
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatalf("Couldn't parse response body. %+v", err)
}
return body
}
func main() {
// c should be re-used for further calls
c := httpClient()
for i := 1; i <= 60; i++ {
response := sendRequest(c, http.MethodPost)
log.Println("Response Body:", string(response))
response = nil
time.Sleep(time.Millisecond * 1000)
}
}
When executed, it keeps the memory size increasing and the growth goes to as much as 90mb in one hour. is the gc not working properly. Even though i am using same httpclient for multiple requests but it still looks like theres something thats increasing the size of memory footprint.

Answer 1

I advice you to use tools like pprof, these are very useful at troubleshooting precisely this kind of issues.
You have set DisableKeepAlives field to false, which means that it will keep open connections even after the requests have been made, leading to further memory leaks. You should also call defer resp.Body.Close() after calling ioutil.ReadAll(resp.Body). This is precisely the purpose of the defer keyword - preventing memory leaks. GC does not mean absolute memory safety.
Also, outside of main avoid using log.Fatal. Use leveled logger, like zap or zerolog instead, since log.Fatal calls os.Exit(1) with an immediate effect, which means your defer statements will take no effect, or call plain panic. See Should a Go package ever use log.Fatal and when?