Allocated a lot of memory in Go. How to fix? - go

Question

Several hundred MB of memory is allocated for 50 requests of 5 MB. Memory is allocated and is no longer released.
How can I clear my memory? Why can this happen?
I've tried on Ubuntu on my home pc and on VPS
package main
import (
"fmt"
"io/ioutil"
"net/http"
"time"
)
func main() {
fmt.Println("start")
for i := 0; i < 50; i++ {
go func() {
DoRequest()
}()
time.Sleep(10 * time.Millisecond)
}
time.Sleep(10 * time.Minute)
}
func DoRequest() error {
requestUrl := "https://blockchain.info/rawblock/0000000000000000000eebedea046425bd54626e6c56eb032e66e714d0141ea6"
req, err := http.NewRequest("GET", requestUrl, nil)
if err != nil {
return err
}
req.Header.Set("user-agent", "free")
httpClient := &http.Client{
Timeout: time.Second * 10,
}
resp, err := httpClient.Do(req)
if resp != nil {
defer resp.Body.Close()
}
if err != nil {
return err
}
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return err
}
fmt.Println("bodylen", len(body))
return nil
}
Allocated somewhere 400MB

Answer 1

You are creating an http client for each go-routine.
Http client is designed to be create once & used many times. They are go-routine safe.
They allow for connection reuse & other efficiency savers.
Create the http client once in main (instead of in your go-routine) & then pass this single reference to all of your 50 go-routines.
Edit: Also, while it may not make a practical difference in your case, the order for a request is usually like so:
resp, err := httpClient.Do(req)
if err != nil {
return err // check error first
}
defer resp.Body.Close() // no error - so resp will *NOT* be nil - so this is safe
Edit 2: As #Adrian has mentioned: go's garbage collection is not instantaneous - nor should it be - as it is an expensive operation. If you no longer need a block of memory - simply don't reference it anymore. Let the GC do its job, so you can focus on yours!
If you're curious about the evolution of go's GC:
https://blog.golang.org/ismmkeynote (heavy on the technical side)
What kind of Garbage Collection does Go use?

Answer 2

for i := 0; i < 50; i++ {
go func() {
DoRequest()
}()
time.Sleep(10 * time.Millisecond)
}
Never create go-routines like this. Always make sure you create go-routines the way it not fill large ( all ) memory in any case ( including worst case )
Simple solution is control the count of go-routines can spawned ( or running ) at time.
You can pre-calculate memory to be occupied in worst case by multiplying max-number of go-routines you want to run at a time and max-memory can be used by one go-routine.
You can control instances of go-routines by using channles.
Refer first answer of this stackoverflow question
Always have x number of goroutines running at any time
Always use balanced solution between perforamce and required resources.
Update June 11,2019
Here is example go program
https://play.golang.org/p/HovNRgp6FxH