How to free memory manually in golang - go

Question

Below is a code to calculte C(36,8) and save result to file
func combine_dfs(n int, k int) (ans [][]int) {
temp := []int{}
var dfs func(int)
dfs = func(cur int) {
if len(temp)+(n-cur+1) < k {
return
}
if len(temp) == k {
comb := make([]int, k)
copy(comb, temp)
ans = append(ans, comb)
return
}
temp = append(temp, cur)
dfs(cur + 1)
temp = temp[:len(temp)-1]
dfs(cur + 1)
}
dfs(1)
return
}
func DoCombin() {
fmt.Printf("%v\n", "calculator...")
cst := []byte{}
for i := 'a'; i <= 'z'; i++ {
cst = append(cst, byte(i))
}
for i := '0'; i <= '9'; i++ {
cst = append(cst, byte(i))
}
n := 36
k := 8
arr := combine_dfs(n, k)
fmt.Printf("%v\n", "writefile...")
file, _ := os.OpenFile("result.txt", os.O_CREATE|os.O_TRUNC|os.O_RDWR|os.O_APPEND, 0666)
defer file.Close()
for _, m := range arr {
b:= bytes.Buffer{}
b.Reset()
for _, i := range m {
b.WriteByte(cst[i-1])
}
b.WriteByte('\n')
file.Write(b.Bytes())
}
}
but i write file so slow..
so i want use goroutine to write file (use pool to limit the number of goroutine):
func DoCombin2() {
fmt.Printf("%v\n", "calculator...")
cst := []byte{}
for i := 'a'; i <= 'z'; i++ {
cst = append(cst, byte(i))
}
for i := '0'; i <= '9'; i++ {
cst = append(cst, byte(i))
}
n := 36
k := 8
arr := combine_dfs(n, k)
fmt.Printf("%v\n", "writefile...")
file, _ := os.OpenFile("result.txt", os.O_CREATE|os.O_TRUNC|os.O_RDWR|os.O_APPEND, 0666)
defer file.Close()
pool := make(chan int, 100)
for _, m := range arr {
go func(m []int) {
pool <- 1
b := bytes.Buffer{}
b.Reset()
for _, i := range m {
b.WriteByte(cst[i-1])
}
b.WriteByte('\n')
file.Write(b.Bytes())
<-pool
}(m)
}
}
but the memory exploded
I try using sync.Pool to avoid it, but it fail:
var bufPool = sync.Pool{
New: func() interface{} {
return new(bytes.Buffer)
},
}
func DoCombin() {
fmt.Printf("%v\n", "calculator...")
cst := []byte{}
for i := 'a'; i <= 'z'; i++ {
cst = append(cst, byte(i))
}
for i := '0'; i <= '9'; i++ {
cst = append(cst, byte(i))
}
n := 36
k := 8
arr := combine_dfs(n, k)
fmt.Printf("%v\n", "writefile...")
file, _ := os.OpenFile("result.txt", os.O_CREATE|os.O_TRUNC|os.O_RDWR|os.O_APPEND, 0666)
defer file.Close()
pool := make(chan int, 100)
for _, m := range arr {
go func(m []int) {
pool <- 1
b, _ := bufPool.Get().(*bytes.Buffer)
b.Reset()
for _, i := range m {
b.WriteByte(cst[i-1])
}
b.WriteByte('\n')
bufPool.Put(b)
file.Write(b.Bytes())
<-pool
}(m)
}
}
Is there any way to avoid memory explosion?
1.Why can't I avoid it after using sync.Pool?
2.Is there any way to limit memory usage in windows(in linux i know) ?
3.Is there other way to avoid memory explosion?
4.Is the memory explosion because of bytes.Buffer? How to free bytes.Buffer manually?

Answer 1

Update 02/20/2023
This proposal arenas is on hold indefinitely due to serious API concerns. The GOEXPERIMENT=arena code may be changed incompatibly or removed at any time, and we do not recommend its use in production.
Per this Proposal: arena: new package providing memory arenas
We propose the addition of a new arena package to the Go standard library. The arena package will allow the allocation of any number of arenas. Objects of arbitrary type can be allocated from the memory of the arena, and an arena automatically grows in size as needed. When all objects in an arena are no longer in use, the arena can be explicitly freed to reclaim its memory efficiently without general garbage collection. We require that the implementation provide safety checks, such that, if an arena free operation is unsafe, the program will be terminated before any incorrect behavior happens.
This feature has been merged to the master branch under arena, and maybe could be released in go 1.20. With the arena package, you could allocate memory by yourself and manually free it if it is no longer in use.
Sample codes
a := arena.NewArena()
defer a.Free()
tt := arena.New[T1](a)
tt.n = 1
ts := arena.MakeSlice[T1](a, 99, 100)
if len(ts) != 99 {
t.Errorf("Slice() len = %d, want 99", len(ts))
}
if cap(ts) != 100 {
t.Errorf("Slice() cap = %d, want 100", cap(ts))
}
ts[1].n = 42

Answer 2

in go 1.19
The garbage collector has added support for a soft memory limit,
The garbage collector has added support for a soft memory limit, discussed in detail in the new garbage collection guide. The limit can be particularly helpful for optimizing Go programs to run as efficiently as possible in containers with dedicated amounts of memory.
the new garbage collection guide