convert byte[] to string in golang strange occupy heap - go

I found strange occupy heap when convert byte[] to string with below code
package main
import (
"bytes"
"fmt"
"net/http"
_ "net/http/pprof"
"strings"
"time"
)
var (
c = make(chan int, 500000)
)
func main() {
go func() {
http.ListenAndServe(":8080", nil)
}()
f := func(ss []string) {
fmt.Println(ss)
time.Sleep(time.Millisecond)
<-c
}
for {
c <- 1
bs := bytes.NewBufferString("A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z").Bytes()
fmt.Println(bs) // will raise memory leak after marked as comment???
s := string(bs)
ss := strings.Split(s, ",")
go f(ss)
}
}
without fmt.Println(bs) will gradually exhausting memory.
with fmt.Println(bs) work fine. i can't understand what happened? i'm worked with version go1.9.2 darwin/amd64

No, there is no memory leak:
You are using 500000 concurrent goroutines, you just need to limit (reduce) the number of concurrent goroutines, e.g.:
c := make(chan int, runtime.NumCPU())
Try this ( and see the end of this edit):
package main
import (
"bytes"
"fmt"
"runtime"
"strings"
"time"
)
func main() {
c := make(chan int, runtime.NumCPU())
for {
c <- 1
bs := bytes.NewBufferString("A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z").Bytes()
s := string(bs)
ss := strings.Split(s, ",")
go func(ss []string) {
fmt.Println(ss)
time.Sleep(time.Millisecond)
<-c
}(ss)
}
}
Your Code:
package main
import (
"bytes"
"fmt"
"net/http"
_ "net/http/pprof"
"strings"
"time"
)
var (
c = make(chan int, 500000)
)
func main() {
go func() {
http.ListenAndServe(":8080", nil)
}()
f := func(ss []string) {
fmt.Println(ss)
time.Sleep(time.Millisecond)
<-c
}
for {
c <- 1
bs := bytes.NewBufferString("A,B,C,D,E,F,G,H,I,J,K,L,M,N,O,P,Q,R,S,T,U,V,W,X,Y,Z").Bytes()
// fmt.Println(bs) // will raise memory leak after marked as comment???
s := string(bs)
ss := strings.Split(s, ",")
go f(ss)
}
}
It reaches steady state after a while and even reduced Memory usage:
// Mem CPU time:
// 5,464,208K 0:1:20
// 5,468,208K 0:2:20
// 5,469,608K 0:3:20
// 5,469,844K 0:4:20
// 5,469,844K 0:5:20
// 5,469,848K 0:6:20
// 5,469,848K 0:7:20 fixed
// 5,469,848K 0:8:20 fixed
// 5,469,616K 0:9:20 reduced

Related

What Did I Miss in Input Process?

I am solving a problem in Hackerearth. Passed all the test cases except 1, showing "Time limit exceeded". What did I really miss in my code?
package main
import(
"fmt"
"strings"
)
func rotateRight(numbers []int, size int, k int) []int {
new_numbers := make([]int, size)
for index, value := range numbers {
new_numbers[(index + k) % size] = value
}
return new_numbers
}
func main() {
var test_case, size, k int
fmt.Scanf("%v", &test_case)
fmt.Scanln()
for i := 0; i < test_case; i++ {
fmt.Scanf("%v %v", &size, &k)
fmt.Scanln()
numbers := make([]int, size)
for i := 0; i<size; i++ {
fmt.Scanf("%v", &numbers[i])
}
result := rotateRight(numbers, size, k)
fmt.Println(strings.Trim(fmt.Sprint(result), "[]"))
}
}
maybe the reason is the way that you read the data, fmt is really slow, try change it with
package main
import (
"bufio"
"os"
)
func main() {
sc := bufio.NewScanner(os.Stdin)
sc.Scan()
sc.Text()//here you have your data
}
this change will improve the time wasted

A Tour of Go Exercise: rot13Reader, why is the reader called twice?

import (
"io"
"os"
"strings"
"fmt"
)
type rot13Reader struct {
r io.Reader
}
func (rot13 rot13Reader) Read(b []byte) (int, error){
n,err := rot13.r.Read(b)
fmt.Printf("\n%v %v\n",n, err)
const A byte ='A'
const a byte ='a'
for i,x := range b{
if x==0 {
n=i
break
}
switch {
case A<=x && x<a:
tmp := x-A
b[i] = A+((tmp+13)%26)
case a<=x && x<a+26 :
tmp := x-a
b[i] = a+((tmp+13)%26)
}
}
return n, err
}
func main() {
s := strings.NewReader("Lbh penpxrq gur pbqr!")
r := rot13Reader{s}
io.Copy(os.Stdout, &r)
}
The code above outputs
21 <nil>
You cracked the code!
0 EOF
Lbh penpxrq gur pbqr!
I have two questions :
Why is the reader being called twice (the second call returning 0 and EOF)
Why does on the second call, b is back to the original chiper? If it reads 0 character, shouldn't it still be "You cracked the code!"?

Access with mutex has more speed on more cpu

I tried some synchronization techniques to share state between goroutines and find out that incorrect variant (without sync) works slowly than same program with mutex.
Given the code:
package main
import (
"sync"
"time"
"fmt"
)
func main() {
var wg sync.WaitGroup
var mu sync.Mutex
hash := make(map[string]string)
hash["test"] = "string"
num := 40000000
wg.Add(num)
start := time.Now()
for i := 0; i < num; i++ {
go func() {
mu.Lock()
_, _ = hash["test"]
mu.Unlock()
wg.Done()
}()
}
wg.Wait()
fmt.Println(time.Since(start))
}
Perform on my laptop with 8 HT cores for 9-10 seconds.
But if just remove sync it works for 11-12 seconds:
package main
import (
"sync"
"time"
"fmt"
)
func main() {
var wg sync.WaitGroup
hash := make(map[string]string)
hash["test"] = "string"
num := 40000000
wg.Add(num)
start := time.Now()
for i := 0; i < num; i++ {
go func() {
_, _ = hash["test"]
wg.Done()
}()
}
wg.Wait()
fmt.Println(time.Since(start))
}
A synced version is faster and unitizes CPU much higher. Question is why?
My thought is about how goroutines scheduled and overhead for context switching because of the more GOMAXPROCS the greater the gap between these two versions. But I can't explain the real reason for that that happens under the hood of the scheduler.

Read space separated integers from stdin into int slice

I'm trying to read from stdin two lines of an unknown number of space-separated integers. I would like to store each lines ints into their own int slice.
For example, my input may look like this:
1 2 3
4 5 6
and I want to read this into two []int:
[1,2,3]
[4,5,6]
This is what I have so far. scanner.Scan() is giving me the line, but I'm not sure how to convert that into a []int:
package main
import (
"fmt"
"os"
"bufio"
)
func main() {
var firstLine []int
var secondLine []int
scanner := bufio.NewScanner(os.Stdin)
for scanner.Scan() {
t := scanner.Text()
}
}
For example,
numbers.go:
package main
import (
"bufio"
"fmt"
"os"
"strconv"
"strings"
)
func numbers(s string) []int {
var n []int
for _, f := range strings.Fields(s) {
i, err := strconv.Atoi(f)
if err == nil {
n = append(n, i)
}
}
return n
}
func main() {
var firstLine, secondLine []int
scanner := bufio.NewScanner(os.Stdin)
for i := 1; i <= 2 && scanner.Scan(); i++ {
switch i {
case 1:
firstLine = numbers(scanner.Text())
case 2:
secondLine = numbers(scanner.Text())
}
}
fmt.Println(firstLine)
fmt.Println(secondLine)
}
Output:
$ go run numbers.go
1 2 3
4 5 6
[1 2 3]
[4 5 6]
$
If you are looking for code to read input to solve problems in hackathons, here is your best solution
package main
import (
"bufio"
"os"
"fmt"
)
func main() {
reader := bufio.NewReader(os.Stdin)
a:= read(reader,100000)
fmt.Println(a)
}
func read (reader *bufio.Reader, n int)([]uint32) {
a := make([]uint32, n)
for i:=0; i<n; i++ {
fmt.Fscan(reader, &a[i])
}
return a
}
So, this is what I ended up doing. There is likely a more idiomatic way of solving it, though.
package main
import (
"fmt"
"os"
"bufio"
"strings"
"strconv"
)
func main() {
scanner := bufio.NewScanner(os.Stdin)
scanner.Scan()
parts := strings.Split(scanner.Text(), " ")
lineOne := createIntSlice(parts)
scanner.Scan()
parts = strings.Split(scanner.Text(), " ")
lineTwo := createIntSlice(parts)
fmt.Println(lineOne)
fmt.Println(lineTwo)
}
func createIntSlice(nums []string) []int {
var r []int
for _, v := range nums {
i, _ := strconv.Atoi(v)
r = append(r, i)
}
return r
}
For another option, you can implement fmt.Scanner:
package main
import (
"bytes"
"fmt"
)
type slice struct {
tok []int
}
func (s *slice) Scan(state fmt.ScanState, verb rune) error {
tok, err := state.Token(false, func(r rune) bool { return r != '\n' })
if err != nil { return err }
if _, _, err := state.ReadRune(); err != nil {
if len(tok) == 0 {
panic(err)
}
}
b := bytes.NewReader(tok)
for {
var d int
_, err := fmt.Fscan(b, &d)
if err != nil { break }
s.tok = append(s.tok, d)
}
return nil
}
func main() {
var s slice
fmt.Scan(&s)
fmt.Println(s.tok) // [1 2 3]
}
https://golang.org/pkg/fmt#Scanner
A simpler way to Handle taking space separated integers in a slice !!
func StringToIntSlice(inputSequence string) []int {
var slicedIntegerSequence = []int{}
var splittedSequence = strings.Split(inputSequence, " ")
for _, value := range splittedSequence {
intValue, _ := strconv.Atoi(value)
slicedIntegerSequence = append(slicedIntegerSequence, intValue)
}
return slicedIntegerSequence
}
func main() {
var inputSequence string
var convertedSliceOfIntegers = []int{}
scanner := bufio.NewScanner(os.Stdin)
fmt.Println("Enter a Sequence of 10 Integers separated by spaces:")
scanner.Scan()
inputSequence = scanner.Text()
convertedSliceOfIntegers = StringToIntSlice(inputSequence)
}
package main
import (
"bufio"
"fmt"
"os"
"strings"
)
// You receive a string as parameter
// List receives N As a string slice
// Returns N as a string slice
func number(n string) []string {
list := strings.Fields(n)
return list
}
func main() {
scanner := bufio.NewScanner(os.Stdin) //Receiving user data ...
list := make([][]string, 0) // Declare a slice to receive other slices inside it
for scanner.Scan() { // Scrolls all typed data to true
// If the user does not type anything, that is, if he presses Enter an interrupt will occur
if scanner.Text() == "" {
break
} else {
list = append(list, number(scanner.Text())) // Adding the slice inside list
}
}
fmt.Println(list) // print list
}
All data is going and returning as string, but you can convert them to integers easily.

A Tour of Go example on Goroutines concurrency

I'm new to the Go Language, and learning here:
https://tour.golang.org/concurrency/1
When I run https://play.golang.org/p/9JvbtSuv5o the result is:
world
hello
hello
So Added sync.WaitGroup: https://play.golang.org/p/vjdhnDssGk
package main
import (
"fmt"
"sync"
"time"
)
var w sync.WaitGroup
func say(s string) {
for i := 0; i < 2; i++ {
time.Sleep(100 * time.Millisecond)
fmt.Println(s)
}
w.Done()
}
func main() {
w.Add(1)
go say("world")
say("hello")
w.Wait()
}
But the result is same:
world
hello
hello
What is wrong with my code?
Please help,
Thank you for your help.
w.Done() decrements the WaitGroup counter.
So your code even sometimes panic: sync: negative WaitGroup counter.
You have two Goroutines:
1 - go say("world")
2 - say("hello") inside main Goroutine
so use w.Add(2), see this working sample (The Go Playground):
package main
import (
"fmt"
"sync"
"time"
)
var w sync.WaitGroup
func say(s string) {
for i := 0; i < 2; i++ {
time.Sleep(100 * time.Millisecond)
fmt.Println(s)
}
w.Done()
}
func main() {
w.Add(2)
go say("world")
say("hello")
w.Wait()
}
output:
world
hello
hello
world
I hope this helps.
You're only adding 1 to the WaitGroup, but calling Done from 2 invocations of say.
In your example, starting the WaitGroup with 2 would work
w.Add(2)
Problem occurs due to unconditional call to w.Done(). So when u called say("hello") this also decremented counter for waitGroup.
Refer https://play.golang.org/p/wJeAyYyjA2
package main
import (
"fmt"
"sync"
"time"
)
var w sync.WaitGroup
func say(s string, async bool) {
if async {
defer w.Done()
}
for i := 0; i < 2; i++ {
time.Sleep(100 * time.Millisecond)
fmt.Println(s)
}
}
func main() {
w.Add(1)
go say("world", true)
say("hello", false)
w.Wait()
}

Resources