I'm testing this Go code on my VirtualBoxed Ubuntu 11.4
package main
import ("fmt";"time";"big")
var c chan *big.Int
func sum( start,stop,step int64) {
bigStop := big.NewInt(stop)
bigStep := big.NewInt(step)
bigSum := big.NewInt(0)
for i := big.NewInt(start);i.Cmp(bigStop)<0 ;i.Add(i,bigStep){
bigSum.Add(bigSum,i)
}
c<-bigSum
}
func main() {
s := big.NewInt( 0 )
n := time.Nanoseconds()
step := int64(4)
c = make( chan *big.Int , int(step))
stop := int64(100000000)
for j:=int64(0);j<step;j++{
go sum(j,stop,step)
}
for j:=int64(0);j<step;j++{
s.Add(s,<-c)
}
n = time.Nanoseconds() - n
fmt.Println(s,float64(n)/1000000000.)
}
Ubuntu has access to all my 4 cores. I checked this with simultaneous run of several executables and System Monitor.
But when I'm trying to run this code, it's using only one core and is not gaining any profit of parallel processing.
What I'm doing wrong?
You probably need to review the Concurrency section of the Go FAQ, specifically these two questions, and work out which (if not both) apply to your case:
Why doesn't my multi-goroutine program
use multiple CPUs?
You must set the GOMAXPROCS shell environment
variable or use the similarly-named function
of the runtime package to allow the run-time
support to utilize more than one OS thread.
Programs that perform parallel computation
should benefit from an increase in GOMAXPROCS.
However, be aware that concurrency is not parallelism.
Why does using GOMAXPROCS > 1
sometimes make my program slower?
It depends on the nature of your
program. Programs that contain several
goroutines that spend a lot of time
communicating on channels will
experience performance degradation
when using multiple OS threads. This
is because of the significant
context-switching penalty involved in
sending data between threads.
Go's goroutine scheduler is not as
good as it needs to be. In future, it
should recognize such cases and
optimize its use of OS threads. For
now, GOMAXPROCS should be set on a
per-application basis.
For more detail on this topic see the
talk entitled Concurrency is not Parallelism.
Related
In my go code, I've a no. of structs that need mutexes:
type vitalMacs struct {
mu *sync.RWMutex
macs map[string]bool
}
//macInfo would store oui info
type macInfo struct {
mu *sync.RWMutex
infoDict map[string]*string
}
var (
NetOPMutex sync.RWMutex
ipMacGlobal = make(map[string]string)
macInfoVar = macInfo{mu: &NetOPMutex}
vitalMacsVar = vitalMacs{mu: &NetOPMutex}
//vitalMacsVar = vitalMacs{mu: &sync.RWMutex{}} // <- like this
)
I could use NetOPMutex for all of them, or, have new mutexes set as &sync.RWMutex{} - which made me wonder:
Are there any side effects to having a large number different of mutexes in your code?
Not trying to avoid mutexes, you gotta guard what you gotta guard, no question there.
I know that if I use the single mutex, things would have to wait for each other and play nice. On the other hand, would having a large number of different mutexes (30+) manifest as higher cpu util or as more time in user vs kernel mode etc?
sync.Mutex and sync.RWMutex are struct types, having mutexes is the same thing as having ordinary struct values. Are there any side effects of having many struct values in your app? Obviously not.
Creating a mutex does not involve magic. It's equivalent to just creating a struct value.
Using mutexes also does not involve magic. Check the sources: Mutex.Lock() and Mutex.Unlock(). They do not launch goroutines or do something that consumes CPU.
Do not use a single mutex just to save / spare some memory. Use multiple mutexes if appropriate to reduce / mitigate contention for the locks.
I'm giving a presentation about the Go Memory Model. The memory model says that without a happens-before relationship between a write in one goroutine, and a read in another goroutine, there is no guarantee that the reader will observe the change.
To have a bigger impact on the audience, instead of just telling them that bad things can happen if you don't synchronize, I'd like to show them.
When I run the below code on my machine (2017 MacBook Pro with 3.5GHz dual-core Intel Core i7), it exits successfully.
Is there anything I can do to demonstrate the memory visibility issues?
For example are there any specific changes to the following values I could make to demonstrate the issue:
use different compiler settings
use an older version of Go
run on a different operating system
run on different hardware (such as ARM or a machine with multiple NUMA nodes).
For example in Java the flags -server and -client affect the optimizations the JVM takes and lead to visibility issues occurring.
I'm aware that the answer may be no, and that the spec may have been written to give future maintainers more flexibility with optimization. I'm aware I can make the code never exit by setting GOMAXPROCS=1 but that doesn't demonstrate visibility issues.
package main
var a string
var done bool
func setup() {
a = "hello, world"
done = true
}
func main() {
go setup()
for !done {
}
print(a)
}
I've written a program in Go that reads a single byte from a file and checks to see which bits are set. These files are usually pretty large (around 10 - 100 GB), so I don't want to read the entire file into memory. The program normally has to check millions of separate bytes.
Right now, the way I'm performing these reads is by using os.File.ReadAt(). This ended up being pretty slow, so I tried to use Goroutines to speed it up. For example:
var wg sync.WaitGroup
threadCount := 8
for i := 0; i < threadCount; i += 1 {
wg.Add(1)
go func(id int) {
defer wg.Done()
index := id
myByte := make([]byte, 1)
for index < numBytesInFile-1 { // Stop when thread would attempt to read byte outside of file
fmt.Println(file.ReadAt(myByte, index))
index += threadCount
}
}(i)
}
wg.Wait()
However, using Goroutines here didn't speed the program up at all (in fact, it made it slightly slower due to overhead). I would have thought that files on the disc could be read concurrently as long as they are opened in read-only mode (which I do in my program). Is what I'm asking for impossible, or is there some way I make concurrent reads to a file in Go?
You slowness is because of I/O and not CPU. Adding more threads will not speed up your program. Read about Amdahl's law. https://en.wikipedia.org/wiki/Amdahl%27s_law
If you do not want to read the full file into memory, you could either use a buffered reader and read in parts https://golang.org/pkg/bufio/#NewReader or you could even consider using the experimental memory-mapped files package too: https://godoc.org/golang.org/x/exp/mmap
To know more about memory mapped files, see https://en.wikipedia.org/wiki/Memory-mapped_file
I am trying to build a small tool that will allow me to run a program and track memory usage through Go. I am using r.exec = exec.Command(r.Command, r.CommandArgs...) to run the command, and runtime.MemStats to track memory usage (in a separate go routine):
func monitorRuntime() {
m := &runtime.MemStats{}
f, err := os.Create(fmt.Sprintf("mmem_%s.csv", getFileTimeStamp()))
if err != nil {
panic(err)
}
f.WriteString("Time;Allocated;Total Allocated; System Memory;Num Gc;Heap Allocated;Heap System;Heap Objects;Heap Released;\n")
for {
runtime.ReadMemStats(m)
f.WriteString(fmt.Sprintf("%s;%d;%d;%d;%d;%d;%d;%d;%d;\n", getTimeStamp(), m.Alloc, m.TotalAlloc, m.Sys, m.NumGC, m.HeapAlloc, m.HeapSys, m.HeapObjects, m.HeapReleased))
time.Sleep(5 * time.Second)
}
}
When I tested my code with simple program that just sits there (for about 12 hours), I noticed that Go is constantly allocating more memory:
System Memory
Heap Allocation
I did a few more tests such as running the monitorRuntime() function without any other code, or using pprof, such as:
package main
import (
"net/http"
_ "net/http/pprof"
)
func main() {
http.ListenAndServe(":8080", nil)
}
Yet I still noticed that memory allocation keeps going up just like in the graphs.
How can I accurately track memory usage of the program I want to run through Go?
I know one way, which I used in the past, is to use /proc/$PID/statm, but that file doesn't exist in every operating system (such as MacOS or Windows)
There isn't a way in standard Go to get the memory usage of a program called from exec.Command. runtime.ReadMemStats only returns memory tracked by the go runtime (which, in this case, is only the file handling and sprintf).
Your best bet would be to execute platform specific commands to get memory usage.
On Linux (RedHat) the following will show memory usage:
ps -No pid,comm,size,vsize,args:90
In go's runtime lib we have NumGoroutine() to return the total number of go routines running at that time. I was wondering if there was a simple way of getting the number of go routines running of a specific function?
Currently I have it telling me I have 1000, or whatever, of all go routines but I would like to know I have 500 go routines running func Foo. Possible? Simple? Shouldn't bother with it?
I'm afraid you have to count the goroutines on your own if you are interested in those numbers. The cheapest way to achieve your goal would be to use the sync/atomic package directly.
import "sync/atomic"
var counter int64
func example() {
atomic.AddInt64(&counter, 1)
defer atomic.AddInt64(&counter, -1)
// ...
}
Use atomic.LoadInt64(&counter) whenever you want to read the current value of the counter.
And it's not uncommon to have a couple of such counters in your program, so that you can monitor it easily. For example, take a look at the CacheStats struct of the recently published groupcache source.