I am trying to get a hang of Go Pro filer by following the example in go blog . I am not sure what I am doing wrong. But my profiled generated output shows 0 samples. Its weird.
rahul#g3ck0:~/programs/go$ go tool pprof parallel cpuprofile
Welcome to pprof! For help, type 'help'.
(pprof) top5
Total: 0 samples
The following is my code :
package main
import (
"fmt"
"os/exec"
"sync"
"strings"
"runtime/pprof"
"os"
)
func exe_cmd(cmd string, wg *sync.WaitGroup) {
out, err := exec.Command(cmd).Output()
if err != nil {
fmt.Println("error occured")
fmt.Printf("%s", err)
}
fmt.Printf("%s", out)
wg.Done()
}
func main() {
f, _ := os.Create("cpuprofile")
pprof.StartCPUProfile(f)
defer pprof.StopCPUProfile()
cmd := "echo newline >> blah.txt"
parts := strings.Fields(cmd)
head := parts[0]
parts = parts[1:len(parts)]
out, err := exec.Command(head,parts...).Output()
if err != nil {
fmt.Println("error occured")
fmt.Printf("%s", err)
}
fmt.Printf("%s", out)
}
Your profiled program runs not long enough for the profiler to pick up any profiling sample.
Basically the profiler looks periodically at the state of your program (which code is executed, what function is that, ...). If the program terminates faster than the routine that looks for a status then no status is sampled and, thus, there are no samples to look at in the end.
This is what happens for you.
One solution is set the sample rate of the profiler to a higher value, the other way
is to have your program actually do something that takes longer. For example:
f, _ := os.Create("cpuprofile")
pprof.StartCPUProfile(f)
defer pprof.StopCPUProfile()
for i := 0; i < 10; i++ {
time.Sleep(1 * time.Second)
}
Alternatively, when trying to figure out what is wrong with a isolated portion of your code,
you can write a benchmark and profile that benchmark.
Related
Go code running in the background
I am a beginner of the go language.
I wrote a small program that made a keyboard sound.
After go build main.go, you can hear the sound of the button in the current shell.
But when running ./main in the background or when re-opening a new shell will not hear the button sound.
This is what I need help with.
package main
import (
"fmt"
"github.com/eiannone/keyboard"
"github.com/faiface/beep"
"github.com/faiface/beep/speaker"
"github.com/faiface/beep/wav"
"os"
"time"
"log"
"path/filepath"
)
func main(){
env:= os.Getenv("HOME")
fmt.Println(env)
err := keyboard.Open()
if err != nil {
panic(err)
}
defer keyboard.Close()
for {
char, key, err := keyboard.GetKey()
if err != nil {
log.Fatal(err)
}else if (key == keyboard.KeyEsc){
break
}
n := int(char)
path := "./wav/*.wav"
fpath,_ := filepath.Glob(path)
name := fpath[n]
f, err := os.Open(name)
if err != nil {
log.Fatal(err)
}
streamer, format, err := wav.Decode(f)
if err != nil {
log.Fatal(err)
}
defer streamer.Close()
speaker.Init(format.SampleRate, format.SampleRate.N(time.Second/10))
done := make(chan bool)
speaker.Play(beep.Seq(streamer, beep.Callback(func() {
done <- true
})))
<-done
}
}
I want it to run in the background. I can hear the corresponding sound after pressing the keyboard.
If the program is running in the background it will not get any input from the keyboard.
Also since n depends on user input it would be easy for it to exceed the number of sound files and you would get an index out of bounds error at run-time. You should do something like this:
if fpath, _ := filepath.Glob("./wav/*.wav"); len(fpath) > 0 {
n := int(char)%len(fpath)
name := fpath[n]
...
everyone!
I'm trying to get my go code work with openstack serial console. It`s exposed via web socket. And i have problems with it.
I found gorrilla websocket lib (which is great) and took this example as a reference
With a few tweaks, now i have a code like this:
package main
import (
"log"
"net/url"
"os"
"os/signal"
"time"
"net/http"
"github.com/gorilla/websocket"
)
func main() {
DialSettings := &websocket.Dialer {
Proxy: http.ProxyFromEnvironment,
HandshakeTimeout: 45 * time.Second,
Subprotocols: []string{"binary",},
ReadBufferSize: 4096,
WriteBufferSize: 4096,
}
log.SetFlags(0)
interrupt := make(chan os.Signal, 1)
signal.Notify(interrupt, os.Interrupt)
u, _ := url.Parse("ws://172.17.0.64:6083/?token=d1763f2b-3466-424c-aece-6aeea2a733d5") //websocket url as it outputs from 'nova get-serial-console test' cmd
log.Printf("connecting to %s", u.String())
c, _, err := DialSettings.Dial(u.String(), nil)
if err != nil {
log.Fatal("dial:", err)
}
defer c.Close()
done := make(chan struct{})
go func() {
defer close(done)
for {
_, message, err := c.ReadMessage()
if err != nil {
log.Println("read:", err)
return
}
log.Printf("%s", message)
}
}()
c.WriteMessage(websocket.TextMessage, []byte("\n")) //just to force output to console
for {
select {
case <-done:
return
case <-interrupt:
log.Println("interrupt")
// Cleanly close the connection by sending a close message and then
// waiting (with timeout) for the server to close the connection.
err := c.WriteMessage(websocket.CloseMessage, websocket.FormatCloseMessage(websocket.CloseNormalClosure, ""))
if err != nil {
log.Println("write close:", err)
return
}
select {
case <-done:
case <-time.After(time.Second):
}
return
}
}
}
And i get output like this:
connecting to ws://172.17.0.64:6083/?token=d1763f2b-3466-424c-aece-6aeea2a733d5
CentOS Linux 7
(C
ore)
K
erne
l
3.10.0-862.el7.x86_64
o
n an
x
86_64
centos
-test login:
Total mess...
I think it's because i recieve just a chunks of bytes with no way to delimit them. I need some buffer to store them and when do something like bufio.ReadLine. But i'm not most experienced go programmer, and i run out of ideas how to do this. At the end i just need strings to work with.
The log package writes each log message on a separate line. If the log message does not end with a newline, then the log package will add one.
These extra newlines are garbling the output. To fix the output, replace the call to log.Printf("%s", message) with a function that does not add newlines to the output. Here are some options:
Write the message to stderr (same destination as default log package config):
os.Stderr.Write(message)
Write the message to stdout (a more conventional location to write program output):
os.Stdout.Write(message)
I want to write a mime/multipart message in Python to standard output and read that message in Golang using the mime/multipart package. This is just a learning exercise.
I tried simulating this example.
output.py
#!/usr/bin/env python2.7
import sys
s = "--foo\r\nFoo: one\r\n\r\nA section\r\n" +"--foo\r\nFoo: two\r\n\r\nAnd another\r\n" +"--foo--\r\n"
print s
main.go
package main
import (
"io"
"os/exec"
"mime/multipart"
"log"
"io/ioutil"
"fmt"
"sync"
)
var wg sync.WaitGroup
func main() {
pr,pw := io.Pipe()
defer pw.Close()
cmd := exec.Command("python","output.py")
cmd.Stdout = pw
mr := multipart.NewReader(pr,"foo")
wg.Add(1)
go func() {
defer wg.Done()
for {
p, err := mr.NextPart()
if err == io.EOF {
fmt.Println("EOF")
return
}
if err != nil {
log.Fatal(err)
}
slurp, err := ioutil.ReadAll(p)
if err != nil {
log.Fatal(err)
}
fmt.Printf("Part : %q\n", slurp)
return
}
}()
if err := cmd.Start(); err != nil {
log.Fatal(err)
}
cmd.Wait()
wg.Wait()
}
Output of go run main.go:
fatal error: all goroutines are asleep - deadlock!
Other answers regarding this topic on StackOverflow are related to channels not being closed, but I am not even using a channel. I understand that somewhere, there is infinite loop or something similar, but I don't see it.
Try something like this (explanation below):
package main
import (
"fmt"
"io"
"io/ioutil"
"log"
"mime/multipart"
"os"
"os/exec"
"sync"
"github.com/pkg/errors"
)
func readCommand(cmdStdout io.ReadCloser, wg *sync.WaitGroup, resc chan<- []byte, errc chan<- error) {
defer wg.Done()
defer close(errc)
defer close(resc)
mr := multipart.NewReader(cmdStdout, "foo")
for {
part, err := mr.NextPart()
if err != nil {
if err == io.EOF {
fmt.Println("EOF")
} else {
errc <- errors.Wrap(err, "failed to get next part")
}
return
}
slurp, err := ioutil.ReadAll(part)
if err != nil {
errc <- errors.Wrap(err, "failed to read part")
return
}
resc <- slurp
}
}
func main() {
cmd := exec.Command("python", "output.py")
cmd.Stderr = os.Stderr
pr, err := cmd.StdoutPipe()
if err != nil {
log.Fatal(err)
}
var wg sync.WaitGroup
wg.Add(1)
resc := make(chan []byte)
errc := make(chan error)
go readCommand(pr, &wg, resc, errc)
if err := cmd.Start(); err != nil {
log.Fatal(err)
}
for {
select {
case err, ok := <-errc:
if !ok {
errc = nil
break
}
if err != nil {
log.Fatal(errors.Wrap(err, "error from goroutine"))
}
case res, ok := <-resc:
if !ok {
resc = nil
break
}
fmt.Printf("Part from goroutine: %q\n", res)
}
if errc == nil && resc == nil {
break
}
}
cmd.Wait()
wg.Wait()
}
In no particular order:
Rather than using an io.Pipe() as the command's Stdout, just ask the command for it's StdoutPipe(). cmd.Wait() will ensure it's closed for you.
Set cmd.Stderr to os.Stderr so that you can see errors generated by your Python program.
I noticed this program was hanging anytime the Python program wrote to standard error. Now it doesn't :)
Don't make the WaitGroup a global variable; pass a reference to it to the goroutine.
Rather than log.Fatal()ing inside the goroutine, create an error channel to communicate errors back to main().
Rather than printing results inside the goroutine, create a result channel to communicate results back to main().
Ensure channels are closed to prevent blocking/goroutine leaks.
Separate out the goroutine into a proper function to make the code easier to read and follow.
In this example, we can create the multipart.Reader() inside our goroutine, since this is the only part of our code that uses it.
Note that I am using Wrap() from the errors package to add context to the error messages. This is, of course, not relevant to your question, but is a good habit.
The for { select { ... } } part may be confusing. This is one article I found introducing the concept. Basically, select is letting us read from whichever of these two channels (resc and errc) are currently readable, and then setting each to nil when the channel is closed. When both channels are nil, the loop exits. This lets us handle "either a result or an error" as they come in.
Edit: As johandalabacka said on the Golang Forum, it looks like the main issue here was that Python on Windows was adding an extra \r to the output, and that the problem is your Python program should omit the \r in the output string or sys.stdout.write() instead of print() ing. The output could also be cleaned up on the Golang side, but, aside from not being able to parse properly without modifying the Python side, this answer will still improve the concurrency mechanics of your program.
First time with go, and trying to get go routines and WaitGroups working.
I have a CSV file with 100 rows of data. (101 including header)
I have the following simple code:
package main
import (
"bufio"
"fmt"
"io"
"os"
"sync"
"time"
)
func main() {
start := time.Now()
numRows := 0
waitGroup := sync.WaitGroup{}
file, _ := os.Open("./data.csv")
scanner := bufio.NewScanner(file)
scanner.Scan() // to read the header
for scanner.Scan() {
err := scanner.Err()
if err != nil && err != io.EOF {
panic(err)
}
waitGroup.Add(1)
go (func() {
numRows++
waitGroup.Done()
})()
}
waitGroup.Wait()
file.Close()
fmt.Println("Finished parsing ", numRows)
fmt.Println("Elapsed time in seconds: ", time.Now().Sub(start))
}
When i run this, the numRows output fluctuates between 94 and 100 each time. I'm expecting it to be 100 each time. If i run the same code on a CSV of 10 rows of data, it outputs 10 each and every time.
Seems to me like the final few go routines aren't finishing in time.
I've tried the following which have failed:
using a CsvReader instead of a Scanner
moving waitGroup.Add(1) to underneath the anonymous func
moving the anonymous func out into a package-level scope func (and passed things round using ptrs)
What am i missing?
It's not safe to modify a single variable simultaneously in different goroutines. Some of your updates to numRows will be lost, and occasionally your program may crash.
Either protect your numRows variable with a mutex, or use one of the atomic functions to do your addition atomically:
var numRows int32
// ...
go (func() {
atomic.AddInt32(&numRows, 1)
waitGroup.Done()
})()
What do you do with this code:
for scanner.Scan() {
err := scanner.Err()
if err != nil && err != io.EOF {
panic(err)
}
waitGroup.Add(1)
go (func() {
numRows++
waitGroup.Done()
})()
}
Really all the work is done in one main goroutine and only numRows increment uses separate goroutines. I think this could be simplified to simple increment:
for scanner.Scan() {
err := scanner.Err()
if err != nil && err != io.EOF {
panic(err)
}
numRows++
}
If you want to simulate parallel parsing and pipelining you may use channels. Make only one goroutine responsible for counter increment. Every time when another goroutine wants to increment counter - it sends a message to that channel.
https://play.golang.org/p/W60twJjY8P
I have this code
subProcess := exec.Cmd{
Path: execAble,
Args: []string{
fmt.Sprintf("-config=%s", *configPath),
fmt.Sprintf("-serverType=%s", *serverType),
fmt.Sprintf("-reload=%t", *reload),
fmt.Sprintf("-listenFD=%d", fd),
},
Dir: here,
}
subProcess.Stdout = os.Stdout
subProcess.Stderr = os.Stderr
logger.Info("starting subProcess:%s ", subProcess.Args)
if err := subProcess.Run(); err != nil {
logger.Fatal(err)
}
and then I do os.Exit(1) to stop the main process
I can get output from the subprocess
but I also want to put stdin to
I try
subProcess.Stdin = os.Stdin
but it does not work
I made a simple program (for testing). It reads a number and writes the given number out.
package main
import (
"fmt"
)
func main() {
fmt.Println("Hello, What's your favorite number?")
var i int
fmt.Scanf("%d\n", &i)
fmt.Println("Ah I like ", i, " too.")
}
And here is the modified code
package main
import (
"fmt"
"io"
"os"
"os/exec"
)
func main() {
subProcess := exec.Command("go", "run", "./helper/main.go") //Just for testing, replace with your subProcess
stdin, err := subProcess.StdinPipe()
if err != nil {
fmt.Println(err) //replace with logger, or anything you want
}
defer stdin.Close() // the doc says subProcess.Wait will close it, but I'm not sure, so I kept this line
subProcess.Stdout = os.Stdout
subProcess.Stderr = os.Stderr
fmt.Println("START") //for debug
if err = subProcess.Start(); err != nil { //Use start, not run
fmt.Println("An error occured: ", err) //replace with logger, or anything you want
}
io.WriteString(stdin, "4\n")
subProcess.Wait()
fmt.Println("END") //for debug
}
You interested about these lines
stdin, err := subProcess.StdinPipe()
if err != nil {
fmt.Println(err)
}
defer stdin.Close()
//...
io.WriteString(stdin, "4\n")
//...
subProcess.Wait()
Explanation of the above lines
We gain the subprocess' stdin, now we can write to it
We use our power and we write a number
We wait for our subprocess to complete
Output
START
Hello, What's your favorite number?
Ah I like 4 too.
END
For better understanding
There's now an updated example available in the Go docs: https://golang.org/pkg/os/exec/#Cmd.StdinPipe
If the subprocess doesn't continue before the stdin is closed, the io.WriteString() call needs to be wrapped inside an anonymous function:
func main() {
cmd := exec.Command("cat")
stdin, err := cmd.StdinPipe()
if err != nil {
log.Fatal(err)
}
go func() {
defer stdin.Close()
io.WriteString(stdin, "values written to stdin are passed to cmd's standard input")
}()
out, err := cmd.CombinedOutput()
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s\n", out)
}
Though this question is a little old, but here is my answer:
This question is of course very platform specific as how standard IO is handled depends on the OS implementation and not on Go language. However, as general rule of thumb (due to some OSes being prevailing), "what you ask is not possible".
On most of modern operating systems you can pipe standard streams (as in #mraron's answer), you can detach them (this is how daemons work), but you cannot reassign or delegate them to another process.
I think this limitation is more because of security concern. There are still from time to time bugs being discovered that allow remote code execution, imagine if OS was allowing to reassign/delegate STDIN/OUT, then with such vulnerabilities the consequences would be disastrous.
While you cannot directly do this as #AlexKey wrote earlier still you can make some workarounds. If os prevents you to pipe your own standard streams who cares all you need 2 channels and 2 goroutines
var stdinChan chan []byte
var stdoutChan chan []byte
//when something happens in stdout of your code just pipe it to stdout chan
stdoutChan<-somehowGotDataFromStdOut
then you need two funcs as i mentioned before
func Pipein(){
for {
stdinFromProg.Write(<-stdInChan)
}
}
The same idea for the stdout