So I have the simple app which starts other apps and reads their output.
package main
import (
"bufio"
"io"
"log"
"os/exec"
"time"
)
func main() {
cmd := exec.Command("perl", "-e", "my $x = 0; while (1) { print ++$x.qx'date'; sleep 1; }")
stdout, _ := cmd.StdoutPipe()
stderr, _ := cmd.StderrPipe()
in := bufio.NewReaderSize(io.MultiReader(stdout, stderr), 100)
cmd.Start()
defer cmd.Wait()
for {
log.Printf("....")
time.Sleep(1 * time.Second)
l, _ := in.ReadString('\n')
log.Printf(string(l))
}
}
The point of the real app is to read output of running process and parse it.. however it doesn't works well with real apps which don't explicitly sync/flush their stdout (it takes ~60 lines of iperf output before starting printing).
What is the most efficient way to read output byte-by-byte?
Related
I have a somewhat challenging situation where I need to write into a system command stdin the same stdout it outputs (in another running program), here's an example program that represents what I mean:
package main
import (
"bufio"
"fmt"
"math/rand"
"os"
)
func main() {
rand.Seed(time.Now().Unix())
var greetings []string = []string{"hi", "hola", "bonjour", "hallo", "whats up"}
var greeting string = greetings[rand.Intn(len(greetings))]
fmt.Println(greeting)
reader := bufio.NewReader(os.Stdin)
message, _ := reader.ReadString('\n')
if message == greeting+"\n" {
fmt.Println("nice to meet you!")
} else {
fmt.Println("oops!")
}
}
Since you greet with a random greeting, you have to read the stdout, send it to stdin and also capture if it was the correct answer or not. I've tried with stdinpipes but it freezes waiting for the stdin close since I think that only works for the start of the command run only, so for a running program it hasn't been working for me...
I appreciate any help!
EDIT
I wanted to add sort of what I was trying to do, I've tried without channels as well but it didn't seem to make a difference on the outcome, it just freezes waiting for stdin to close and I need to get first stdout before closing stdin since it consists of it:
package main
import (
"io"
"os/exec"
)
func main() {
cmd := exec.Command("./executable_program")
stdout, _ := cmd.StdoutPipe()
stdin, _ := cmd.StdinPipe()
var c chan []byte = make(chan []byte)
cmd.Start()
go func() {
b, _ := io.ReadAll(stdout)
c <- b
}()
stdin.Write(<-c)
stdin.Close()
cmd.Wait()
}
You can use a pipe to join the stdout to the stdin of the program that you execute:
package main
import (
"io"
"os/exec"
)
func main() {
r, w := io.Pipe()
cmd := exec.Command("<name-of-program-to-run>")
cmd.Stdin = r
cmd.Stdout = w
cmd.Run()
}
To see this in action, first let's prepare a test program to be executed by the program above. This test program simply prints a line to stdout, and then reads each line of stdin and prints it to stdout until stdin is closed.
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
fmt.Fprint(os.Stdout, "priming the pump!\n")
s := bufio.NewScanner(os.Stdin)
for s.Scan() {
line := s.Text()
fmt.Fprint(os.Stdout, line+"\n")
}
}
Then, we modify our initial program to print the bytes traversing through the pipe so we see what's going on.
package main
import (
"fmt"
"io"
"os/exec"
)
func main() {
r, w := io.Pipe()
sr := &readSpy{r: r}
wr := &writeSpy{w: w}
cmd := exec.Command("./test-program")
cmd.Stdin = sr
cmd.Stdout = wr
cmd.Run()
}
type readSpy struct {
r io.Reader
}
func (s *readSpy) Read(d []byte) (int, error) {
size, err := s.r.Read(d)
fmt.Println("readSpy read", string(d[:size]))
return size, err
}
type writeSpy struct {
w io.Writer
}
func (s *writeSpy) Write(d []byte) (int, error) {
size, err := s.w.Write(d)
fmt.Println("writeSpy wrote", string(d[:size]))
return size, err
}
Running the above, you will see the following getting printed in a infinite loop, which makes sense since the priming the pump! string is printed to stdout and fed right back to the stdin of the test program:
writeSpy wrote priming the pump!
readSpy read priming the pump!
...repeated forever...
I'm running a command in Go via exec.Command and scanning the output. On some systems the output is immediate. But on some systems the output seems to be buffered. Unless the amount of data produced by the command is large enough, I don't actually receive the output.
Is there anyway to get more immediate output, reliably?
package main
import (
"fmt"
"log"
"os/exec"
"time"
)
func main() {
cmd := exec.Command("udevadm", "monitor")
stdout, err := cmd.StdoutPipe()
if err != nil {
log.Fatal(err)
}
err = cmd.Start()
if err != nil {
log.Fatal(err)
}
for {
p := make([]byte, 10)
n, _ := stdout.Read(p)
fmt.Println("# ", time.Now().Unix(), " ", n)
}
}
I will propose that running stdbuf -oL udevadm <args> will achieve effectively what i"m after (line buffered output).
The following works and prints the command output:
out, err := exec.Command("ps", "cax").Output()
but this one fails (with exit status 1):
out, err := exec.Command("ps", "cax | grep myapp").Output()
Any suggestions?
Passing everything to bash works, but here's a more idiomatic way of doing it.
package main
import (
"fmt"
"os/exec"
)
func main() {
grep := exec.Command("grep", "redis")
ps := exec.Command("ps", "cax")
// Get ps's stdout and attach it to grep's stdin.
pipe, _ := ps.StdoutPipe()
defer pipe.Close()
grep.Stdin = pipe
// Run ps first.
ps.Start()
// Run and get the output of grep.
res, _ := grep.Output()
fmt.Println(string(res))
}
You could do:
out, err := exec.Command("bash", "-c", "ps cax | grep myapp").Output()
In this specific case, you don't really need a pipe, a Go can grep as well:
package main
import (
"bufio"
"bytes"
"os/exec"
"strings"
)
func main() {
c, b := exec.Command("go", "env"), new(bytes.Buffer)
c.Stdout = b
c.Run()
s := bufio.NewScanner(b)
for s.Scan() {
if strings.Contains(s.Text(), "CACHE") {
println(s.Text())
}
}
}
I'm trying to read and write to an exec.Command, but I'm struggling with piping.
This first one is a example of the behaviour I want. The go application just proxies stdin, stdout and stderr to the command:
package main
import (
"fmt"
"os"
"os/exec"
)
func main() {
cmd := exec.Command("bash", "-i")
cmd.Stdin = os.Stdin
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
cmd.Run()
fmt.Println("Done")
}
Running this is like:
pensarando#home: ~/go/src/test$ ./test
pensarando#home: ~/go/src/test$ echo "hello"
hello
pensarando#home: ~/go/src/test$ exit
Done
pensarando#home: ~/go/src/test$
But this one does not work at all:
package main
import (
"fmt"
"io"
"os"
"os/exec"
"sync"
)
func main() {
cmd := exec.Command("bash", "-i")
in, _ := cmd.StdinPipe()
out, _ := cmd.StdoutPipe()
err, _ := cmd.StderrPipe()
exit := make(chan struct{})
done := func() {
in.Close()
out.Close()
err.Close()
cmd.Wait()
close(exit)
}
var once sync.Once
go func() {
io.Copy(os.Stdout, out)
fmt.Println("done stdout")
once.Do(done)
}()
go func() {
io.Copy(in, os.Stdin)
fmt.Println("done stdin")
once.Do(done)
}()
go func() {
io.Copy(os.Stderr, err)
fmt.Println("done stderr")
once.Do(done)
}()
cmd.Start()
<-exit
fmt.Println("Done")
}
When I run this one from the terminal, the output is for example:
pensarando#home:~/go/src/test$ go build
pensarando#home:~/go/src/test$ ./test
pensarando#home:~/go/src/test$ echo "Hello"
[1]+ Stopped ./test
pensarando#home:~/go/src/test$ echo "Hello"
Hello
pensarando#home:~/go/src/test$ fg
./test
done stdin
done stdout
done stderr
Done
pensarando#home:~/go/src/test$
What exactly happens here?
I have a bunch of systems commands which are somwhat similar to appending new content to a file. I wrote a simple script to execute system commands, which works well if there are single words like 'ls' , 'date' etc. But if the command is greater than that, program dies.
The following is the code
package main
import (
"fmt"
"os/exec"
"sync"
)
func exe_cmd(cmd string, wg *sync.WaitGroup) {
fmt.Println(cmd)
c = cmd.Str
out, err := exec.Command(cmd).Output()
if err != nil {
fmt.Println("error occured")
fmt.Printf("%s", err)
}
fmt.Printf("%s", out)
wg.Done()
}
func main() {
wg := new(sync.WaitGroup)
wg.Add(3)
x := []string{"echo newline >> foo.o", "echo newline >> f1.o", "echo newline >> f2.o"}
go exe_cmd(x[0], wg)
go exe_cmd(x[1], wg)
go exe_cmd(x[2], wg)
wg.Wait()
}
The following is the error i see
exec: "echo newline >> foo.o": executable file not found in $PATHexec:
"echo newline >> f2.o": executable file not found in $PATHexec:
"echo newline >> f1.o": executable file not found in $PATH
I guess, this may be due to, not sending cmds and arguments seperately ( http://golang.org/pkg/os/exec/#Command ). I am wondering how to subvert this, since I don't know how many arguments will be there in my command which needs to be executed.
I found a relatively decent way to achieve the same .
out, err := exec.Command("sh","-c",cmd).Output()
Works for me until now. Still finding better ways to achieve the same.
Edit1:
Finally a easier and efficient (atleast so far) way to do would be like this
func exeCmd(cmd string, wg *sync.WaitGroup) {
fmt.Println("command is ",cmd)
// splitting head => g++ parts => rest of the command
parts := strings.Fields(cmd)
head := parts[0]
parts = parts[1:len(parts)]
out, err := exec.Command(head,parts...).Output()
if err != nil {
fmt.Printf("%s", err)
}
fmt.Printf("%s", out)
wg.Done() // Need to signal to waitgroup that this goroutine is done
}
Thanks to variadic arguments in go and people that pointed that out to me :)
For exec.Command() the first argument needs to be the path to the executable. Then the remaining arguments will be supplied as arguments to the executable. Use strings.Fields() to help split the word into a []string.
Example:
package main
import (
"fmt"
"os/exec"
"sync"
"strings"
)
func exe_cmd(cmd string, wg *sync.WaitGroup) {
fmt.Println(cmd)
parts := strings.Fields(cmd)
out, err := exec.Command(parts[0],parts[1]).Output()
if err != nil {
fmt.Println("error occured")
fmt.Printf("%s", err)
}
fmt.Printf("%s", out)
wg.Done()
}
func main() {
wg := new(sync.WaitGroup)
commands := []string{"echo newline >> foo.o", "echo newline >> f1.o", "echo newline >> f2.o"}
for _, str := range commands {
wg.Add(1)
go exe_cmd(str, wg)
}
wg.Wait()
}
Here's an alternative approach that just writes all the commands to a file then executes that file within the context of the new created output directory.
Example 2
package main
import (
"os"
"os/exec"
"fmt"
"strings"
"path/filepath"
)
var (
output_path = filepath.Join("./output")
bash_script = filepath.Join( "_script.sh" )
)
func checkError( e error){
if e != nil {
panic(e)
}
}
func exe_cmd(cmds []string) {
os.RemoveAll(output_path)
err := os.MkdirAll( output_path, os.ModePerm|os.ModeDir )
checkError(err)
file, err := os.Create( filepath.Join(output_path, bash_script))
checkError(err)
defer file.Close()
file.WriteString("#!/bin/sh\n")
file.WriteString( strings.Join(cmds, "\n"))
err = os.Chdir(output_path)
checkError(err)
out, err := exec.Command("sh", bash_script).Output()
checkError(err)
fmt.Println(string(out))
}
func main() {
commands := []string{
"echo newline >> foo.o",
"echo newline >> f1.o",
"echo newline >> f2.o",
}
exe_cmd(commands)
}
out, _ := exec.Command("sh", "-c", "date +\"%Y-%m-%d %H:%M:%S %Z\"").Output()
exec.Command("sh","-c","ls -al -t | grep go >>test.txt").Output()
fmt.Printf("%s\n\n",out)
Tested couple cases and all work good. This is a lifesaver if you are dealing with quick shell commands in your program. Not tested with complex cases.