I need to find a way to read a line from a io.ReadCloser object OR find a way to split a byte array on a "end line" symbol. However I don't know the end line symbol and I can't find it.
My application execs a php script and needs to get the live output from the script and do "something" with it when it gets it.
Here's a small piece of my code:
cmd := exec.Command(prog, args)
/* cmd := exec.Command("ls")*/
out, err := cmd.StdoutPipe()
if err != nil {
fmt.Println(err)
}
err = cmd.Start()
if err != nil {
fmt.Println(err)
}
after this I monitor the out buffer in a go routine. I've tried 2 ways.
1) nr, er := out.Read(buf) where buf is a byte array. the problem here is that I need to brake the array for each new line
2) my second option is to create a new bufio.reader
r := bufio.NewReader(out)
line,_,e := r.ReadLine()
it runs fine if I exec a command like ls, I get the output line by line, but if I exec a php script it immediately get an End Of File error and exits(I'm guessing that's because of the delayed output from php)
EDIT: My problem was I was creating the bufio.Reader inside the go routine whereas if I do it right after the StdoutPipe() like minikomi suggested, it works fine
You can create a reader using bufio, and then read until the next line break character (Note, single quotes to denote character!):
stdout, err := cmd.StdoutPipe()
rd := bufio.NewReader(stdout)
if err := cmd.Start(); err != nil {
log.Fatal("Buffer Error:", err)
}
for {
str, err := rd.ReadString('\n')
if err != nil {
log.Fatal("Read Error:", err)
return
}
fmt.Println(str)
}
If you're trying to read from the reader in a goroutine with nothing to stop the script, it will exit.
Another option is bufio.NewScanner:
package main
import (
"bufio"
"os/exec"
)
func main() {
cmd := exec.Command("go", "env")
out, err := cmd.StdoutPipe()
if err != nil {
panic(err)
}
buf := bufio.NewScanner(out)
cmd.Start()
defer cmd.Wait()
for buf.Scan() {
println(buf.Text())
}
}
https://golang.org/pkg/bufio#NewScanner
Related
In Go, I would like to execute a binary from within my application and continually read what the command prints to stdout. However, the one caveat is that the binary is programmed to execute its task infinitely until it reads the enter key, and I don't have access to the binary's source code. If I execute the binary directly from a terminal, it behaves correctly. However, if I execute the binary from within my application, it somehow thinks that it reads the enter key, and closes almost immediately. Here is a code snippet demonstrating how I'm trying to execute the binary, pipe it's stdout, and print it to the screen:
func main() {
// The binary that I want to execute.
cmd := exec.Command("/usr/lib/demoApp")
// Pipe the command's output.
stdout, err := cmd.StdoutPipe()
if err != nil {
fmt.Println(err)
}
stdoutReader := bufio.NewReader(stdout)
// Start the command.
err = cmd.Start()
if err != nil {
fmt.Println(err)
}
// Read and print the command's output.
buff := make([]byte, 1024)
var n int
for err == nil {
n, err = stdoutReader.Read(buff)
if n > 0 {
fmt.Printf(string(buff[0:n]))
}
}
_ = cmd.Wait()
}
Any ideas if what I'm trying to accomplish is possible?
As #mgagnon mentioned, your problem might lie somewhere else; like perhaps the external dependency just bails due to not running in a terminal. Using following to simulate demoApp:
func main() {
fmt.Println("Press enter to exit")
// Every second, report fake progress
go func() {
for {
fmt.Print("Doing stuff...\n")
time.Sleep(time.Second)
}
}()
for {
// Read single character and if enter, exit.
consoleReader := bufio.NewReaderSize(os.Stdin, 1)
input, _ := consoleReader.ReadByte()
// Enter = 10 | 13 (LF or CR)
if input == 10 || input == 13 {
fmt.Println("Exiting...")
os.Exit(0)
}
}
}
... this works fine for me:
func main() {
cmd := exec.Command("demoApp.exe")
stdout, err := cmd.StdoutPipe()
if err != nil {
panic(err)
}
stdin, err := cmd.StdinPipe()
if err != nil {
log.Fatal(err)
}
go func() {
defer stdin.Close()
// After 3 seconds of running, send newline to cause program to exit.
time.Sleep(time.Second * 3)
io.WriteString(stdin, "\n")
}()
cmd.Start()
// Scan and print command's stdout
scanner := bufio.NewScanner(stdout)
for scanner.Scan() {
fmt.Println(scanner.Text())
}
// Wait for program to exit.
cmd.Wait()
}
$ go run main.go
Press enter to exit
Doing stuff...
Doing stuff...
Doing stuff...
Exiting...
The only difference between this and your code is that I'm using stdin to send a newline after 3 seconds to terminate the cmd. Also using scanner for brevity.
Using this as my /usr/lib/demoApp:
package main
import (
"fmt"
"time"
)
func main() {
for {
fmt.Print("North East South West")
time.Sleep(time.Second)
}
}
This program works as expected:
package main
import (
"os"
"os/exec"
)
func main() {
cmd := exec.Command("demoApp")
stdout, err := cmd.StdoutPipe()
if err != nil {
panic(err)
}
cmd.Start()
defer cmd.Wait()
for {
var b [1024]byte
stdout.Read(b[:])
os.Stdout.Write(b[:])
}
}
I've been trying to write a program that record what is passed to a subprocess and the console returns in live (in the future, to record SSH sessions, for now on Python shell for testing)
I can record without issue stdout and stderr (it shows and record it correctly) but I can't find a way to do the same on stdin ?
Basically that my stdin will both map to the subprocess stdin and write to the log file.
There is my current code :
func SSH(cmd *cobra.Command, args []string) {
logFile := fmt.Sprintf("%v#%s.log", args[0], time.Now().Format(SSHLogDateFormat))
usr, _ := user.Current()
home := usr.HomeDir
logDir := fmt.Sprintf("%s/%s/logs", home, config.ConfigDir)
if _, err := os.Stat(logDir); os.IsNotExist(err) {
err = os.Mkdir(logDir, os.FileMode(int(0700)))
if err != nil {
log.Fatalf("Failed to create %s: %s", logDir, err)
}
}
fullLogFile := fmt.Sprintf("%s/%s", logDir, logFile)
log.Infof("Started recording to %s", fullLogFile)
bash, err := exec.LookPath("bash")
if err != nil {
log.Errorf("Could not locate bash: %v", err)
}
f, err := os.Create(fullLogFile)
if err != nil {
log.Fatalf("Failed to open device logs: %s", err)
}
command := exec.Command(bash, "-c", "python")
out := io.MultiWriter(os.Stdout, f)
command.Stderr = out
command.Stdout = out
if err := command.Start(); nil != err {
log.Fatalf("Error starting program: %s, %s", command.Path, err.Error())
}
err = command.Wait()
if err != nil {
log.Fatalf("Error waiting program: %s, %s", command.Path, err.Error())
}
f.Close()
log.Infof("Finished recording to %s", fullLogFile)
}
Tried this too without success :
out := io.MultiWriter(os.Stdout, f)
in := io.TeeReader(os.Stdin, out)
command.Stderr = out
command.Stdout = out
command.Stdin = in
You need to write to the process's stdin. Get a write pipe to that:
procIn, err := command.StdinPipe()
if nil!=err {
log.Fatal(err)
}
Then create a multiWriter to write to both log and process:
inWriter := io.MultiWriter(procIn, f)
Finally, copy Stdin into the MultiWriter:
go func() {
io.Copy(inWriter, os.Stdin)
procIn.Close()
}()
We do the copy in a goroutine, so as not to hang everything up: we haven't started the command yet, so there's nothing receiving the written bytes. It needs to occur in parallel to the command running.
Here's a very simple example:
package main
import (
`os`
`os/exec`
`io`
)
// pipeto copies stdin to logOut and to the command,
// and copies the commands stdout and stderr to logOut and
// to our stderr.
func pipeto(logOut os.Writer, cmd string, args ...string) error {
cmd := exec.Command(cmd, args...)
out := io.MultiWriter(os.Stdout, logOut)
cmd.Stderr, cmd.Stdout = out, out
procIn, err := cmd.StdinPipe()
if nil!=err {
return err
}
go func() {
io.Copy( io.MultiWriter(procIn, logOut) , os.Stdin )
procIn.Close()
}()
return cmd.Run()
}
func main() {
logOut, err := os.Create(`logout.log`)
if nil!=err {
panic(err)
}
defer logOut.Close()
if err := pipeto(logOut, `sed`, `s/tea/coffee/g`); nil!=err {
panic(err)
}
}
You can test it, where I've named my go file pipetest.go:
echo this is a test of tea | go run pipetest.go
The you will see both the input and the output reflected in logout.log:
this is a test of tea
this is a test of coffee
At the end I found the solution by using the PTY library (That would have been needed anyway to handle special signals and tabs on subprocesses): https://github.com/creack/pty
I took the Shell example and just replaced the io.Copy with my MultiWriter
I'm trying to run a fairly simple bash command from my Go code. My program writes out an IPTables config file and I need to issue a command to make IPTables refresh from this config. This is very straightforward at the commandline:
/sbin/iptables-restore < /etc/iptables.conf
However, I can't for the life of me figure out how to issue this command with exec.Command(). I tried a few things to accomplish this:
cmd := exec.Command("/sbin/iptables-restore", "<", "/etc/iptables.conf")
// And also
cmd := exec.Command("/sbin/iptables-restore", "< /etc/iptables.conf")
No surprise, neither of those worked. I also tried to feed the filename into the command by piping in the file name to stdin:
cmd := exec.Command("/sbin/iptables-restore")
stdin, err := cmd.StdinPipe()
if err != nil {
log.Fatal(err)
}
err = cmd.Start()
if err != nil {
log.Fatal(err)
}
io.WriteString(stdin, "/etc/iptables.conf")
That doesn't work either, no surprise. I can use stdin to pipe in the contents of the file, but this seems silly when I can just tell iptables-restore what data to go read. So how might I get Go to run the command /sbin/iptables-restore < /etc/iptables.conf?
first read this /etc/iptables.conf file content then write it to cmd.StdinPipe() like this:
package main
import (
"io"
"io/ioutil"
"log"
"os/exec"
)
func main() {
bytes, err := ioutil.ReadFile("/etc/iptables.conf")
if err != nil {
log.Fatal(err)
}
cmd := exec.Command("/sbin/iptables-restore")
stdin, err := cmd.StdinPipe()
if err != nil {
log.Fatal(err)
}
err = cmd.Start()
if err != nil {
log.Fatal(err)
}
_, err = io.WriteString(stdin, string(bytes))
if err != nil {
log.Fatal(err)
}
}
cmd := exec.Command("/usr/sbin/iptables-restore", "--binary", iptablesFilePath)
_, err := cmd.CombinedOutput()
if err != nil {
return err
}
return nil
this work fine on my Raspberry Pi3
The os/exec package does not invoke the system shell, nor does it implement the < redirection syntax typically handled by a shell.
Open the input file and use that file as stdin:
stdin, err := os.Open("/etc/iptables.conf")
if err != nil {
log.Fatal(err)
}
defer stdin.Close()
cmd := exec.Command("/sbin/iptables-restore")
cmd.Stdin = stdin // <-- use open file as stdin
result, err := cmd.CombinedOutput()
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s\n", result)
I have an app called "myapp". That app simply writes to stderr.
The important bit is, I want to capture what is written in stderr and process it in real-time. How would I go about doing that?
I tried the code below. :
cmd := exec.Command("myapp") // this app prints lines to stderr
stderr, err := cmd.StderrPipe()
if err != nil {
log.Fatal(err)
}
if err := cmd.Start(); err != nil {
log.Fatal(err)
}
if b, err := ioutil.ReadAll(stderr); err == nil {
log.Println(string(b))
}
if err := cmd.Wait(); err != nil {
log.Fatal(err)
}
The code doesn't print out anyting. I suspect it's because ioutil.ReadAll() is not the proper func to call since it waits for EOF. How else would I read from the stderr pipe?
You can replace the command executed with anything that outputs to stdout or stderr like tail -f mylogfile. The point is, I want to process the lines as they are written to stdout.
StderrPipe returns a ReadCloser. You can use that to create a bufio.Scanner and then read lines one by one:
sc := bufio.NewScanner(stderr)
for sc.Scan() {
fmt.Printf("Line: %s\n", sc.Text());
}
Create a type that implements io.Writer and set that as the command's stderr writer.
type Processor struct{}
func (Processor) Write(b []byte) (int, error) {
// intercept data here
return os.Stdout.Write(b)
}
func main() {
cmd := exec.Command("mycommand")
cmd.Stderr = Processor{}
_ = cmd.Run()
}
Let's say I want to run 'ls' in a go program, and store the results in a string. There seems to be a few commands to fork processes in the exec and os packages, but they require file arguments for stdout, etc. Is there a way to get the output as a string?
There is an easier way now:
package main
import (
"fmt"
"log"
"os/exec"
)
func main() {
out, err := exec.Command("date").Output()
if err != nil {
log.Fatal(err)
}
fmt.Printf("The date is %s\n", out)
}
Where out is the standard output. It's in the format []byte, but you can change it to string easily with:
string(out)
You can also use CombinedOutput() instead of Output() which returns standard output and standard error.
exec.Command
To get both stdout and stderr into separate strings, you can use byte buffers like so:
cmd := exec.Command("date")
var outb, errb bytes.Buffer
cmd.Stdout = &outb
cmd.Stderr = &errb
err := cmd.Run()
if err != nil {
log.Fatal(err)
}
fmt.Println("out:", outb.String(), "err:", errb.String())
cmd := exec.Command("ls", "-al")
output, _ := cmd.CombinedOutput()
fmt.Println(string(output))
or
cmd := exec.Command(name, arg...)
stdout, err := cmd.StdoutPipe()
cmd.Stderr = cmd.Stdout
if err != nil {
return err
}
if err = cmd.Start(); err != nil {
return err
}
for {
tmp := make([]byte, 1024)
_, err := stdout.Read(tmp)
fmt.Print(string(tmp))
if err != nil {
break
}
}
I used this with a recent version of GO (~1.11)
// CmdExec Execute a command
func CmdExec(args ...string) (string, error) {
baseCmd := args[0]
cmdArgs := args[1:]
log.Debugf("Exec: %v", args)
cmd := exec.Command(baseCmd, cmdArgs...)
out, err := cmd.Output()
if err != nil {
return "", err
}
return string(out), nil
}
// Usage:
// out, err := CmdExec("ls", "/home")
Two options, depending on the paradigm you prefer:
os.ForkExec()
exec.Run()
Use exec.Run, passing Pipe for stdout. Read from the pipe that it returns.
If you are wanting string output, strings.Builder is more efficient [1] than
bytes.Buffer:
package main
import (
"os/exec"
"strings"
)
func main() {
c, b := exec.Command("go", "version"), new(strings.Builder)
c.Stdout = b
c.Run()
print(b.String())
}
https://golang.org/pkg/bytes#Buffer.String
Edit: This answer is obsolete. Please see Fatih Arslan's answer below.
Use exec.Run by specifying Pipe as the stdout (and stderr if you want). It will return cmd, which contains an os.File in the Stdout (and Stderr) fields. Then you can read it using for example ioutil.ReadAll.
Example:
package main
import (
"exec";
"io/ioutil";
)
func main() {
if cmd, e := exec.Run("/bin/ls", nil, nil, exec.DevNull, exec.Pipe, exec.MergeWithStdout); e == nil {
b, _ := ioutil.ReadAll(cmd.Stdout)
println("output: " + string(b))
}
}