Print the stdout from exec command in real time in Go [duplicate] - go

This question already has an answer here:
How can I redirect the stdout and stderr of a command to both the console and a log file while outputting in real time?
(1 answer)
Closed 5 years ago.
I have a small Go tool which basically allows the user to define an command that than will be run using os/exec.
My problem is that I want to show the user the output (stdout/stderr) of the command.
An example could look like this:
The user defines a command that in the end is sh test.sh.
Content of test.sh:
echo "Start"
sleep 7s
echo "Done"
With my current implementation the user can only see the output once the complete command finished. In the example above the user wouldn't see the output Start until the sleep command and the second echo finish.
I currently retrieve the output of the command like this:
cmd := exec.Command(command, args...)
cmd.Dir = dir
// Attach to the standard out to read what the command might print
stdout, err := cmd.StdoutPipe()
if err != nil {
log.Panic(err)
}
// Execute the command
if err := cmd.Start(); err != nil {
log.Panic(err)
}
buf := new(bytes.Buffer)
buf.ReadFrom(stdout)
log.Print(buf.String())
Is it somehow possible to read the stdout/stderr in real-time. Meaning that as soon as the user defined command creates and output it is printed?

Thank you mh-cbon. That pushed me in the right direction.
The code now looks like this and does exactly what I want it to do. I also found that when I use Run() instead of Start() the execution of the program only continues after the command has finished.
cmd := exec.Command(command, args...)
cmd.Dir = dir
var stdBuffer bytes.Buffer
mw := io.MultiWriter(os.Stdout, &stdBuffer)
cmd.Stdout = mw
cmd.Stderr = mw
// Execute the command
if err := cmd.Run(); err != nil {
log.Panic(err)
}
log.Println(stdBuffer.String())

Related

Write file from exec.Command

I am trying to write a file from a bash command into a file in Go.
Note there are several reasons for using Go over bash here: I have some more logic such as parsing configuration files, I would like to run that code for multiple DBs in parallele and finally performing some more complex data manipulation after.
dumpStr := fmt.Sprintf("pg_dump -U %s -h %s %s | gzip", DbUserName, DbHost, DbName)
cmd := exec.Command("bash", "-c", dumpStr)
cmd.Env = append(cmd.Env, "PGPASSWORD="+DbPassword)
outfile, err := os.Create(DbName + ".gz")
if err != nil {
panic(err)
}
outfile = cmd.Stdout
defer outfile.Close()
err = cmd.Start()
if err != nil {
panic(err)
}
cmd.Wait()
However, I am getting an emtpy result.
I am getting data if I am executing dumpStr from the CLI but not from that code...
What am I missing?
As Flimzy said, you're not capturing the output of pg_dump. You can do that with Go, or you can use pg_dump-s --file. It can also compress with --compress so no need to pipe to gzip. Then there's no need for bash and you can avoid shell quoting issues.
cmd := exec.Command(
"pg_dump",
"--compress=9",
"--file="+DbName + ".gz",
"-U"+DbUserName,
"-h"+DbHost,
DbName,
)
log.Print("Running pg_dump...")
if err := cmd.Run(); err != nil {
log.Fatal(err)
}
Much simpler and more secure.
For illustration here's how you'd do it all in Go.
Use Cmd.StdoutPipe to get an open IO reader to pg_dump's stdout. Then use io.Copy to copy from stdout to your open file.
#Peter points out that since Cmd.Stdout is an io.Reader it's simpler to assign the open file to cmd.Stdout and let cmd write to it directly.
// Same as above, but no --file.
cmd := exec.Command(
"pg_dump",
"--compress=9",
"-U"+DbUserName,
"-h"+DbHost,
DbName,
)
// Open the output file
outfile, err := os.Create(DbName + ".gz")
if err != nil {
log.Fatal(err)
}
defer outfile.Close()
// Send stdout to the outfile. cmd.Stdout will take any io.Writer.
cmd.Stdout = outfile
// Start the command
if err = cmd.Start(); err != nil {
log.Fatal(err)
}
log.Print("Waiting for command to finish...")
// Wait for the command to finish.
if err = cmd.Wait(); err != nil {
log.Fatal(err)
}
In addition, you're only checking if the command started, not if it successfully ran.
From the docs for Cmd.Start.
Start starts the specified command but does not wait for it to complete.
The Wait method will return the exit code and release associated resources once the command exits.
You're checking cmd.Start for an error, but not cmd.Wait. Checking the error from cmd.Start only means the command started. If there is an error while the program is running you won't know what it is.
You need to actually use the output of your command. You're not doing that. To do so, use the StdoutPipe method, then you can copy the stdout from your program, into your file.

Start detached command with redirect to file

I'm trying to start a command in a detached process so that it can continue after go program exits. I need to redirect the output of the command to a file.
What I need is something like this:
func main() {
command := exec.Command("/tmp/test.sh", ">", "/tmp/out")
if err := command.Start(); err != nil {
fmt.Fprintln(os.Stderr, "Command failed.", err)
os.Exit(1)
}
fmt.Println("Process ID:", command.Process.Pid)
}
Obviously such redirect doesn't work. As I immediately exit from the program after starting the long running command, I cannot open a file and bind it to the Stdout.
Is there any way to achieve such a redirect?
You may start a shell which executes your command / app, and you may redirect its output to a file. The shell will continue to run and execute your script / app even if your Go app exits.
Example:
cmd := exec.Command("sh", "-c", "/tmp/test.sh > /tmp/out")
if err := cmd.Start(); err != nil {
panic(err)
}
fmt.Println("Process ID:", cmd.Process.Pid)
Test it with this simple Go app (replace /tmp/test.sh with the name of the executable binary you compile this into):
package main
import ("fmt"; "time")
func main() {
for i := 0; i < 10; i++ {
fmt.Printf("%d.: %v\n", i, time.Now())
time.Sleep(time.Second)
}
}
This app simply prints a line to the standard output once every second. You can see how the output file is being written e.g. with tail -f /tmp/out.
Note that you may use other shells to execute your scripts to your liking (and to what the test.sh script dictates).
For example to use bash:
cmd := exec.Command("/bin/bash", "-c", "/tmp/test.sh > /tmp/out")
// rest is unchanged
Note that the command to be executed by the shell is passed as a single string argument, and it is not broken down into multiple as you would do it if you were to execute it directly in the command prompt.
Maybe you can try to use this: https://stackoverflow.com/a/28918814/2728768
Opening a file (and os.File implements io.Writer), and then passing it as the command.Stdout could do the trick:
func main() {
command := exec.Command("./tmp/test.sh")
f, err := os.OpenFile("/tmp/out", os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
if err != nil {
fmt.Printf("error opening file: %v", err)
}
defer f.Close()
// On this line you're going to redirect the output to a file
command.Stdout = f
if err := command.Start(); err != nil {
fmt.Fprintln(os.Stderr, "Command failed.", err)
os.Exit(1)
}
fmt.Println("Process ID:", command.Process.Pid)
}
Not sure this could be a viable solution for your case. I've tried it locally and it seems working... remember that your user should be able to create/update the file.

How to capture/log everything after spawning an interactive program

I have a method that can spawn an interactive process, now how do I log everything (including stdin and stdout) after spawning ?
e.g.,
func execute(cmd1 string, slice []string) {
cmd := exec.Command(cmd1, slice...)
// redirect the output to terminal
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
cmd.Stdin = os.Stdin
cmd.Run()
}
..
The interactive program could be :
execute(ftp)
I think I have to dup stdin, stdout and read write in separate thread.
Rather than redirecting it's output to the terminal read it and then you can log/print do whatever you want with it.
stdout, err := cmd.StdoutPipe()
b, _ := ioutil.ReadAll(stdout)
fmt.Println(string(b))
Something like the code above would work though there are many options. I think you'll want to remove all that code you have to redirect to the terminal.
you could store the output in a temporary buffer and write it to several places
outBuf := bytes.Buffer{}
cmd := exec.Command(cmd1, slice...)
cmd.Stdout = &outBuf
cmd.Run()
if outBuf.Len() > 0 {
log.Printf("%s", outBuf.String())
fmt.Fprintf(os.Stdout, "%s", outBuf.String())
}

How to print the realtime output of running child process in go?

I have the following bash script bash_loop.shthat prints 1 to 10 and sleep 3 s in between.
#!/bin/bash
# Basic while loop
counter=1
while [ $counter -le 10 ]
do
echo $counter
((counter++))
sleep 3
done
echo All done
Right now, I have my go code as follow:
burstingScript := "bash_loop.sh"
cmd := exec.Command("/bin/sh", burstingScript)
var out bytes.Buffer
cmd.Stdout = &out
if err := cmd.Run(); err != nil {
fmt.Println("An error has occurred..")
log.Fatal(err)
}
fmt.Println(out.String())
However, this only print out everything after the cmd finish running after 30s, instead of print things as they become available.
So my question is that if I can print each number while the bash script is still running instead of printing everything all together after the bash script finishes execution.
P.S.1: In the real use case, I have to process the output of bash script in realtime, instead of simply printing things out to os.Stdout, so I am wondering if there's any command poll() interface or equivalence in go.
P.S.2:In the real use case, I want to detach from the child process as soon as I find interesting message. For example, after I read 3, I want my function return 3 immediately and not wait for the rest of output anymore, though I still want the child process (the bash script) itself to be up and running.
P.S.3: In python, I would do something like this
cmd = "./bash_loop.sh"
p = subprocess.Popen(cmd, shell=True, stderr=subprocess.PIPE, stdout=subprocess.PIPE)
result = {}
while True:
out = p.stdout.readline()
if out == '' and p.poll() != None:
break
if out != '':
#process(out)
sys.stdout.write(out)
sys.stdout.flush()
P.S.4: Now I have evolved my go snippet to follow. Will the child become a zombie if I return before the command finishes running?
burstingScript := path.Join(rootDir, "bash_loop.sh")
cmd := exec.Command("/bin/sh", burstingScript)
stdout, _ := cmd.StdoutPipe()
cmd.Start()
scanner := bufio.NewScanner(stdout)
for scanner.Scan() {
m := scanner.Text()
fmt.Println(m)
//if m=="3" {
// return process(m)
//}
}
cmd.Wait()
Set the command output to stdout:
cmd.Stdout = os.Stdout

Executing web2exe from Golang is giving me 'exit status 2'

Im trying the following, to use go to bundle a folder of html files using the CMD web2exe.
cmd := exec.Command("web2exe-win.exe", "html-folder --main index.html --export- to windows-x32 --output-dir")
var out bytes.Buffer
cmd.Stdout = &out
err := cmd.Run()
if err != nil {
fmt.Println(err)
}
fmt.Println(out)
When a program exits non-zero it means that it could not run successfully and typically it has written an error message to STDERR (or STDOUT). You should somehow capture or print the output streams so you can inspect them for error messages. For example:
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
Note also that your command line arguments should be separate array elements (instead of space separated elements in a single string as they are now):
cmd := exec.Command("web2exe-win.exe", "html-folder", "--main", "index.html", "--export-to", "windows-x32", "--output-dir")

Resources