This question is similar to Golang - Copy Exec output to Log except it is concerned with the buffering of output from exec commands.
I have the following test program:
package main
import (
"fmt"
"log"
"os/exec"
)
func main() {
cmd := exec.Command("python", "inf_loop.py")
var out outstream
cmd.Stdout = out
if err := cmd.Start(); err != nil {
log.Fatal(err)
}
fmt.Println(cmd.Wait())
}
type outstream struct{}
func (out outstream) Write(p []byte) (int, error) {
fmt.Println(string(p))
return len(p), nil
}
inf_loop.py, which the above refers to, simply contains:
print "hello"
while True:
pass
The go program hangs when I run it and doesn't output anything, but if I use os.Stdout instead of out then it outputs "hello" before it hangs. Why is there a discrepancy between the two io.Writers and how can it be fixed?
Some more diagnostic information:
When the loop is removed from inf_loop.py then "hello" is output from both programs, as expected.
When using yes as the program instead of the python script and outputting len(p) in outstream.Write then there is output, and the output is usually 16384 or 32768. This indicates to me that this is a buffering issue, as I originally anticipated, but I still don't understand why the outstream structure is being blocked by buffering but os.Stdout isn't. One possibility is that the behaviour is the result of the way that exec passes the io.Writer directly to os.StartProcess if it is an os.File (see source for details), otherwise it creates an os.Pipe() between the process and the io.Writer, and this pipe may be causing the buffering. However, the operation and possible buffering of os.Pipe() is too low-level for me to investigate.
Python buffers stdout by default. Try this program:
import sys
print "hello"
sys.stdout.flush()
while True:
pass
or run Python with unbuffered stdout and stderr:
cmd := exec.Command("python", "-u", "foo.py")
Note the -u flag.
You see different results when using cmd.Stout = os.Stdout because Python uses line buffering when stdout is a terminal.
Related
Good evening,
I'm working on converting some tools written in python to Go in order to better understand it.
I need the program to call an external .exe with some arguments in order for it to correctly format some data. In the windows shell I can do C:\path_to_exe\file.exe arg1 arg2 "C:\path_to_output\output.txt"
I believe the correct method to do this in Go would be using exec.Command, but I'm not getting any...meaningful results.
out, err := exec.Command("cmd", "C:\\path\\tools\\util\\Utility.exe C:\\file_Location \"select * from TABLE\" C:\\output_path\\output.txt").Output()
fmt.Printf("\n", string(out))
if err != nil {
println(" Error running decomp ", err)
}
This appears to be running command, as the output I am receiving is:
%!(EXTRA string=Microsoft Windows [Version 10.0.22000.739]
(c) Microsoft Corporation. All rights reserved.
Process finished with the exit code 0
Just for giggles I tried breaking up the arguments, but the same result was achieved
out, err := exec.Command("cmd", exPath, utilPath, statement, textOutputPath+"test.txt").Output()
I'm expecting the executed program to run, parse the correct file based on the input, and output the specified txt file. I am left with no .txt file, and the go program runs much faster then the parsing should take.
There must be something I'm missing, could someone please provide some insight on the correct usage of exec.Command, because every example I can find appears to show that this should work.
Why are you spawning cmd.exe and having it run your utility.exe?
You can just spawn utility on its own.
For instance, suppose you have two binaries, hello and say-hello living in the same directory, compiled from
hello.go → hello:
package main
import (
"fmt"
"os"
)
func main() {
argv := os.Args[1:]
if len(argv) == 0 {
argv = []string{"world"}
}
for _, arg := range argv {
fmt.Printf("Hello, %s!\n", arg)
}
}
say-hello.go → say-hello:
package main
import (
"fmt"
"os"
"os/exec"
)
func main() {
process := exec.Command("./hello", os.Args[1:]...)
process.Stdin = os.Stdin
process.Stdout = os.Stdout
process.Stderr = os.Stderr
if err := process.Run(); err != nil {
fmt.Printf("Command failed with exit code %d\n", process.ProcessState.ExitCode())
fmt.Println(err)
}
}
You can then run the command:
$ ./say-hello Arawn Gywdion Sarah Hannah
And get back the expected
Hello, Arawn!
Hello, Gwydion!
Hello, Sarah!
Hello, Hannah!
It appears to be working correctly according to the outputs in your question.
A few suggestions:
It might be useful to print the command out as a string before running it, to check it's what you want.
You may find backticks useful when you have a string containing backslashes and quotation marks.
You have not supplied any format to fmt.Printf, hence the EXTRA in that output.
Using println to print the error will not stringify it, so use fmt.Printf for that too.
package main
import (
"fmt"
"os/exec"
)
func main() {
cmd := exec.Command("cmd", `C:\path\tools\util\Utility.exe C:\file_Location "select * from TABLE" C:\output_path\output.txt`)
fmt.Printf("%s\n", cmd.String())
out, err := cmd.Output()
fmt.Printf("%s\n", string(out))
if err != nil {
fmt.Printf(" Error running decomp %s\n", err)
}
}
Playground: https://go.dev/play/p/3t0aOxAZRtU
I encounter a weird behavior with exec.Wait() with a modified Stdin.
I'm just modifying Stdin in order to be able to duplicate its content, count the amount of data… but that's not the problem here.
I've made this stripped down program just to demonstrate the strange behavior :
with a modified Stdin, cmd.Wait() waits indefinitely… until I press "enter" or "^C"
with an unmodified Stdin (uncomment the line cmd.Stdin = os.Stdin), the program processes to the end flawlessly.
when I launch this program (with a modified Stdin) with Delve (dlv debug), the program processes to the end flawlessly !
I also added a time.Sleep of 30 seconds between cmd.Start() and cmd.Wait(), and then attached the program to Delve (dlv attach PID). When I enter continue, cmd.Wait() waits indefinitely… until I press "enter" or "^C"
I tested these behaviours with go1.11 and go1.12
package main
import (
"fmt"
"os"
"os/exec"
)
type Splitter struct {
f *os.File
fd int
}
func NewSplitter(f *os.File) *Splitter {
return &Splitter{f, int(f.Fd())}
}
func (s *Splitter) Close() error {
return s.f.Close()
}
func (s *Splitter) Read(p []byte) (int, error) {
return s.f.Read(p)
}
func (s *Splitter) Write(p []byte) (int, error) {
return s.f.Write(p)
}
func main() {
var cmd *exec.Cmd
cmd = exec.Command("cat", "foobarfile")
cmd.Stdin = NewSplitter(os.Stdin)
//cmd.Stdin = os.Stdin
cmd.Stdout = NewSplitter(os.Stdout)
cmd.Stderr = NewSplitter(os.Stderr)
cmd.Start()
cmd.Wait()
fmt.Println("done")
}
Is there something I'm doing wrong ?
Thanks for your help.
This program duplicates the content as you asked. You can although try the commented part as well. And the comments are self - explanatory, I hope it explains your query.
package main
import (
"io"
"log"
"os"
"os/exec"
)
func main() {
// Execute cat command w/ arguments
// cmd := exec.Command("cat", "hello.txt")
// Execute cat command w/o arguments
cmd := exec.Command("cat")
// Attach STDOUT stream
stdout, err := cmd.StdoutPipe()
if err != nil {
log.Println(err)
}
// Attach STDIN stream
stdin, err := cmd.StdinPipe()
if err != nil {
log.Println(err)
}
// Attach STDERR stream
stderr, err := cmd.StderrPipe()
if err != nil {
log.Println(err)
}
// Spawn go-routine to copy os's stdin to command's stdin
go io.Copy(stdin, os.Stdin)
// Spawn go-routine to copy command's stdout to os's stdout
go io.Copy(os.Stdout, stdout)
// Spawn go-routine to copy command's stderr to os's stderr
go io.Copy(os.Stderr, stderr)
// Run() under the hood calls Start() and Wait()
cmd.Run()
// Note: The PIPES above will be closed automatically after Wait sees the command exit.
// A caller need only call Close to force the pipe to close sooner.
log.Println("Command complete")
}
You are replacing the process file descriptors, which are normally *os.File, with other Go types. In order for stdin to act like a stream, the os/exec package needs to launch a goroutine to copy the data between the io.Reader and the process. This is documented in the os/exec package:
// Otherwise, during the execution of the command a separate
// goroutine reads from Stdin and delivers that data to the command
// over a pipe. In this case, Wait does not complete until the goroutine
// stops copying, either because it has reached the end of Stdin
// (EOF or a read error) or because writing to the pipe returned an error.
If you look at the stack trace from your program, you'll see that it is waiting for the io goroutines to complete in Wait():
goroutine 1 [chan receive]:
os/exec.(*Cmd).Wait(0xc000076000, 0x0, 0x0)
/usr/local/go/src/os/exec/exec.go:510 +0x125
main.main()
Because you are now in control of the data stream, it is up to you to close it as necessary. If Stdin is not necessary here, then don't assign it at all. If it is going to be used, then you must Close() it to have Wait() return.
Another option is to ensure that you are using an *os.File, which the easiest method is to use the StdinPipe, StdoutPipe and StderrPipe methods, which in turn use os.Pipe(). This way ensures that the process is dealing only with *os.File, and not with other Go types.
I am not sure if this is a problem of the called program or if the problem is caused by the way I call the program. Because of this I start at the source code.
I need to call ssh from a program (if you are interested in the reasons I will mention them below) but ssh silently exits.
When I call ssh -v user#remotehost from shell this succeeds:
the wanted debug output on stderr is shown
I am asked for the password
I can see the remote hosts shell
But when I do the same from within my program (myssh -v user#remotehost only this happens:
I am asked for the password
Neither the debug output on stderr is shown nor do I reach the remote hosts shell.
This is my sourcecode:
package main
import (
"fmt"
"log"
"os"
"os/exec"
)
func main() {
params := os.Args[1:]
fmt.Printf("passing this to ssh: %s\n", params)
cmd := exec.Command("ssh", params...)
err := cmd.Run()
if err != nil {
log.Fatal(err)
}
}
Reason why I wrote this code: I use Ansible which calls ssh. I need to "manipulate" the parameters that Ansible passes to ssh. So far I asked on the OpenSSH and Ansible mailing lists, there is no means in OpenSSH and Ansible to change the parameters (for others it is, but not those I need). The best suggestion I got and that I want to implement is to provide an alternative ssh command to Ansible, use that to receive and modify the parameters and pass them on to the real ssh.
Are you capturing Stdout and Stderr from cmd? This is where the respective outputs of the command is sent to. The documentation for exec.Command has a really good example on this.
In your case, you'll also want to setup Stdin so that you can pass the password for example.
Here's a really basic example based on your code:
package main
import (
"bytes"
"fmt"
"log"
"os"
"os/exec"
)
func main() {
params := os.Args[1:]
fmt.Println("Passing this to ssh: %s", params)
var stdin bytes.Buffer
var stdout bytes.Buffer
var stderr bytes.Bufer
cmd := exec.Command("ssh", params...)
cmd.Stdin = &stdin
cmd.Stdout = &stdout
cmd.Stderr = &stderr
if err := cmd.Run(); err != nil {
log.Fatal(err)
}
fmt.Println("Stdout: %s", stdout.String())
fmt.Println("stderr: %s", stderr.String())
}
Since stdin, stdout, and stderr are all bytes.Buffers, you can read and write from them just like any other buffer.
You might also want to consider using the golang.org/x/crypto/ssh package which provides a native SSH interface for Go instead of using sub-processes.
I started learning and playing around with Go to see what it is like to make some more complex console/cli type tools instead of using shells or Python. I want to execute commands and display the output. I figured out how to print the output like this:
out, err := exec.Command("pwd").Output()
print(string(out))
Is there a way to execute the commands and have it default to stdout like a shell script, or do I need to make a helper function for this?
Update: After getting IntelliJ and the Go plugin, I poked around in the Go source and agree there is currently no way to do with without a helper method.
It is not possible to reuse a Cmd object as per this comment in the exec.go source code:
// A Cmd cannot be reused after calling its Run, Output or CombinedOutput
// methods.
I did incorporate the stdout option into my own helper, including other options like shell integration. I will try turn that into open source if I can make it useful. An interesting first day of Go.
The solution
Actually, it is pretty easy. You can set the stdout of the command to os.Stdout and Bob's your uncle:
package main
import (
"os"
"os/exec"
)
func main() {
cmd := exec.Command("pwd")
cmd.Stdout = os.Stdout
err := cmd.Run()
if err != nil {
panic(err)
}
}
What's happening here?
By default, the output of a command is stored in a bytes.Buffer if cmd.Stdout is not set to another io.Writer. The call of cmd.Output() then runs the command and saves the output to said buffer.
Since os.Stdout implements io.Writer interface, we simply set cmd.Stdout to be os.Stdout. Now when .Run() is called, the output of the command gets written to the io.Writer defined in cmd.Stdout, which happens to be os.Stdout and the output gets written in the shell.
EDIT: As per comment, if all commands should write to os.Stdout, there of course is no way to prevent some helper. I'd do it like this:
package main
import (
"os"
"os/exec"
)
func CmdToStdout( c string ) (err error){
cmd := exec.Command(c)
cmd.Stdout = os.Stdout
err = cmd.Run()
return
}
func main() {
err := CmdToStdout("pwd")
if err != nil {
panic(err)
}
}
You have to create a helper if you need this often (and 5 lines looks too much). Based on the documentation this is a recommended way:
package main
import (
"fmt"
"log"
"os/exec"
)
func main() {
out, err := exec.Command("date").Output()
if err != nil {
log.Fatal(err)
}
fmt.Printf("The date is %s\n", out)
}
I have a large buffer (buffer []byte) that I would like to print to stdout but pipe through a pager like less or more. Kind of like the man command. I don't want to write the buffer to tmp file first or make the user manually pipe the output to a pager on the command line.
I can find examples of how to pipe the output of one command to another, but nothing starting with an internal buffer.
Any ideas?
Thanks.
In order to pipe to a pager, you can do something like this:
package main
import (
"fmt"
"io"
"os"
"os/exec"
)
func main() {
// declare your pager
cmd := exec.Command("less")
// create a pipe (blocking)
r, stdin := io.Pipe()
// Set your i/o's
cmd.Stdin = r
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
// Create a blocking chan, Run the pager and unblock once it is finished
c := make(chan struct{})
go func() {
defer close(c)
cmd.Run()
}()
// Pass anything to your pipe
fmt.Fprintf(stdin, "hello world\n")
// Close stdin (result in pager to exit)
stdin.Close()
// Wait for the pager to be finished
<-c
}
Sounds like what you need is an Encoder. Are you using a package for the pager? If so you might want to look for an Encoder in the package, or create your own if one isn't provided.
Here is an example of how you could use a JSON Encoder to achieve something similar to what you're trying to do:
b := []byte(`{ ... some json object ... }`)
json_encoder := json.NewEncoder(os.Stdout)
json_encoder.Encode(b)
In this example, the JSON encoder accepts the []byte and does all the work to encode it into a JSON document and write to the provided io.writer. If you're using a package and it doesn't provide an encoder, you can get ideas on how to write one by looking into the JSON Encoder source code to create your own.