How to read/manipulate the input from a connection that is passed to a command stdin?
For example, given the following code.
c, _ := net.Dial("tcp", somehost)
cmd := exec.Command("/bin/sh")
cmd.Stdin, cmd.Stdout, cmd.Stderr = c, c, c
cmd.Run()
How would it be possible to reverse the string from the connection before it is passed to the cmd.Stdin or how could I parse the string and not pass it on to cmd.Stdin?
Ive considered reading from the connection with bufio and then passing it to Command second argument, the params, but I was hoping for a better solution that does not require me to handle all the different cases for args input in a command, but instead just passing it on to Stdin after analysing the input
Ok since you mentioned in comments that "my real issue is how to intercept the input from the connection, parse it and parse it to the Stdin of the command. Seems when I do cmd.Run() I block and hence cant really continously parse"
Here is how I will do it:
import (
"io"
"os"
"os/exec"
)
func main() {
//All errors are not checked
cmd := exec.Command("/bin/sh")
cmdStdin, _ := cmd.StdinPipe()
go func() {
defer cmdStdin.Close()
//here you will need to loop on reading the connection,
//for simplicity lets assume you do that & receive data
//let says you got ls from connection
cmdStdin.Write([]byte("ls\n"))
}()
cmdStdout, _ := cmd.StdoutPipe()
go io.Copy(os.Stdout, cmdStdout)
cmd.Run()
}
Related
Im using the following code which run command against binary and need to provide output
if I run the command ftr get apps in the in my mac I got
[app1 apps2]
Now I copy the binary to the test data folder
and run the code below and I want to get the apps, currenlty there is no error but Im not getting also the data, what could be missing here?
Cmd := exec.Command("ftr", "get", "apps")
Cmd.Dir = "./testdata/"
err := Cmd.Start()
fmt.Println(err)
bytes, e := Cmd.Output()
fmt.Println(bytes, e)
You won't directly have an output, since the commands takes some time before it writes in stdout/stderr, so you need to basically wait for something to come out.
The way you can do it is by using bufio.NewScanner, like this:
package main
import (
"bufio"
"fmt"
"os/exec"
"strings"
)
func main() {
args := "get apps"
cmd := exec.Command("ftr", strings.Split(args, " ")...)
cmd.Dir = "./testdata/"
stdout, _ := cmd.StdoutPipe()
cmd.Start()
scanner := bufio.NewScanner(stdout)
scanner.Split(bufio.ScanWords)
for scanner.Scan() {
m := scanner.Text()
fmt.Println(m)
}
cmd.Wait()
}
If your command writes on stderr instead of stdout, you might need to use cmd.StderrPipe instead of cmd.Stdoutpipe in order to get the output.
Calling both Start and Output is redundant.
Output runs the command and returns its standard output.
Start is used to run the command asynchronously. Simply remove that call:
cmd := exec.Command("ftr", "get", "apps")
cmd.Dir = "./testdata/"
b, err := cmd.Output()
fmt.Println(string(b), err)
While playing with subprocesses and reading stdout through pipes I noticed interesting behaviour.
If I use an io.Pipe() to read the stdout of a subprocess created through os/exec, reading from that pipe hangs forever even when EOF is reached (the process is finished):
cmd := exec.Command("/bin/echo", "Hello, world!")
r, w := io.Pipe()
cmd.Stdout = w
cmd.Start()
io.Copy(os.Stdout, r) // Prints "Hello, World!" but never returns
However, if I use the built-in method StdoutPipe() it works:
cmd := exec.Command("/bin/echo", "Hello, world!")
p := cmd.StdoutPipe()
cmd.Start()
io.Copy(os.Stdout, p) // Prints "Hello, World!" and returns
Digging into the source code of /usr/lib/go/src/os/exec/exec.go, I can see that the StdoutPipe() method actually uses os.Pipe(), not io.Pipe():
pr, pw, err := os.Pipe()
cmd.Stdout = pw
cmd.closeAfterStart = append(c.closeAfterStart, pw)
cmd.closeAfterWait = append(c.closeAfterWait, pr)
return pr, nil
This gives me two clues:
File descriptors are being closed at certain points. Critically, the "write" end of the pipe is being closed after process start.
Instead of io.Pipe() as I used above, os.Pipe() (a lower level call that roughly maps to pipe(2) in POSIX) is used.
However I am still unable to understand why my original example behaves the way it does after taking into account this newfound knowledge.
If I try to close the write end of an io.Pipe() (instead of an os.Pipe()) then it appears to break it completely and nothing gets read (as if I'm reading from a closed pipe even though I thought I passed it to the subprocess):
cmd := exec.Command("/bin/echo", "Hello, world!")
r, w := io.Pipe()
cmd.Stdout = w
cmd.Start()
w.Close()
io.Copy(os.Stdout, r) // Prints nothing, no read buffer available
Okay, so I guess an io.Pipe() is quite different than an os.Pipe(), and probably doesn't behave like Unix pipes where one close() doesn't close it for everybody.
Just so you don't think I'm asking for a quick fix, I already know I can achieve my expected behaviour by using this code:
cmd := exec.Command("/bin/echo", "Hello, world!")
r, w, _ := os.Pipe() // using os.Pipe() instead of io.Pipe()
cmd.Stdout = w
cmd.Start()
w.Close()
io.Copy(os.Stdout, r) // Prints "Hello, World!" and returns on EOF. Works. :-)
What I'm asking for is why does io.Pipe() seem to ignore an EOF from the writer, leaving the reader blocking forever? A valid answer could be that io.Pipe() is the wrong tool for the job because $REASONS but I can't figure out what those $REASONS are because according to the documentation what I'm trying to do seems perfectly reasonable.
Here is a complete example to illustrate what I'm talking about:
package main
import (
"fmt"
"os"
"os/exec"
"io"
)
func main() {
cmd := exec.Command("/bin/echo", "Hello, world!")
r, w := io.Pipe()
cmd.Stdout = w
cmd.Start()
io.Copy(os.Stdout, r) // Blocks here even though EOF is reached
fmt.Println("Finished io.Copy()")
cmd.Wait()
}
"why does io.Pipe() seem to ignore an EOF from the writer, leaving the reader blocking forever?" Because there is no such thing as "EOF from the writer". All an EOF is (in unix) is an indication to the reader that no processes hold the write side of the pipe open. When a process attempts to read from a pipe which has no writers, the read system call returns a value that is conveniently named EOF. Since your parent still has one copy of the write side of the pipe open, read blocks. Stop thinking of EOF as a thing. It is merely an abstraction, and the writer never "sends" it.
You can use a goroutine:
package main
import (
"os"
"os/exec"
"io"
)
func main() {
r, w := io.Pipe()
c := exec.Command("go", "version")
c.Stdout = w
c.Start()
go func() {
io.Copy(os.Stdout, r)
r.Close()
}()
c.Wait()
}
I started learning and playing around with Go to see what it is like to make some more complex console/cli type tools instead of using shells or Python. I want to execute commands and display the output. I figured out how to print the output like this:
out, err := exec.Command("pwd").Output()
print(string(out))
Is there a way to execute the commands and have it default to stdout like a shell script, or do I need to make a helper function for this?
Update: After getting IntelliJ and the Go plugin, I poked around in the Go source and agree there is currently no way to do with without a helper method.
It is not possible to reuse a Cmd object as per this comment in the exec.go source code:
// A Cmd cannot be reused after calling its Run, Output or CombinedOutput
// methods.
I did incorporate the stdout option into my own helper, including other options like shell integration. I will try turn that into open source if I can make it useful. An interesting first day of Go.
The solution
Actually, it is pretty easy. You can set the stdout of the command to os.Stdout and Bob's your uncle:
package main
import (
"os"
"os/exec"
)
func main() {
cmd := exec.Command("pwd")
cmd.Stdout = os.Stdout
err := cmd.Run()
if err != nil {
panic(err)
}
}
What's happening here?
By default, the output of a command is stored in a bytes.Buffer if cmd.Stdout is not set to another io.Writer. The call of cmd.Output() then runs the command and saves the output to said buffer.
Since os.Stdout implements io.Writer interface, we simply set cmd.Stdout to be os.Stdout. Now when .Run() is called, the output of the command gets written to the io.Writer defined in cmd.Stdout, which happens to be os.Stdout and the output gets written in the shell.
EDIT: As per comment, if all commands should write to os.Stdout, there of course is no way to prevent some helper. I'd do it like this:
package main
import (
"os"
"os/exec"
)
func CmdToStdout( c string ) (err error){
cmd := exec.Command(c)
cmd.Stdout = os.Stdout
err = cmd.Run()
return
}
func main() {
err := CmdToStdout("pwd")
if err != nil {
panic(err)
}
}
You have to create a helper if you need this often (and 5 lines looks too much). Based on the documentation this is a recommended way:
package main
import (
"fmt"
"log"
"os/exec"
)
func main() {
out, err := exec.Command("date").Output()
if err != nil {
log.Fatal(err)
}
fmt.Printf("The date is %s\n", out)
}
given the following example:
// test.go
package main
import (
"fmt"
"os/exec"
)
func main() {
cmd := exec.Command("login")
in, _ := cmd.StdinPipe()
in.Write([]byte("user"))
out, err := cmd.CombinedOutput()
if err != nil {
fmt.Println("error:", err)
}
fmt.Printf("%s", out)
}
How can I detect that the process is not going to finish, because it is waiting for user input?
I'm trying to be able to run any script, but abort it if for some reason it tries to read from stdin.
Thanks!
Detecting that the process is not going to finish is a difficult problem. In fact, it is one of the classic "unsolvable" problems in Computer Science: the Halting Problem.
In general, when you are calling exec.Command and will not be passing it any input, it will cause the program to read from your OS's null device (see documentation in the exec.Cmd fields). In your code (and mine below), you explicitly create a pipe (though you should check the error return of StdinPipe in case it is not created correctly), so you should subsequently call in.Close(). In either case, the subprocess will get an EOF and should clean up after itself and exit.
To help with processes that don't handle input correctly or otherwise get themselves stuck, the general solution is to use a timeout. In Go, you can use goroutines for this:
// Set your timeout
const CommandTimeout = 5 * time.Second
func main() {
cmd := exec.Command("login")
// Set up the input
in, err := cmd.StdinPipe()
if err != nil {
log.Fatalf("failed to create pipe for STDIN: %s", err)
}
// Write the input and close
go func() {
defer in.Close()
fmt.Fprintln(in, "user")
}()
// Capture the output
var b bytes.Buffer
cmd.Stdout, cmd.Stderr = &b, &b
// Start the process
if err := cmd.Start(); err != nil {
log.Fatalf("failed to start command: %s", err)
}
// Kill the process if it doesn't exit in time
defer time.AfterFunc(CommandTimeout, func() {
log.Printf("command timed out")
cmd.Process.Kill()
}).Stop()
// Wait for the process to finish
if err := cmd.Wait(); err != nil {
log.Fatalf("command failed: %s", err)
}
// Print out the output
fmt.Printf("Output:\n%s", b.String())
}
In the code above, there are actually three main goroutines of interest: the main goroutine spawns the subprocess and waits for it to exit; a timer goroutine is sent off in the background to kill the process if it's not Stopped in time; and a goroutine that writes the output to the program when it's ready to read it.
Although this would not allow you to "detect" the program trying to read from stdin, I would just close stdin. This way, the child process will just receive an EOF when it tried to read. Most programs know how to handle a closed stdin.
// All error handling excluded
cmd := exec.Command("login")
in, _ := cmd.StdinPipe()
cmd.Start()
in.Close()
cmd.Wait()
Unfortunately, this means you can't use combined output, the following code should allow you to do the same thing. It requires you to import the bytes package.
var buf = new(bytes.Buffer)
cmd.Stdout = buf
cmd.Stderr = buf
After cmd.Wait(), you can then do:
out := buf.Bytes()
I think the solution is to run the child process with closed stdin - by adjusting the Cmd.Stdin appropriately and then Runinng it afterwards instead of using CombinedOutput().
Finally, I'm going to implement a combination of Kyle Lemons answer and forcing the new process have it's own session without a terminal attached to it, so that the executed comand will be aware that there is no terminal to read from.
// test.go
package main
import (
"log"
"os/exec"
"syscall"
)
func main() {
cmd := exec.Command("./test.sh")
cmd.SysProcAttr = &syscall.SysProcAttr{Setsid: true}
out, err := cmd.CombinedOutput()
if err != nil {
log.Fatal("error:", err)
}
log.Printf("%s", out)
}
I'm writing a program in Go that executes a server like program (also Go). Now I want to have the stdout of the child program in my terminal window where I started the parent program. One way to do this is with the cmd.Output() function, but this prints the stdout only after the process has exited. (That's a problem because this server-like program runs for a long time and I want to read the log output)
The variable out is of type io.ReadCloser and I don't know what I should do with it to achieve my task, and I can't find anything helpful on the web on this topic.
func main() {
cmd := exec.Command("/path/to/my/child/program")
out, err := cmd.StdoutPipe()
if err != nil {
fmt.Println(err)
}
err = cmd.Start()
if err != nil {
fmt.Println(err)
}
//fmt.Println(out)
cmd.Wait()
}
Explanation to the code: uncomment the Println function to get the code to compile, I know that Println(out io.ReadCloser) is not a meaningful function.
(it produces the output &{3 |0 <nil> 0} ) These two lines are just required to get the code to compile.
Now I want to have the stdout of the child program in my terminal
window where I started the parent program.
No need to mess with pipes or goroutines, this one is easy.
func main() {
// Replace `ls` (and its arguments) with something more interesting
cmd := exec.Command("ls", "-l")
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
cmd.Run()
}
I believe that if you import io and os and replace this:
//fmt.Println(out)
with this:
go io.Copy(os.Stdout, out)
(see documentation for io.Copy and for os.Stdout), it will do what you want. (Disclaimer: not tested.)
By the way, you'll probably want to capture standard-error as well, by using the same approach as for standard-output, but with cmd.StderrPipe and os.Stderr.
For those who don't need this in a loop, but would like the command output to echo into the terminal without having cmd.Wait() blocking other statements:
package main
import (
"fmt"
"io"
"log"
"os"
"os/exec"
)
func checkError(err error) {
if err != nil {
log.Fatalf("Error: %s", err)
}
}
func main() {
// Replace `ls` (and its arguments) with something more interesting
cmd := exec.Command("ls", "-l")
// Create stdout, stderr streams of type io.Reader
stdout, err := cmd.StdoutPipe()
checkError(err)
stderr, err := cmd.StderrPipe()
checkError(err)
// Start command
err = cmd.Start()
checkError(err)
// Don't let main() exit before our command has finished running
defer cmd.Wait() // Doesn't block
// Non-blockingly echo command output to terminal
go io.Copy(os.Stdout, stdout)
go io.Copy(os.Stderr, stderr)
// I love Go's trivial concurrency :-D
fmt.Printf("Do other stuff here! No need to wait.\n\n")
}