I'm trying to read and write to an exec.Command, but I'm struggling with piping.
This first one is a example of the behaviour I want. The go application just proxies stdin, stdout and stderr to the command:
package main
import (
"fmt"
"os"
"os/exec"
)
func main() {
cmd := exec.Command("bash", "-i")
cmd.Stdin = os.Stdin
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
cmd.Run()
fmt.Println("Done")
}
Running this is like:
pensarando#home: ~/go/src/test$ ./test
pensarando#home: ~/go/src/test$ echo "hello"
hello
pensarando#home: ~/go/src/test$ exit
Done
pensarando#home: ~/go/src/test$
But this one does not work at all:
package main
import (
"fmt"
"io"
"os"
"os/exec"
"sync"
)
func main() {
cmd := exec.Command("bash", "-i")
in, _ := cmd.StdinPipe()
out, _ := cmd.StdoutPipe()
err, _ := cmd.StderrPipe()
exit := make(chan struct{})
done := func() {
in.Close()
out.Close()
err.Close()
cmd.Wait()
close(exit)
}
var once sync.Once
go func() {
io.Copy(os.Stdout, out)
fmt.Println("done stdout")
once.Do(done)
}()
go func() {
io.Copy(in, os.Stdin)
fmt.Println("done stdin")
once.Do(done)
}()
go func() {
io.Copy(os.Stderr, err)
fmt.Println("done stderr")
once.Do(done)
}()
cmd.Start()
<-exit
fmt.Println("Done")
}
When I run this one from the terminal, the output is for example:
pensarando#home:~/go/src/test$ go build
pensarando#home:~/go/src/test$ ./test
pensarando#home:~/go/src/test$ echo "Hello"
[1]+ Stopped ./test
pensarando#home:~/go/src/test$ echo "Hello"
Hello
pensarando#home:~/go/src/test$ fg
./test
done stdin
done stdout
done stderr
Done
pensarando#home:~/go/src/test$
What exactly happens here?
Related
I am trying to run a simple program that spawns a vim process.
The user should be able (when the exec.Command starts) to switch to vim window and the process execution should halt there.
When user closes vim (wq!) the program execution should resume from that point.
The following simple attempt fails but I cannot figure out why
package main
import (
"log"
"os/exec"
)
func main() {
cmd := exec.Command("vim", "lala")
err := cmd.Run()
if err != nil {
log.Fatal(err)
}
}
▶ go run main.go
2022/11/25 09:16:44 exit status 1
exit status 1
Why the exit status 1?
You missed these two lines:
cmd.Stdin = os.Stdin
cmd.Stdout = os.Stdout
Thanks to these two lines the user is able to edit with vim the file in the terminal. The control is returned to the program when the user quit from the terminal (e.g., with the command :wq). Below, you can find the whole code:
package main
import (
"log"
"os"
"os/exec"
)
func main() {
cmd := exec.Command("vim", "lala")
cmd.Stdin = os.Stdin
cmd.Stdout = os.Stdout
err := cmd.Run()
if err != nil {
log.Fatal(err)
}
}
Hope this helps!
Because you should set Stdin and Stdoutfor cmd:
package main
import (
"log"
"os"
"os/exec"
)
func main() {
cmd := exec.Command("vim", "lala")
cmd.Stdin = os.Stdin
cmd.Stdout = os.Stdout
err := cmd.Run()
if err != nil {
log.Fatal(err)
}
}
I'm trying to execute a shell command and compress it's output.
The problem is that I then need to interface with an API that expects a Reader.
For that I tried with the following (simplified code):
package main
import (
"encoding/hex"
"testing"
"fmt"
"io"
"io/ioutil"
"os/exec"
"compress/gzip"
)
func TestPipe(t *testing.T) {
cmd := exec.Command("echo", "hello_from_echo")
reader, writer := io.Pipe()
gzW := gzip.NewWriter(writer)
cmd.Stdout = gzW
cmd.Start()
go func() {
fmt.Println("Waiting")
cmd.Wait()
fmt.Println("wait done")
// writer.Close()
// gzW.Close()
}()
msg, _ := ioutil.ReadAll( reader )
fmt.Println( hex.EncodeToString( msg ) )
}
The problem is that ReadAll hangs forever. If I close gzW nothing really changes. However, if I close the writer variable, now the program finishes without hanging, but the output is:
$ go test -run Pipe
Waiting
wait done
1f8b080000096e8800ff
PASS
However, no matter what I echo the output is the same. If I try it from the command line like this: echo "hello_from_echo" | gzip | hexdump the output is totally different, so there's something wrong with that approach.
Any clue what could be the problem?
Thanks in advance
You're closing the gzip writer and pipe writer in the wrong order. You need to close the gzip.Writer to flush any buffers and write the gzip footer, then you can close the PipeWriter to unblock the ReadAll. Also adding the WaitGroup ensures that you're not blocked on any of the close calls.
cmd := exec.Command("echo", "hello_from_echo and more")
pr, pw := io.Pipe()
gzW := gzip.NewWriter(pw)
cmd.Stdout = gzW
cmd.Start()
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
err := cmd.Wait()
if err != nil {
log.Println(err)
}
gzW.Close()
pw.Close()
}()
buf, err := ioutil.ReadAll(pr)
if err != nil {
t.Fatal(err)
}
wg.Wait()
fmt.Println(hex.EncodeToString(buf))
Streaming commands output progress question addresses the problem of printing progress of a long running command.
I tried to put the printing code within a goroutine but the scanner claims to have already hit the EOF immediately and the for block is never executed.
The bufio.scan code that gets executed on the first execution of the Scan() method is:
// We cannot generate a token with what we are holding.
// If we've already hit EOF or an I/O error, we are done.
if s.err != nil {
// Shut it down.
s.start = 0
s.end = 0
return false
}
And if I print s.err the output is EOF.
The code I'm trying to run is:
cmd := exec.Command("some", "command")
c := make(chan int, 1)
go func(cmd *exec.Cmd, c chan int) {
stdout, _ := cmd.StdoutPipe()
<-c
scanner := bufio.NewScanner(stdout)
for scanner.Scan() {
m := scanner.Text()
fmt.Println(m)
}
}(cmd, c)
cmd.Start()
c <- 1
cmd.Wait()
The idea is to start the Goroutine, get a hold of the cmd.stdout, wait that the cmd is started, and start processing its output.
The result is that the long command gets executed and the program waits for its completion, but nothing is printed to terminal.
Any idea why by the time scanner.Scan() is invoked for the first time the stdout has already reached EOF?
There are some problems:
The pipe is being closed before reading all data.
Always check for errors
Start cmd.Start() after c <- struct{}{} and use unbuffered channel c := make(chan struct{})
Two working sample codes:
1: Wait using channel then close the pipe after EOF using defer func() { c <- struct{}{} }(), like this working sample code:
package main
import (
"bufio"
"fmt"
"os/exec"
)
func main() {
cmd := exec.Command("Streamer")
c := make(chan struct{})
go run(cmd, c)
c <- struct{}{}
cmd.Start()
<-c
if err := cmd.Wait(); err != nil {
fmt.Println(err)
}
fmt.Println("done.")
}
func run(cmd *exec.Cmd, c chan struct{}) {
defer func() { c <- struct{}{} }()
stdout, err := cmd.StdoutPipe()
if err != nil {
panic(err)
}
<-c
scanner := bufio.NewScanner(stdout)
for scanner.Scan() {
m := scanner.Text()
fmt.Println(m)
}
fmt.Println("EOF")
}
2: Also you may Wait using sync.WaitGroup, like this working sample code:
package main
import (
"bufio"
"fmt"
"os/exec"
"sync"
)
var wg sync.WaitGroup
func main() {
cmd := exec.Command("Streamer")
c := make(chan struct{})
wg.Add(1)
go func(cmd *exec.Cmd, c chan struct{}) {
defer wg.Done()
stdout, err := cmd.StdoutPipe()
if err != nil {
panic(err)
}
<-c
scanner := bufio.NewScanner(stdout)
for scanner.Scan() {
m := scanner.Text()
fmt.Println(m)
}
}(cmd, c)
c <- struct{}{}
cmd.Start()
wg.Wait()
fmt.Println("done.")
}
And Streamer sample code (just for testing):
package main
import "fmt"
import "time"
func main() {
for i := 0; i < 10; i++ {
time.Sleep(1 * time.Second)
fmt.Println(i, ":", time.Now().UTC())
}
}
And see func (c *Cmd) StdoutPipe() (io.ReadCloser, error) Docs:
StdoutPipe returns a pipe that will be connected to the command's
standard output when the command starts.
Wait will close the pipe after seeing the command exit, so most
callers need not close the pipe themselves; however, an implication is
that it is incorrect to call Wait before all reads from the pipe have
completed. For the same reason, it is incorrect to call Run when using
StdoutPipe. See the example for idiomatic usage.
From godocs:
StdoutPipe returns a pipe that will be connected to the command's
standard output when the command starts.
Wait will close the pipe after seeing the command exit, so most
callers need not close the pipe themselves; however, an implication is
that it is incorrect to call Wait before all reads from the pipe have
completed.
You are calling Wait() immediately after starting the command. So the pipe gets closed as soon as the command completes, before making sure you have read all the data from the pipe. Try moving Wait() to your go routine after the scan loop.
go func(cmd *exec.Cmd, c chan int) {
stdout, _ := cmd.StdoutPipe()
<-c
scanner := bufio.NewScanner(stdout)
for scanner.Scan() {
m := scanner.Text()
fmt.Println(m)
}
cmd.Wait()
c <- 1
}(cmd, c)
cmd.Start()
c <- 1
// This is here so we don't exit the program early,
<-c
There's also a simpler way to do things, which is to just assign os.stdout as the cmd's stdout, causing the command to directly write to the os.stdout:
cmd := exec.Command("some", "command")
cmd.Stdout = os.Stdout
cmd.Run()
I am executing bash commands from a golang application. Now the stdout and stderr go directly to console:
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
But I would like stdout and stderr to be returned as string variables from the runBashCommandAndKillIfTooSlow function without printing to the console immediately.
How to implement this?
The code:
package main
import (
"fmt"
"log"
"os"
"os/exec"
"time"
)
func main() {
ok, outString, errString := runBashCommandAndKillIfTooSlow("ls -la", 2000)
fmt.Println("ok")
fmt.Println(ok)
fmt.Println("outString")
fmt.Println(outString)
fmt.Println("errString")
fmt.Println(errString)
}
/*
run bash command and kill it if it works longer than "killInMilliSeconds" milliseconds
*/
func runBashCommandAndKillIfTooSlow(command string, killInMilliSeconds time.Duration) (okResult bool, stdout, stderr string) {
fmt.Println("running bash command...")
fmt.Println(command)
cmd := exec.Command("sh", "-c", command)
cmd.Stdout = os.Stdout // cmd.Stdout -> stdout
cmd.Stderr = os.Stderr // cmd.Stderr -> stderr
okResult = true
err := cmd.Start()
log.Printf("Waiting for command to finish...")
done := make(chan error, 1)
go func() {
done <- cmd.Wait()
}()
select {
case <-time.After(killInMilliSeconds * time.Millisecond):
if err := cmd.Process.Kill(); err != nil {
log.Fatal("failed to kill: ", err)
okResult = false
}
<-done // allow goroutine to exit
// log.Println("process killed")
case err := <-done:
if err != nil {
log.Printf("process done with error = %v", err)
okResult = false
}
}
if err != nil {
log.Fatal(err)
okResult = false
}
return
}
By the way, the program should keep its ability to kill the bash command if it was too slow (killInMilliSeconds parameter).
Set the output to a strings.Builder (in Go versions 1.10 or later) or a bytes.Buffer
var outbuf, errbuf strings.Builder // or bytes.Buffer
cmd.Stdout = &outbuf
cmd.Stderr = &errbuf
After running the command, you can get the stdout and stderr as a string by calling the Builder.String() method:
stdout := outbuf.String()
stderr := errbuf.String()
You can simplify this quite a bit by using cmd.Run() instead of cmd.Start() to have it automatically wait for it to be finish, and use exec.CommandContext() to have it timeout. This will also output in the correct order, whereas the original program is out of order due to go routines.
Here's the exact same program simplified and using #Mello Marmot's answer:
package main
import (
"bytes"
"fmt"
"log"
"os"
"os/exec"
"time"
"golang.org/x/net/context"
)
func main() {
ctx := context.Background()
ok, outString, errString := runBashCommandAndKillIfTooSlow(ctx, "ls -la", 2000*time.Millisecond)
fmt.Println("ok")
fmt.Println(ok)
fmt.Println("outString")
fmt.Println(outString)
fmt.Println("errString")
fmt.Println(errString)
}
/*
run bash command and kill it if it works longer than "killIn"
*/
func runBashCommandAndKillIfTooSlow(ctx context.Context, command string, killIn time.Duration) (okResult bool, stdout, stderr string) {
fmt.Println("running bash command...")
fmt.Println(command)
ctx, _ = context.WithTimeout(ctx, killIn)
cmd := exec.CommandContext(ctx, "sh", "-c", command)
// Set output to Byte Buffers
var outb, errb bytes.Buffer
cmd.Stdout = &outb
cmd.Stderr = &errb
okResult = true
err := cmd.Run()
stdout = outb.String()
stderr = errb.String()
if err != nil {
log.Fatal(err)
okResult = false
}
return
}
Another option is strings.Builder:
package main
import (
"os/exec"
"strings"
)
func main() {
b := new(strings.Builder)
c := exec.Command("go", "version")
c.Stdout = b
c.Run()
println(b.String() == "go version go1.16.3 windows/amd64\n")
}
https://golang.org/pkg/strings#Builder
I have a bunch of systems commands which are somwhat similar to appending new content to a file. I wrote a simple script to execute system commands, which works well if there are single words like 'ls' , 'date' etc. But if the command is greater than that, program dies.
The following is the code
package main
import (
"fmt"
"os/exec"
"sync"
)
func exe_cmd(cmd string, wg *sync.WaitGroup) {
fmt.Println(cmd)
c = cmd.Str
out, err := exec.Command(cmd).Output()
if err != nil {
fmt.Println("error occured")
fmt.Printf("%s", err)
}
fmt.Printf("%s", out)
wg.Done()
}
func main() {
wg := new(sync.WaitGroup)
wg.Add(3)
x := []string{"echo newline >> foo.o", "echo newline >> f1.o", "echo newline >> f2.o"}
go exe_cmd(x[0], wg)
go exe_cmd(x[1], wg)
go exe_cmd(x[2], wg)
wg.Wait()
}
The following is the error i see
exec: "echo newline >> foo.o": executable file not found in $PATHexec:
"echo newline >> f2.o": executable file not found in $PATHexec:
"echo newline >> f1.o": executable file not found in $PATH
I guess, this may be due to, not sending cmds and arguments seperately ( http://golang.org/pkg/os/exec/#Command ). I am wondering how to subvert this, since I don't know how many arguments will be there in my command which needs to be executed.
I found a relatively decent way to achieve the same .
out, err := exec.Command("sh","-c",cmd).Output()
Works for me until now. Still finding better ways to achieve the same.
Edit1:
Finally a easier and efficient (atleast so far) way to do would be like this
func exeCmd(cmd string, wg *sync.WaitGroup) {
fmt.Println("command is ",cmd)
// splitting head => g++ parts => rest of the command
parts := strings.Fields(cmd)
head := parts[0]
parts = parts[1:len(parts)]
out, err := exec.Command(head,parts...).Output()
if err != nil {
fmt.Printf("%s", err)
}
fmt.Printf("%s", out)
wg.Done() // Need to signal to waitgroup that this goroutine is done
}
Thanks to variadic arguments in go and people that pointed that out to me :)
For exec.Command() the first argument needs to be the path to the executable. Then the remaining arguments will be supplied as arguments to the executable. Use strings.Fields() to help split the word into a []string.
Example:
package main
import (
"fmt"
"os/exec"
"sync"
"strings"
)
func exe_cmd(cmd string, wg *sync.WaitGroup) {
fmt.Println(cmd)
parts := strings.Fields(cmd)
out, err := exec.Command(parts[0],parts[1]).Output()
if err != nil {
fmt.Println("error occured")
fmt.Printf("%s", err)
}
fmt.Printf("%s", out)
wg.Done()
}
func main() {
wg := new(sync.WaitGroup)
commands := []string{"echo newline >> foo.o", "echo newline >> f1.o", "echo newline >> f2.o"}
for _, str := range commands {
wg.Add(1)
go exe_cmd(str, wg)
}
wg.Wait()
}
Here's an alternative approach that just writes all the commands to a file then executes that file within the context of the new created output directory.
Example 2
package main
import (
"os"
"os/exec"
"fmt"
"strings"
"path/filepath"
)
var (
output_path = filepath.Join("./output")
bash_script = filepath.Join( "_script.sh" )
)
func checkError( e error){
if e != nil {
panic(e)
}
}
func exe_cmd(cmds []string) {
os.RemoveAll(output_path)
err := os.MkdirAll( output_path, os.ModePerm|os.ModeDir )
checkError(err)
file, err := os.Create( filepath.Join(output_path, bash_script))
checkError(err)
defer file.Close()
file.WriteString("#!/bin/sh\n")
file.WriteString( strings.Join(cmds, "\n"))
err = os.Chdir(output_path)
checkError(err)
out, err := exec.Command("sh", bash_script).Output()
checkError(err)
fmt.Println(string(out))
}
func main() {
commands := []string{
"echo newline >> foo.o",
"echo newline >> f1.o",
"echo newline >> f2.o",
}
exe_cmd(commands)
}
out, _ := exec.Command("sh", "-c", "date +\"%Y-%m-%d %H:%M:%S %Z\"").Output()
exec.Command("sh","-c","ls -al -t | grep go >>test.txt").Output()
fmt.Printf("%s\n\n",out)
Tested couple cases and all work good. This is a lifesaver if you are dealing with quick shell commands in your program. Not tested with complex cases.