Below is a command line wrapper which can parse user input command line string to Go exec.Command(). Here is why I want to write a wrapper on it:
exec.Command can only access command parameters 1 by 1, but I want to feed the shell command line as a whole
I want to run all commands in parallel(for me access multiple urls in parallel and retreive the data) - this is in exeCmd(cmdline string, output string, wg sync.WaitGroup)
I want to choose where the output data goes: stdout, local file or network - I defined a map which maps cmdline to output
Here is my code
package main
import (
"fmt"
"log"
"os"
"os/exec"
"strings"
"sync"
)
// command line parser , generate exec.Command
// cmd is same command line as running in shell(remove single quote)
func GenCmd(cmdline string) *exec.Cmd {
fmt.Println("orgin command is ", cmdline)
// splitting head => g++ parts => rest of the command
parts := strings.Fields(cmdline)
// loopArr(parts)
head := parts[0]
parts = parts[1:len(parts)]
// exec cmd & collect output
cmd := exec.Command(head, parts...)
fmt.Printf("Generated cmdline : %s\n", cmd)
return cmd
}
func exeCmd(cmdline string, output string, wg *sync.WaitGroup) {
fmt.Println("Start execCmd() ")
cmd := GenCmd(cmdline)
// check if assigned output file
if output != "" {
f, err := os.Create(output)
if err != nil {
log.Fatal(err)
}
defer f.Close()
cmd.Stdout = f // set stdout to short-response.json
err = cmd.Run()
if err != nil {
log.Fatal(err)
}
} else {
out, err := cmd.Output()
if err != nil {
fmt.Printf("%s", err)
}
fmt.Printf("%s", out)
}
wg.Done() // signal to waitgroup this goroutine complete
}
func main() {
x := make(map[string]string)
x["echo newline >> foo.o"] = ""
x["echo newline >> f1.o"] = "cmd1.txt"
cmdCnt := len(x)
wg := new(sync.WaitGroup)
wg.Add(cmdCnt)
for cmd, output := range x {
go exeCmd(cmd, output, wg) // empty string output to stdout
}
wg.Wait()
}
Go Playground for code above
My question is :
Is there a more decent way of doing this ? any exsiting go package already doing this ?
(better to have) Can someone help on network output part, write the output to another host
Related
In Go, I would like to execute a binary from within my application and continually read what the command prints to stdout. However, the one caveat is that the binary is programmed to execute its task infinitely until it reads the enter key, and I don't have access to the binary's source code. If I execute the binary directly from a terminal, it behaves correctly. However, if I execute the binary from within my application, it somehow thinks that it reads the enter key, and closes almost immediately. Here is a code snippet demonstrating how I'm trying to execute the binary, pipe it's stdout, and print it to the screen:
func main() {
// The binary that I want to execute.
cmd := exec.Command("/usr/lib/demoApp")
// Pipe the command's output.
stdout, err := cmd.StdoutPipe()
if err != nil {
fmt.Println(err)
}
stdoutReader := bufio.NewReader(stdout)
// Start the command.
err = cmd.Start()
if err != nil {
fmt.Println(err)
}
// Read and print the command's output.
buff := make([]byte, 1024)
var n int
for err == nil {
n, err = stdoutReader.Read(buff)
if n > 0 {
fmt.Printf(string(buff[0:n]))
}
}
_ = cmd.Wait()
}
Any ideas if what I'm trying to accomplish is possible?
As #mgagnon mentioned, your problem might lie somewhere else; like perhaps the external dependency just bails due to not running in a terminal. Using following to simulate demoApp:
func main() {
fmt.Println("Press enter to exit")
// Every second, report fake progress
go func() {
for {
fmt.Print("Doing stuff...\n")
time.Sleep(time.Second)
}
}()
for {
// Read single character and if enter, exit.
consoleReader := bufio.NewReaderSize(os.Stdin, 1)
input, _ := consoleReader.ReadByte()
// Enter = 10 | 13 (LF or CR)
if input == 10 || input == 13 {
fmt.Println("Exiting...")
os.Exit(0)
}
}
}
... this works fine for me:
func main() {
cmd := exec.Command("demoApp.exe")
stdout, err := cmd.StdoutPipe()
if err != nil {
panic(err)
}
stdin, err := cmd.StdinPipe()
if err != nil {
log.Fatal(err)
}
go func() {
defer stdin.Close()
// After 3 seconds of running, send newline to cause program to exit.
time.Sleep(time.Second * 3)
io.WriteString(stdin, "\n")
}()
cmd.Start()
// Scan and print command's stdout
scanner := bufio.NewScanner(stdout)
for scanner.Scan() {
fmt.Println(scanner.Text())
}
// Wait for program to exit.
cmd.Wait()
}
$ go run main.go
Press enter to exit
Doing stuff...
Doing stuff...
Doing stuff...
Exiting...
The only difference between this and your code is that I'm using stdin to send a newline after 3 seconds to terminate the cmd. Also using scanner for brevity.
Using this as my /usr/lib/demoApp:
package main
import (
"fmt"
"time"
)
func main() {
for {
fmt.Print("North East South West")
time.Sleep(time.Second)
}
}
This program works as expected:
package main
import (
"os"
"os/exec"
)
func main() {
cmd := exec.Command("demoApp")
stdout, err := cmd.StdoutPipe()
if err != nil {
panic(err)
}
cmd.Start()
defer cmd.Wait()
for {
var b [1024]byte
stdout.Read(b[:])
os.Stdout.Write(b[:])
}
}
Is it possible to read a commands output with its color attributes. I mean, can we read the actual escape sequences.
for instance;
A command output is red colored:
Hello
I want to read it as :
\033[31;1;4mHello\033[0m
Currently I am reading it like:
func stat(hash string) string {
cmd := exec.Command("git", "show", "--stat", hash)
out, err := cmd.Output()
if err != nil {
return err.Error()
}
return string(out)
}
Use the github.com/creack/pty library to run the command in a pty
This works for me
The escape sequences are visible in the output
package main
import (
"github.com/creack/pty"
"io"
"os"
"os/exec"
)
func main() {
hash := os.Args[1]
cmd := exec.Command("git", "show", "--stat", hash)
f, err := pty.Start(cmd)
if err != nil {
panic(err)
}
io.Copy(os.Stdout, f)
}
In this minimal working example I'm trying to do the following:
Prompt user for password
Unmarshal JSON either from files specified as arguments or from STDIN
Here's the source code:
package main
import (
"encoding/json"
"fmt"
"io"
"log"
"os"
"syscall"
"golang.org/x/crypto/ssh/terminal"
)
const correctPassword = "secret"
func main() {
args := os.Args[1:]
var passwd string
for {
passwd = promptPassword()
if passwd == correctPassword {
log.Println("Correct password! Begin processing...")
break
}
log.Println("Incorrect password!")
}
if len(args) == 0 { // Read from stdin
log.Println("Reading from stdin")
dec := json.NewDecoder(os.Stdin)
for {
var v interface{}
if err := dec.Decode(&v); err == io.EOF {
break
} else if err != nil {
log.Fatal(err)
}
log.Printf("%#v", v)
}
}
for _, fileName := range args {
log.Println("Reading from", fileName)
f, err := os.Open(fileName)
if err != nil {
log.Println(err)
continue
}
defer f.Close()
dec := json.NewDecoder(f)
for {
var v interface{}
if err := dec.Decode(&v); err == io.EOF {
break
} else if err != nil {
log.Fatal(err)
}
log.Printf("%#v", v)
}
}
}
func promptPassword() (passwd string) {
for {
fmt.Fprintln(os.Stderr, "Enter password:")
b, _ := terminal.ReadPassword(int(syscall.Stdin))
passwd = string(b)
if passwd != "" {
break
}
}
return passwd
}
Everything works all right except when already prepared data is piped or redirected (e.g. go run main.go < mydata.json, or echo 42 | go run main.go, etc).
When I pipe or redirect some data to the program, the data gets processed by the password prompt, not the JSON decoder part. Is there any way to at first prompt for the password, and only after process the incoming data?
I was trying to detect if there's any data in STDIN to read it and store in some temporary bytes slice, but I can't find how to close/truncate the STDIN, so it won't read data twice.
Without any changes to your program, you can include password in stdin before json, eg (bash): {echo pass; cat data.json; } | goprog, or cat pass.txt data.json | goprog
For better method for password passing (eg environment or file descriptor) look at sshpass: https://linux.die.net/man/1/sshpass
You can also buffer all stdin, and reuse its content later (via io.Reader)
Redesign your application logic to function which accept io.Reader as source of data to unmarshall.
In main() pass os.Stdin to mentioned function if there is no file argument on command line, otherwise (try to) open file and pass it to unmarshalling function.
Note: for deciding whether to print prompt or not you may use isatty like function, which tells if stdin is interactive: https://github.com/mattn/go-isatty
You wont be able because the shell will close the stdin file descriptor once it has finished to write the content.
Once stdin is closed you cant re open it, this is controlled by the shell.
Check this program, to test this behavior
package main
import (
"fmt"
"io"
"os"
)
func main() {
io.Copy(os.Stdout, os.Stdin)
fmt.Println("done")
some := make([]byte, 100)
_, err := os.Stdin.Read(some)
fmt.Println(err)
fmt.Println(string(some))
}
The output will be
$ echo "some" | go run main.go
some
done
EOF
I have a bunch of systems commands which are somwhat similar to appending new content to a file. I wrote a simple script to execute system commands, which works well if there are single words like 'ls' , 'date' etc. But if the command is greater than that, program dies.
The following is the code
package main
import (
"fmt"
"os/exec"
"sync"
)
func exe_cmd(cmd string, wg *sync.WaitGroup) {
fmt.Println(cmd)
c = cmd.Str
out, err := exec.Command(cmd).Output()
if err != nil {
fmt.Println("error occured")
fmt.Printf("%s", err)
}
fmt.Printf("%s", out)
wg.Done()
}
func main() {
wg := new(sync.WaitGroup)
wg.Add(3)
x := []string{"echo newline >> foo.o", "echo newline >> f1.o", "echo newline >> f2.o"}
go exe_cmd(x[0], wg)
go exe_cmd(x[1], wg)
go exe_cmd(x[2], wg)
wg.Wait()
}
The following is the error i see
exec: "echo newline >> foo.o": executable file not found in $PATHexec:
"echo newline >> f2.o": executable file not found in $PATHexec:
"echo newline >> f1.o": executable file not found in $PATH
I guess, this may be due to, not sending cmds and arguments seperately ( http://golang.org/pkg/os/exec/#Command ). I am wondering how to subvert this, since I don't know how many arguments will be there in my command which needs to be executed.
I found a relatively decent way to achieve the same .
out, err := exec.Command("sh","-c",cmd).Output()
Works for me until now. Still finding better ways to achieve the same.
Edit1:
Finally a easier and efficient (atleast so far) way to do would be like this
func exeCmd(cmd string, wg *sync.WaitGroup) {
fmt.Println("command is ",cmd)
// splitting head => g++ parts => rest of the command
parts := strings.Fields(cmd)
head := parts[0]
parts = parts[1:len(parts)]
out, err := exec.Command(head,parts...).Output()
if err != nil {
fmt.Printf("%s", err)
}
fmt.Printf("%s", out)
wg.Done() // Need to signal to waitgroup that this goroutine is done
}
Thanks to variadic arguments in go and people that pointed that out to me :)
For exec.Command() the first argument needs to be the path to the executable. Then the remaining arguments will be supplied as arguments to the executable. Use strings.Fields() to help split the word into a []string.
Example:
package main
import (
"fmt"
"os/exec"
"sync"
"strings"
)
func exe_cmd(cmd string, wg *sync.WaitGroup) {
fmt.Println(cmd)
parts := strings.Fields(cmd)
out, err := exec.Command(parts[0],parts[1]).Output()
if err != nil {
fmt.Println("error occured")
fmt.Printf("%s", err)
}
fmt.Printf("%s", out)
wg.Done()
}
func main() {
wg := new(sync.WaitGroup)
commands := []string{"echo newline >> foo.o", "echo newline >> f1.o", "echo newline >> f2.o"}
for _, str := range commands {
wg.Add(1)
go exe_cmd(str, wg)
}
wg.Wait()
}
Here's an alternative approach that just writes all the commands to a file then executes that file within the context of the new created output directory.
Example 2
package main
import (
"os"
"os/exec"
"fmt"
"strings"
"path/filepath"
)
var (
output_path = filepath.Join("./output")
bash_script = filepath.Join( "_script.sh" )
)
func checkError( e error){
if e != nil {
panic(e)
}
}
func exe_cmd(cmds []string) {
os.RemoveAll(output_path)
err := os.MkdirAll( output_path, os.ModePerm|os.ModeDir )
checkError(err)
file, err := os.Create( filepath.Join(output_path, bash_script))
checkError(err)
defer file.Close()
file.WriteString("#!/bin/sh\n")
file.WriteString( strings.Join(cmds, "\n"))
err = os.Chdir(output_path)
checkError(err)
out, err := exec.Command("sh", bash_script).Output()
checkError(err)
fmt.Println(string(out))
}
func main() {
commands := []string{
"echo newline >> foo.o",
"echo newline >> f1.o",
"echo newline >> f2.o",
}
exe_cmd(commands)
}
out, _ := exec.Command("sh", "-c", "date +\"%Y-%m-%d %H:%M:%S %Z\"").Output()
exec.Command("sh","-c","ls -al -t | grep go >>test.txt").Output()
fmt.Printf("%s\n\n",out)
Tested couple cases and all work good. This is a lifesaver if you are dealing with quick shell commands in your program. Not tested with complex cases.
I'm writing a small program with an interpreter, I would like to pipe any command that is not recognized by my shell to bash, and print the output as if written in a normal terminal.
func RunExtern(c *shell.Cmd) (string, os.Error) {
cmd := exec.Command(c.Cmd(), c.Args()...)
out, err := cmd.Output()
return string(out), err
}
this is what I've written so far, but it only executes a program with its args, I would like to send the whole line to bash and get the output, any idea how to do so ?
For example, to list directory entries in columns,
package main
import (
"exec"
"fmt"
"os"
)
func BashExec(argv []string) (string, os.Error) {
cmdarg := ""
for _, arg := range argv {
cmdarg += `"` + arg + `" `
}
cmd := exec.Command("bash", "-c", cmdarg)
out, err := cmd.Output()
return string(out), err
}
func main() {
out, err := BashExec([]string{`ls`, `-C`})
if err != nil {
fmt.Println(err)
}
fmt.Println(out)
}