I am refer to the post How to execute system command with unknown arguments? to run a jq command on my ubuntu shell.
Below is the code I tried
import (
"fmt"
"os/exec"
"sync"
"strings"
)
func exeCmd(cmd string, wg *sync.WaitGroup) {
fmt.Println("command is ",cmd)
// splitting head => g++ parts => rest of the command
parts := strings.Fields(cmd)
head := parts[0]
parts = parts[1:len(parts)]
out, err := exec.Command(head,parts...).Output()
if err != nil {
fmt.Printf("%s", err)
}
fmt.Printf("%s", out)
wg.Done() // Need to signal to waitgroup that this goroutine is done
}
func main() {
wg := new(sync.WaitGroup)
wg.Add(1)
x := []string{"jq '(.data.legacyCollection.collectionsPage.stream.edges|map({node:(.node|{url,firstPublished,headline:{default:.headline.default},summary})})) as $edges|{data:{legacyCollection:{collectionsPage:{stream:{$edges}}}}}' long-response.json > short-response.json"}
exeCmd(x[0], wg)
wg.Wait()
}
The output is as below, seems like the commans is correctly detected but shell returns exit status 3 which is "no such process" ?
command is jq '(.data.legacyCollection.collectionsPage.stream.edges|map({node:(.node|{url,firstPublished,headline:{default:.headline.default},summary})})) as $edges|{data:{legacyCollection:{collectionsPage:{stream:{$edges}}}}}' long-response.json > short-repsonse.json exit status 3
Anyone can help on this ?
What I want is a go function that can wrapper and run bash command line the same way as I do on Linux shell
PS: the jq command I tried above works pretty well when I paste it on my Linux shell
Tried somethine else: deleted the single quote in my jq command and my command got executed with output I expect - a parsed json file
but still, I got a exist status 2 , anyone can explain
why the single quote in my command line affects how G parse the command ?
why I still go exist 2 after my shell commands complete?
The program executes the jq command, not a shell. The jq command does not understand the shell syntax passed to the command (the quotes and I/O redirection).
Use the following code to run the command with stdout redirected to short-response.json.
cmd := exec.Command("jq",
"(.data.legacyCollection.collectionsPage.stream.edges|map({node:(.node|{url,firstPublished,headline:{default:.headline.default},summary})})) as $edges|{data:{legacyCollection:{collectionsPage:{stream:{$edges}}}}}",
"long-response.json")
f, err := os.Create("short-response.json")
if err != nil {
log.Fatal(err)
}
defer f.Close()
cmd.Stdout = f // set stdout to short-response.json
err = cmd.Run()
if err != nil {
log.Fatal(err)
}
Related
Below is a command line wrapper which can parse user input command line string to Go exec.Command(). Here is why I want to write a wrapper on it:
exec.Command can only access command parameters 1 by 1, but I want to feed the shell command line as a whole
I want to run all commands in parallel(for me access multiple urls in parallel and retreive the data) - this is in exeCmd(cmdline string, output string, wg sync.WaitGroup)
I want to choose where the output data goes: stdout, local file or network - I defined a map which maps cmdline to output
Here is my code
package main
import (
"fmt"
"log"
"os"
"os/exec"
"strings"
"sync"
)
// command line parser , generate exec.Command
// cmd is same command line as running in shell(remove single quote)
func GenCmd(cmdline string) *exec.Cmd {
fmt.Println("orgin command is ", cmdline)
// splitting head => g++ parts => rest of the command
parts := strings.Fields(cmdline)
// loopArr(parts)
head := parts[0]
parts = parts[1:len(parts)]
// exec cmd & collect output
cmd := exec.Command(head, parts...)
fmt.Printf("Generated cmdline : %s\n", cmd)
return cmd
}
func exeCmd(cmdline string, output string, wg *sync.WaitGroup) {
fmt.Println("Start execCmd() ")
cmd := GenCmd(cmdline)
// check if assigned output file
if output != "" {
f, err := os.Create(output)
if err != nil {
log.Fatal(err)
}
defer f.Close()
cmd.Stdout = f // set stdout to short-response.json
err = cmd.Run()
if err != nil {
log.Fatal(err)
}
} else {
out, err := cmd.Output()
if err != nil {
fmt.Printf("%s", err)
}
fmt.Printf("%s", out)
}
wg.Done() // signal to waitgroup this goroutine complete
}
func main() {
x := make(map[string]string)
x["echo newline >> foo.o"] = ""
x["echo newline >> f1.o"] = "cmd1.txt"
cmdCnt := len(x)
wg := new(sync.WaitGroup)
wg.Add(cmdCnt)
for cmd, output := range x {
go exeCmd(cmd, output, wg) // empty string output to stdout
}
wg.Wait()
}
Go Playground for code above
My question is :
Is there a more decent way of doing this ? any exsiting go package already doing this ?
(better to have) Can someone help on network output part, write the output to another host
I am translating my script from bash to go and unfortunately I am not able to make pass command work. When I run the bash one I receive a window to provide my password to the pass manager.
Here is part of my go code:
package main
import (
"fmt"
"io/ioutil"
"log"
"os"
"os/exec"
)
func run_command(command *exec.Cmd) string {
attach_json, err := command.Output()
if err != nil {
fmt.Println(err.Error())
// os.Exit(0)
}
fmt.Println(attach_json)
return string(attach_json)
}
email := "xyz#abc.com"
cmd_emamil := "GoJira/api-token:" + email
pass_cmd := exec.Command("pass", cmd_emamil, "> /dev/null")
pass_cmd := exec.Command("pass", cmd_emamil)
run_command(pass_cmd)
In the shell command pass >/dev/null, >/dev/null is not an argument to pass; instead, it's an instruction to the shell, telling it to replace file descriptor 1 (stdout) with a handle on /dev/null before pass is started.
When you use exec.Command() there is no shell, so you can't use shell syntax. Instead, assign the file you want stdout to be redirected to to the Stdout of your exec.Command.
devnull, err := os.OpenFile(os.DevNull, os.O_WRONLY, 0755)
if err != nil { panic(err) }
cmd_email := "GoJira/api-token:" + email
pass_cmd := exec.Command("pass", cmd_email)
pass_cmd.Stdout = devnull
I would like to manage a process in Go with the package os/exec. I would like to start it and be able to read the output and write several times to the input.
The process I launch in the code below, menu.py, is just a python script that does an echo of what it has in input.
func ReadOutput(rc io.ReadCloser) (string, error) {
x, err := ioutil.ReadAll(rc)
s := string(x)
return s, err
}
func main() {
cmd := exec.Command("python", "menu.py")
stdout, err := cmd.StdoutPipe()
Check(err)
stdin, err := cmd.StdinPipe()
Check(err)
err = cmd.Start()
Check(err)
go func() {
defer stdin.Close() // If I don't close the stdin pipe, the python code will never take what I write in it
io.WriteString(stdin, "blub")
}()
s, err := ReadOutput(stdout)
if err != nil {
Log("Process is finished ..")
}
Log(s)
// STDIN IS CLOSED, I CAN'T RETRY !
}
And the simple code of menu.py :
while 1 == 1:
name = raw_input("")
print "Hello, %s. \n" % name
The Go code works, but if I don't close the stdin pipe after I write in it, the python code never take what is in it. It is okay if I want to send only one thing in the input on time, but what is I want to send something again few seconds later? Pipe is closed! How should I do? The question could be "How do I flush a pipe from WriteCloser interface?" I suppose
I think the primary problem here is that the python process doesn't work the way you might expect. Here's a bash script echo.sh that does the same thing:
#!/bin/bash
while read INPUT
do echo "Hello, $INPUT."
done
Calling this script from a modified version of your code doesn't have the same issue with needing to close stdin:
func ReadOutput(output chan string, rc io.ReadCloser) {
r := bufio.NewReader(rc)
for {
x, _ := r.ReadString('\n')
output <- string(x)
}
}
func main() {
cmd := exec.Command("bash", "echo.sh")
stdout, err := cmd.StdoutPipe()
Check(err)
stdin, err := cmd.StdinPipe()
Check(err)
err = cmd.Start()
Check(err)
go func() {
io.WriteString(stdin, "blab\n")
io.WriteString(stdin, "blob\n")
io.WriteString(stdin, "booo\n")
}()
output := make(chan string)
defer close(output)
go ReadOutput(output, stdout)
for o := range output {
Log(o)
}
}
The Go code needed a few minor changes - ReadOutput method needed to be modified in order to not block - ioutil.ReadAll would have waited for an EOF before returning.
Digging a little deeper, it looks like the real problem is the behaviour of raw_input - it doesn't flush stdout as expected. You can pass the -u flag to python to get the desired behaviour:
cmd := exec.Command("python", "-u", "menu.py")
or update your python code to use sys.stdin.readline() instead of raw_input() (see this related bug report: https://bugs.python.org/issue526382).
Even though there is some problem with your python script. The main problem is the golang pipe. A trick to solve this problem is use two pipes as following:
// parentprocess.go
package main
import (
"bufio"
"log"
"io"
"os/exec"
)
func request(r *bufio.Reader, w io.Writer, str string) string {
w.Write([]byte(str))
w.Write([]byte("\n"))
str, err := r.ReadString('\n')
if err != nil {
panic(err)
}
return str[:len(str)-1]
}
func main() {
cmd := exec.Command("bash", "menu.sh")
inr, inw := io.Pipe()
outr, outw := io.Pipe()
cmd.Stdin = inr
cmd.Stdout = outw
if err := cmd.Start(); err != nil {
panic(err)
}
go cmd.Wait()
reader := bufio.NewReader(outr)
log.Printf(request(reader, inw, "Tom"))
log.Printf(request(reader, inw, "Rose"))
}
The subprocess code is the same logic as your python code as following:
#!/usr/bin/env bash
# menu.sh
while true; do
read -r name
echo "Hello, $name."
done
If you want to use your python code you should do some changes:
while 1 == 1:
try:
name = raw_input("")
print "Hello, %s. \n" % name
sys.stdout.flush() # there's a stdout buffer
except:
pass # make sure this process won't die when come across 'EOF'
// StdinPipe returns a pipe that will be connected to the command's
// standard input when the command starts.
// The pipe will be closed automatically after Wait sees the command exit.
// A caller need only call Close to force the pipe to close sooner.
// For example, if the command being run will not exit until standard input`enter code here`
// is closed, the caller must close the pipe.
func (c *Cmd) StdinPipe() (io.WriteCloser, error) {}
Im using the following code which run command against binary and need to provide output
if I run the command ftr get apps in the in my mac I got
[app1 apps2]
Now I copy the binary to the test data folder
and run the code below and I want to get the apps, currenlty there is no error but Im not getting also the data, what could be missing here?
Cmd := exec.Command("ftr", "get", "apps")
Cmd.Dir = "./testdata/"
err := Cmd.Start()
fmt.Println(err)
bytes, e := Cmd.Output()
fmt.Println(bytes, e)
You won't directly have an output, since the commands takes some time before it writes in stdout/stderr, so you need to basically wait for something to come out.
The way you can do it is by using bufio.NewScanner, like this:
package main
import (
"bufio"
"fmt"
"os/exec"
"strings"
)
func main() {
args := "get apps"
cmd := exec.Command("ftr", strings.Split(args, " ")...)
cmd.Dir = "./testdata/"
stdout, _ := cmd.StdoutPipe()
cmd.Start()
scanner := bufio.NewScanner(stdout)
scanner.Split(bufio.ScanWords)
for scanner.Scan() {
m := scanner.Text()
fmt.Println(m)
}
cmd.Wait()
}
If your command writes on stderr instead of stdout, you might need to use cmd.StderrPipe instead of cmd.Stdoutpipe in order to get the output.
Calling both Start and Output is redundant.
Output runs the command and returns its standard output.
Start is used to run the command asynchronously. Simply remove that call:
cmd := exec.Command("ftr", "get", "apps")
cmd.Dir = "./testdata/"
b, err := cmd.Output()
fmt.Println(string(b), err)
I'm trying to start a command in a detached process so that it can continue after go program exits. I need to redirect the output of the command to a file.
What I need is something like this:
func main() {
command := exec.Command("/tmp/test.sh", ">", "/tmp/out")
if err := command.Start(); err != nil {
fmt.Fprintln(os.Stderr, "Command failed.", err)
os.Exit(1)
}
fmt.Println("Process ID:", command.Process.Pid)
}
Obviously such redirect doesn't work. As I immediately exit from the program after starting the long running command, I cannot open a file and bind it to the Stdout.
Is there any way to achieve such a redirect?
You may start a shell which executes your command / app, and you may redirect its output to a file. The shell will continue to run and execute your script / app even if your Go app exits.
Example:
cmd := exec.Command("sh", "-c", "/tmp/test.sh > /tmp/out")
if err := cmd.Start(); err != nil {
panic(err)
}
fmt.Println("Process ID:", cmd.Process.Pid)
Test it with this simple Go app (replace /tmp/test.sh with the name of the executable binary you compile this into):
package main
import ("fmt"; "time")
func main() {
for i := 0; i < 10; i++ {
fmt.Printf("%d.: %v\n", i, time.Now())
time.Sleep(time.Second)
}
}
This app simply prints a line to the standard output once every second. You can see how the output file is being written e.g. with tail -f /tmp/out.
Note that you may use other shells to execute your scripts to your liking (and to what the test.sh script dictates).
For example to use bash:
cmd := exec.Command("/bin/bash", "-c", "/tmp/test.sh > /tmp/out")
// rest is unchanged
Note that the command to be executed by the shell is passed as a single string argument, and it is not broken down into multiple as you would do it if you were to execute it directly in the command prompt.
Maybe you can try to use this: https://stackoverflow.com/a/28918814/2728768
Opening a file (and os.File implements io.Writer), and then passing it as the command.Stdout could do the trick:
func main() {
command := exec.Command("./tmp/test.sh")
f, err := os.OpenFile("/tmp/out", os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
if err != nil {
fmt.Printf("error opening file: %v", err)
}
defer f.Close()
// On this line you're going to redirect the output to a file
command.Stdout = f
if err := command.Start(); err != nil {
fmt.Fprintln(os.Stderr, "Command failed.", err)
os.Exit(1)
}
fmt.Println("Process ID:", command.Process.Pid)
}
Not sure this could be a viable solution for your case. I've tried it locally and it seems working... remember that your user should be able to create/update the file.