I am trying to write a file from a bash command into a file in Go.
Note there are several reasons for using Go over bash here: I have some more logic such as parsing configuration files, I would like to run that code for multiple DBs in parallele and finally performing some more complex data manipulation after.
dumpStr := fmt.Sprintf("pg_dump -U %s -h %s %s | gzip", DbUserName, DbHost, DbName)
cmd := exec.Command("bash", "-c", dumpStr)
cmd.Env = append(cmd.Env, "PGPASSWORD="+DbPassword)
outfile, err := os.Create(DbName + ".gz")
if err != nil {
panic(err)
}
outfile = cmd.Stdout
defer outfile.Close()
err = cmd.Start()
if err != nil {
panic(err)
}
cmd.Wait()
However, I am getting an emtpy result.
I am getting data if I am executing dumpStr from the CLI but not from that code...
What am I missing?
As Flimzy said, you're not capturing the output of pg_dump. You can do that with Go, or you can use pg_dump-s --file. It can also compress with --compress so no need to pipe to gzip. Then there's no need for bash and you can avoid shell quoting issues.
cmd := exec.Command(
"pg_dump",
"--compress=9",
"--file="+DbName + ".gz",
"-U"+DbUserName,
"-h"+DbHost,
DbName,
)
log.Print("Running pg_dump...")
if err := cmd.Run(); err != nil {
log.Fatal(err)
}
Much simpler and more secure.
For illustration here's how you'd do it all in Go.
Use Cmd.StdoutPipe to get an open IO reader to pg_dump's stdout. Then use io.Copy to copy from stdout to your open file.
#Peter points out that since Cmd.Stdout is an io.Reader it's simpler to assign the open file to cmd.Stdout and let cmd write to it directly.
// Same as above, but no --file.
cmd := exec.Command(
"pg_dump",
"--compress=9",
"-U"+DbUserName,
"-h"+DbHost,
DbName,
)
// Open the output file
outfile, err := os.Create(DbName + ".gz")
if err != nil {
log.Fatal(err)
}
defer outfile.Close()
// Send stdout to the outfile. cmd.Stdout will take any io.Writer.
cmd.Stdout = outfile
// Start the command
if err = cmd.Start(); err != nil {
log.Fatal(err)
}
log.Print("Waiting for command to finish...")
// Wait for the command to finish.
if err = cmd.Wait(); err != nil {
log.Fatal(err)
}
In addition, you're only checking if the command started, not if it successfully ran.
From the docs for Cmd.Start.
Start starts the specified command but does not wait for it to complete.
The Wait method will return the exit code and release associated resources once the command exits.
You're checking cmd.Start for an error, but not cmd.Wait. Checking the error from cmd.Start only means the command started. If there is an error while the program is running you won't know what it is.
You need to actually use the output of your command. You're not doing that. To do so, use the StdoutPipe method, then you can copy the stdout from your program, into your file.
Related
Code in function to run a fzf against an input, while debugging i discovered my code doesn't return errors, this code runs successfully:
reader := strings.NewReader(listOutput.String())
r, w, _ := os.Pipe()
os.Stdout = w
cmd := exec.Command("fzf", "--multi")
cmd.Stdin = reader
cmd.Stderr = os.Stderr
if err := cmd.Run(); err != nil {
fmt.Println("Couldn't call fzf: %v", err)
}
w.Close()
So i changed the command to something that doesn't exist, but the code still doesn't return "couldn't call command: command not found", just exits.
reader := strings.NewReader(listOutput.String())
r, w, _ := os.Pipe()
os.Stdout = w
cmd := exec.Command("idontexist")
cmd.Stdin = reader
cmd.Stderr = os.Stderr
if err := cmd.Run(); err != nil {
fmt.Println("Couldn't call command: %v", err)
}
w.Close()
I don't have an idea what could be wrong.
cmd.Run() does return an error, and your if block gets properly executed, but since you change the standard output os.Stdout = w you just don't see the result on your console / terminal.
The fmt package writes to the standard output.
If you use the log package, you will see the error as the log package writes to the standard error (which you didn't change):
log.Printf("Couldn't call command: %v", err)
This will output something like (note the default log format includes the timestamp too):
2022/12/07 13:46:19 Couldn't call command: exec: "idontexist": executable file not found in $PATH
Or don't change the standard output.
Also do note that log.Println() and fmt.Println() do not require a format string. Do use log.Printf() and fmt.Printf() when you specify a format string and arguments.
I'm trying to start a command in a detached process so that it can continue after go program exits. I need to redirect the output of the command to a file.
What I need is something like this:
func main() {
command := exec.Command("/tmp/test.sh", ">", "/tmp/out")
if err := command.Start(); err != nil {
fmt.Fprintln(os.Stderr, "Command failed.", err)
os.Exit(1)
}
fmt.Println("Process ID:", command.Process.Pid)
}
Obviously such redirect doesn't work. As I immediately exit from the program after starting the long running command, I cannot open a file and bind it to the Stdout.
Is there any way to achieve such a redirect?
You may start a shell which executes your command / app, and you may redirect its output to a file. The shell will continue to run and execute your script / app even if your Go app exits.
Example:
cmd := exec.Command("sh", "-c", "/tmp/test.sh > /tmp/out")
if err := cmd.Start(); err != nil {
panic(err)
}
fmt.Println("Process ID:", cmd.Process.Pid)
Test it with this simple Go app (replace /tmp/test.sh with the name of the executable binary you compile this into):
package main
import ("fmt"; "time")
func main() {
for i := 0; i < 10; i++ {
fmt.Printf("%d.: %v\n", i, time.Now())
time.Sleep(time.Second)
}
}
This app simply prints a line to the standard output once every second. You can see how the output file is being written e.g. with tail -f /tmp/out.
Note that you may use other shells to execute your scripts to your liking (and to what the test.sh script dictates).
For example to use bash:
cmd := exec.Command("/bin/bash", "-c", "/tmp/test.sh > /tmp/out")
// rest is unchanged
Note that the command to be executed by the shell is passed as a single string argument, and it is not broken down into multiple as you would do it if you were to execute it directly in the command prompt.
Maybe you can try to use this: https://stackoverflow.com/a/28918814/2728768
Opening a file (and os.File implements io.Writer), and then passing it as the command.Stdout could do the trick:
func main() {
command := exec.Command("./tmp/test.sh")
f, err := os.OpenFile("/tmp/out", os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
if err != nil {
fmt.Printf("error opening file: %v", err)
}
defer f.Close()
// On this line you're going to redirect the output to a file
command.Stdout = f
if err := command.Start(); err != nil {
fmt.Fprintln(os.Stderr, "Command failed.", err)
os.Exit(1)
}
fmt.Println("Process ID:", command.Process.Pid)
}
Not sure this could be a viable solution for your case. I've tried it locally and it seems working... remember that your user should be able to create/update the file.
I'm currently trying to execute a simple echo command in Golang on Linux. My code is the following:
cmd = exec.Command("echo", "\"foo 0x50\"", ">", "test.txt")
_, err = cmd.Output()
if err != nil {
fmt.Println(err)
}
But test.txt doesn't appear in my folder (even after compile and run the code). That not the first that that I use this method to execute commands and I never thought that I will be block on an echo command.
So how can I fix this code in order to have "foo 0x50" (with the quotes) in the test.txt?
You can redirect the stdout like this:
// Remove the redirect from command
cmd := exec.Command("echo", "\"foo 0x50\"")
// Make test file
testFile, err := os.Create("test.txt")
if err != nil {
panic(err)
}
defer outfile.Close()
// Redirect the output here (this is the key part)
cmd.Stdout = testFile
err = cmd.Start(); if err != nil {
panic(err)
}
cmd.Wait()
For anyone who wants to pass the complete command in oneline/string ,
func call(command string) {
cmd := exec.Command("sudo", "bash", "-c", command)
cmd.Stdin = os.Stdin
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
err := cmd.Start()
if err != nil {
fmt.Println(err)
}
err1 := cmd.Wait()
if err1 != nil {
fmt.Println(err1)
}
}
The Wait waits for the command to exit and waits for any copying to stdin or copying from stdout or stderr to complete. It closes the pipe after seeing the command exit.
Sources :
https://stackoverflow.com/a/43246464/9892358
https://stackoverflow.com/a/43246464/9892358
https://zetcode.com/golang/exec-command/
I'm trying to run a fairly simple bash command from my Go code. My program writes out an IPTables config file and I need to issue a command to make IPTables refresh from this config. This is very straightforward at the commandline:
/sbin/iptables-restore < /etc/iptables.conf
However, I can't for the life of me figure out how to issue this command with exec.Command(). I tried a few things to accomplish this:
cmd := exec.Command("/sbin/iptables-restore", "<", "/etc/iptables.conf")
// And also
cmd := exec.Command("/sbin/iptables-restore", "< /etc/iptables.conf")
No surprise, neither of those worked. I also tried to feed the filename into the command by piping in the file name to stdin:
cmd := exec.Command("/sbin/iptables-restore")
stdin, err := cmd.StdinPipe()
if err != nil {
log.Fatal(err)
}
err = cmd.Start()
if err != nil {
log.Fatal(err)
}
io.WriteString(stdin, "/etc/iptables.conf")
That doesn't work either, no surprise. I can use stdin to pipe in the contents of the file, but this seems silly when I can just tell iptables-restore what data to go read. So how might I get Go to run the command /sbin/iptables-restore < /etc/iptables.conf?
first read this /etc/iptables.conf file content then write it to cmd.StdinPipe() like this:
package main
import (
"io"
"io/ioutil"
"log"
"os/exec"
)
func main() {
bytes, err := ioutil.ReadFile("/etc/iptables.conf")
if err != nil {
log.Fatal(err)
}
cmd := exec.Command("/sbin/iptables-restore")
stdin, err := cmd.StdinPipe()
if err != nil {
log.Fatal(err)
}
err = cmd.Start()
if err != nil {
log.Fatal(err)
}
_, err = io.WriteString(stdin, string(bytes))
if err != nil {
log.Fatal(err)
}
}
cmd := exec.Command("/usr/sbin/iptables-restore", "--binary", iptablesFilePath)
_, err := cmd.CombinedOutput()
if err != nil {
return err
}
return nil
this work fine on my Raspberry Pi3
The os/exec package does not invoke the system shell, nor does it implement the < redirection syntax typically handled by a shell.
Open the input file and use that file as stdin:
stdin, err := os.Open("/etc/iptables.conf")
if err != nil {
log.Fatal(err)
}
defer stdin.Close()
cmd := exec.Command("/sbin/iptables-restore")
cmd.Stdin = stdin // <-- use open file as stdin
result, err := cmd.CombinedOutput()
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s\n", result)
I need to find a way to read a line from a io.ReadCloser object OR find a way to split a byte array on a "end line" symbol. However I don't know the end line symbol and I can't find it.
My application execs a php script and needs to get the live output from the script and do "something" with it when it gets it.
Here's a small piece of my code:
cmd := exec.Command(prog, args)
/* cmd := exec.Command("ls")*/
out, err := cmd.StdoutPipe()
if err != nil {
fmt.Println(err)
}
err = cmd.Start()
if err != nil {
fmt.Println(err)
}
after this I monitor the out buffer in a go routine. I've tried 2 ways.
1) nr, er := out.Read(buf) where buf is a byte array. the problem here is that I need to brake the array for each new line
2) my second option is to create a new bufio.reader
r := bufio.NewReader(out)
line,_,e := r.ReadLine()
it runs fine if I exec a command like ls, I get the output line by line, but if I exec a php script it immediately get an End Of File error and exits(I'm guessing that's because of the delayed output from php)
EDIT: My problem was I was creating the bufio.Reader inside the go routine whereas if I do it right after the StdoutPipe() like minikomi suggested, it works fine
You can create a reader using bufio, and then read until the next line break character (Note, single quotes to denote character!):
stdout, err := cmd.StdoutPipe()
rd := bufio.NewReader(stdout)
if err := cmd.Start(); err != nil {
log.Fatal("Buffer Error:", err)
}
for {
str, err := rd.ReadString('\n')
if err != nil {
log.Fatal("Read Error:", err)
return
}
fmt.Println(str)
}
If you're trying to read from the reader in a goroutine with nothing to stop the script, it will exit.
Another option is bufio.NewScanner:
package main
import (
"bufio"
"os/exec"
)
func main() {
cmd := exec.Command("go", "env")
out, err := cmd.StdoutPipe()
if err != nil {
panic(err)
}
buf := bufio.NewScanner(out)
cmd.Start()
defer cmd.Wait()
for buf.Scan() {
println(buf.Text())
}
}
https://golang.org/pkg/bufio#NewScanner