How can I read from `exec.Cmd` ExtraFiles fd in child process? - go

I read the explanation from golang.org, it says like below.
// ExtraFiles specifies additional open files to be inherited by the
// new process. It does not include standard input, standard output, or
// standard error. If non-nil, entry i becomes file descriptor 3+i.
//
// BUG: on OS X 10.6, child processes may sometimes inherit unwanted fds.
// http://golang.org/issue/2603
ExtraFiles []*os.File
I'm not very understand about it ? For example I have such code below.
cmd := &exec.Cmd{
Path: init,
Args: initArgs,
}
cmd.Stdin = Stdin
cmd.Stdout = Stdout
cmd.Stderr = Stderr
cmd.Dir = Rootfs
cmd.ExtraFiles = []*os.File{childPipe}
Is that mean, since I have written a childpipe in cmd.ExtraFiles = []*os.File{childPipe}, I can use it by writing fd 3 directly.
pipe = os.NewFile(uintptr(3), "pipe")
json.NewEncoder(pipe).Encode(newThing)
Thanks if anyone can give some help!

Correct; you can read from the pipe by creating a new *File whose file descriptor is that of the child pipe. Below is a example of piping data from the child process to the parent:
Parent:
package main
import (
"fmt"
"os/exec"
"os"
"encoding/json"
)
func main() {
init := "child"
initArgs := []string{"hello world"}
r, w, err := os.Pipe()
if err != nil {
panic(err)
}
cmd := exec.Command(init, initArgs...)
cmd.Stdin = os.Stdin
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
cmd.ExtraFiles = []*os.File{w}
if err := cmd.Start(); err != nil {
panic(err)
}
var data interface{}
decoder := json.NewDecoder(r)
if err := decoder.Decode(&data); err != nil {
panic(err)
}
fmt.Printf("Data received from child pipe: %v\n", data)
}
Child:
package main
import (
"os"
"encoding/json"
"strings"
"fmt"
)
func main() {
if len(os.Args) < 2 {
os.Exit(1)
}
arg := strings.ToUpper(os.Args[1])
pipe := os.NewFile(uintptr(3), "pipe")
err := json.NewEncoder(pipe).Encode(arg)
if err != nil {
panic(err)
}
fmt.Println("This message printed to standard output, not to the pipe")
}

Related

Golang io.Copy blocks in internal ReadFrom

I am building a terminal emulator in golang and I was trying to run detached processes from which I can copy output and display it to the user but the io.Copy function blocks and hence I cannot continue to the output part
I looked in the source code and it blocks in the internal ReadFrom method, I cannot understand why this is happening
package main
import (
"bytes"
"fmt"
"io"
"os"
)
func main() {
inputReader, inputWriter, _ := os.Pipe()
outputReader, outputWriter, _ := os.Pipe()
io.Copy(inputWriter, bytes.NewReader([]byte("\n")))
stdin := inputReader
stdout := outputWriter
stderr := outputWriter
var attr = os.ProcAttr{
Dir: "/tmp",
Env: nil,
Files: []*os.File{
stdin,
stdout,
stderr,
},
Sys: nil,
}
process, startProcessErr := os.StartProcess("/usr/bin/ls", []string{"ls"}, &attr)
if startProcessErr != nil {
panic(startProcessErr)
}
if releaseProcessErr := process.Release(); releaseProcessErr != nil {
panic(releaseProcessErr)
}
var output bytes.Buffer
io.Copy(&output, outputReader)
fmt.Println(output)
}
Maybe it is because I release the process but I dont think it should happen
The call io.Copy(&output, outputReader) blocks until read on outputReader returns EOF or some other error. Read on outputReader does not return EOF because the write side of the pipe is still open in the parent process. Fix by closing the writer in the parent process.
...
if releaseProcessErr := process.Release(); releaseProcessErr != nil {
panic(releaseProcessErr)
}
outputWriter.Close() // <-- add this line
var output bytes.Buffer
io.Copy(&output, outputReader)
fmt.Println(output)
...
Use the os/exec package to simplify the code:
cmd := exec.Command("/usr/bin/ls")
cmd.Dir = "/tmp"
output, err := cmd.CombinedOutput()
if err != nil {
log.Fatal(err)
}
fmt.Println(string(output))

return cmd stdout and stderr as string instead of printing to console in golang

I am executing bash commands from a golang application. Now the stdout and stderr go directly to console:
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
But I would like stdout and stderr to be returned as string variables from the runBashCommandAndKillIfTooSlow function without printing to the console immediately.
How to implement this?
The code:
package main
import (
"fmt"
"log"
"os"
"os/exec"
"time"
)
func main() {
ok, outString, errString := runBashCommandAndKillIfTooSlow("ls -la", 2000)
fmt.Println("ok")
fmt.Println(ok)
fmt.Println("outString")
fmt.Println(outString)
fmt.Println("errString")
fmt.Println(errString)
}
/*
run bash command and kill it if it works longer than "killInMilliSeconds" milliseconds
*/
func runBashCommandAndKillIfTooSlow(command string, killInMilliSeconds time.Duration) (okResult bool, stdout, stderr string) {
fmt.Println("running bash command...")
fmt.Println(command)
cmd := exec.Command("sh", "-c", command)
cmd.Stdout = os.Stdout // cmd.Stdout -> stdout
cmd.Stderr = os.Stderr // cmd.Stderr -> stderr
okResult = true
err := cmd.Start()
log.Printf("Waiting for command to finish...")
done := make(chan error, 1)
go func() {
done <- cmd.Wait()
}()
select {
case <-time.After(killInMilliSeconds * time.Millisecond):
if err := cmd.Process.Kill(); err != nil {
log.Fatal("failed to kill: ", err)
okResult = false
}
<-done // allow goroutine to exit
// log.Println("process killed")
case err := <-done:
if err != nil {
log.Printf("process done with error = %v", err)
okResult = false
}
}
if err != nil {
log.Fatal(err)
okResult = false
}
return
}
By the way, the program should keep its ability to kill the bash command if it was too slow (killInMilliSeconds parameter).
Set the output to a strings.Builder (in Go versions 1.10 or later) or a bytes.Buffer
var outbuf, errbuf strings.Builder // or bytes.Buffer
cmd.Stdout = &outbuf
cmd.Stderr = &errbuf
After running the command, you can get the stdout and stderr as a string by calling the Builder.String() method:
stdout := outbuf.String()
stderr := errbuf.String()
You can simplify this quite a bit by using cmd.Run() instead of cmd.Start() to have it automatically wait for it to be finish, and use exec.CommandContext() to have it timeout. This will also output in the correct order, whereas the original program is out of order due to go routines.
Here's the exact same program simplified and using #Mello Marmot's answer:
package main
import (
"bytes"
"fmt"
"log"
"os"
"os/exec"
"time"
"golang.org/x/net/context"
)
func main() {
ctx := context.Background()
ok, outString, errString := runBashCommandAndKillIfTooSlow(ctx, "ls -la", 2000*time.Millisecond)
fmt.Println("ok")
fmt.Println(ok)
fmt.Println("outString")
fmt.Println(outString)
fmt.Println("errString")
fmt.Println(errString)
}
/*
run bash command and kill it if it works longer than "killIn"
*/
func runBashCommandAndKillIfTooSlow(ctx context.Context, command string, killIn time.Duration) (okResult bool, stdout, stderr string) {
fmt.Println("running bash command...")
fmt.Println(command)
ctx, _ = context.WithTimeout(ctx, killIn)
cmd := exec.CommandContext(ctx, "sh", "-c", command)
// Set output to Byte Buffers
var outb, errb bytes.Buffer
cmd.Stdout = &outb
cmd.Stderr = &errb
okResult = true
err := cmd.Run()
stdout = outb.String()
stderr = errb.String()
if err != nil {
log.Fatal(err)
okResult = false
}
return
}
Another option is strings.Builder:
package main
import (
"os/exec"
"strings"
)
func main() {
b := new(strings.Builder)
c := exec.Command("go", "version")
c.Stdout = b
c.Run()
println(b.String() == "go version go1.16.3 windows/amd64\n")
}
https://golang.org/pkg/strings#Builder

How to read from Go additional file descriptors when running a command

Let's say I have a program than outputs things on file descriptor 3; something like this:
package main
import "os"
func main() {
fd3 := os.NewFile(3, "fd3")
fd3.Write([]byte("FOOBAR\n"))
fd3.Close()
}
Now, I want to get the output sent to file descriptor 3 from a Go program:
package main
import (
"bufio"
"fmt"
"os/exec"
"os"
)
func main() {
cmd := exec.Command("./client")
cmd.Stderr = os.Stderr
fd3 := os.NewFile(3, "fd3")
defer fd3.Close()
cmd.ExtraFiles = []*os.File{fd3}
err := cmd.Start()
if err != nil {
panic(err)
}
go func() {
for {
reader := bufio.NewReader(fd3)
line, err := reader.ReadString('\n')
if err != nil {
panic(err)
}
fmt.Print(line)
}
}()
cmd.Wait()
fmt.Println("--- END ---")
}
But that does not work as it outputs the following error:
panic: read fd3: bad file descriptor
I don't understand what's wrong with my code. Anyone willing to help?
os.NewFile doesn't actually open a file descriptor. It's really an API to wrap a fd that was given to you.
look at the godoc: http://golang.org/pkg/os/#Create
(click the name Create, which currently points to this)
I think you want to call os.Create(name) and pass the fd to the child process
or potentiall os.Open / os.OpenFile if you need to set mode and stuff

Communicating with program process using pipes

I want to be able to fully communicate with some programs after spawning them from Golang program. What I already have is spawning process and talking through pipes based on last line read from stdout:
package main
import (
"fmt"
"io"
"log"
"os/exec"
"strings"
)
var stdinPipe io.WriteCloser
var stdoutPipe io.ReadCloser
var err error
func main() {
cmd := &exec.Cmd{
Path: "/Users/seba/Projects/go/src/bootstrap/in",
Args: []string{"program"},
}
stdinPipe, err = cmd.StdinPipe()
if err != nil {
log.Fatal(err)
}
stdoutPipe, err = cmd.StdoutPipe()
if err != nil {
log.Fatal(err)
}
err = cmd.Start()
if err != nil {
log.Fatal(err)
}
var stdoutLines []string
go stdoutManage(stdoutLines, stdoutController)
cmd.Wait()
}
// TODO: imporove as in io.Copy
func stdoutManage(lines []string, manager func(string)) {
buf := make([]byte, 32*1024)
for {
nr, err := stdoutPipe.Read(buf)
if nr > 0 {
thelines := strings.Split(string(buf), "\n")
for _, l := range thelines {
manager(l)
lines = append(lines, l)
}
}
buf = make([]byte, 32*1024) // clear buf
if err != nil {
break
}
}
}
However this approach have problems with programs clearing terminal output and programs which somehow buffer it's stdin or don't use stdin at all (don't know if it's possible).
So the question: is there a portable way of talking with programs (it can be non-Golang solution)?
Problems like this are usually to do with the C library which changes its default buffering mode depending on exactly what stdin / stdout / stderr are.
If stdout is a terminal then buffering is automatically set to line buffered, else it is set to buffered.
This is relevant to you because when you run the programs through a pipe they aren't connected to a terminal and so will have buffering which messes up this sort of use.
To fix, you need to use a pseudo tty which pretends to be a terminal but acts just like a pipe. Here is a library implementing the pty interface which I haven't actually tried but it looks like it does the right thing!

How do you get the output of a system command in Go?

Let's say I want to run 'ls' in a go program, and store the results in a string. There seems to be a few commands to fork processes in the exec and os packages, but they require file arguments for stdout, etc. Is there a way to get the output as a string?
There is an easier way now:
package main
import (
"fmt"
"log"
"os/exec"
)
func main() {
out, err := exec.Command("date").Output()
if err != nil {
log.Fatal(err)
}
fmt.Printf("The date is %s\n", out)
}
Where out is the standard output. It's in the format []byte, but you can change it to string easily with:
string(out)
You can also use CombinedOutput() instead of Output() which returns standard output and standard error.
exec.Command
To get both stdout and stderr into separate strings, you can use byte buffers like so:
cmd := exec.Command("date")
var outb, errb bytes.Buffer
cmd.Stdout = &outb
cmd.Stderr = &errb
err := cmd.Run()
if err != nil {
log.Fatal(err)
}
fmt.Println("out:", outb.String(), "err:", errb.String())
cmd := exec.Command("ls", "-al")
output, _ := cmd.CombinedOutput()
fmt.Println(string(output))
or
cmd := exec.Command(name, arg...)
stdout, err := cmd.StdoutPipe()
cmd.Stderr = cmd.Stdout
if err != nil {
return err
}
if err = cmd.Start(); err != nil {
return err
}
for {
tmp := make([]byte, 1024)
_, err := stdout.Read(tmp)
fmt.Print(string(tmp))
if err != nil {
break
}
}
I used this with a recent version of GO (~1.11)
// CmdExec Execute a command
func CmdExec(args ...string) (string, error) {
baseCmd := args[0]
cmdArgs := args[1:]
log.Debugf("Exec: %v", args)
cmd := exec.Command(baseCmd, cmdArgs...)
out, err := cmd.Output()
if err != nil {
return "", err
}
return string(out), nil
}
// Usage:
// out, err := CmdExec("ls", "/home")
Two options, depending on the paradigm you prefer:
os.ForkExec()
exec.Run()
Use exec.Run, passing Pipe for stdout. Read from the pipe that it returns.
If you are wanting string output, strings.Builder is more efficient [1] than
bytes.Buffer:
package main
import (
"os/exec"
"strings"
)
func main() {
c, b := exec.Command("go", "version"), new(strings.Builder)
c.Stdout = b
c.Run()
print(b.String())
}
https://golang.org/pkg/bytes#Buffer.String
Edit: This answer is obsolete. Please see Fatih Arslan's answer below.
Use exec.Run by specifying Pipe as the stdout (and stderr if you want). It will return cmd, which contains an os.File in the Stdout (and Stderr) fields. Then you can read it using for example ioutil.ReadAll.
Example:
package main
import (
"exec";
"io/ioutil";
)
func main() {
if cmd, e := exec.Run("/bin/ls", nil, nil, exec.DevNull, exec.Pipe, exec.MergeWithStdout); e == nil {
b, _ := ioutil.ReadAll(cmd.Stdout)
println("output: " + string(b))
}
}

Resources