Paging output from Go - go

I'm trying print to stdout from golang using $PAGER or manually invoking more or less to allow the user to easily scroll through a lot of options. How can I achieve this?

You can use the os/exec package to start a process that runs less (or whatever is in $PAGER) and then pipe a string to its standard input. The following worked for me:
func main() {
// Could read $PAGER rather than hardcoding the path.
cmd := exec.Command("/usr/bin/less")
// Feed it with the string you want to display.
cmd.Stdin = strings.NewReader("The text you want to show.")
// This is crucial - otherwise it will write to a null device.
cmd.Stdout = os.Stdout
// Fork off a process and wait for it to terminate.
err := cmd.Run()
if err != nil {
log.Fatal(err)
}
}

I assume you already have output being printed from your program to stdout that you want to have captured and sent to your pager, you don't want to
rewrite the I/O to use another input stream like the other responses require.
You can create an os.Pipe which works the same as running a command with "|less" by attaching one side to your pager and the other side to stdout like this:
// Create a pipe for a pager to use
r, w, err := os.Pipe()
if err != nil {
panic("You probably want to fail more gracefully than this")
}
// Capture STDOUT for the Pager. Keep the old
// value so we can restore it later.
stdout := os.Stdout
os.Stdout = w
// Create the pager process to execute and attach
// the appropriate I/O streams.
pager := exec.Command("less")
pager.Stdin = r
pager.Stdout = stdout // the pager uses the original stdout, not the pipe
pager.Stderr = os.Stderr
// Defer a function that closes the pipe and invokes
// the pager, then restores os.Stdout after this function
// returns and we've finished capturing output.
//
// Note that it's very important that the pipe is closed,
// so that EOF is sent to the pager, otherwise weird things
// will happen.
defer func() {
// Close the pipe
w.Close()
// Run the pager
if err := pager.Run(); err != nil {
fmt.Fprintln(os.Stderr, err)
}
// restore stdout
os.Stdout = stdout
}()

Here is a somewhat naive cat example that uses $PAGER when set.
package main
import (
"io"
"log"
"os"
"os/exec"
)
func main() {
var out io.WriteCloser
var cmd *exec.Cmd
if len(os.Args) != 2 {
log.Fatal("Wrong number of args: gcat <file>")
}
fileName := os.Args[1]
file, err := os.Open(fileName)
if err != nil {
log.Fatal("Error opening file: ", err)
}
pager := os.Getenv("PAGER")
if pager != "" {
cmd = exec.Command(pager)
var err error
out, err = cmd.StdinPipe()
if err != nil {
log.Fatal(err)
}
cmd.Stdout = os.Stdout
if err := cmd.Start(); err != nil {
log.Fatal("Unable to start $PAGER: ", err)
}
} else {
out = os.Stdout
}
_, err = io.Copy(out, file)
if err != nil {
log.Fatal(err)
}
file.Close()
out.Close()
if cmd != nil {
if err := cmd.Wait(); err != nil {
log.Fatal("Error waiting for cmd: ", err)
}
}
}

This version creates an io.Writer called pager for all the output that you want paged (you can assign it to os.Stdout if you like) and correctly closes that and waits for the $PAGER when main() returns.
import (
"fmt"
"io"
"log"
"os"
"os/exec"
)
var pager io.WriteCloser
func main() {
var cmd *exec.Cmd
cmd, pager = runPager()
defer func() {
pager.Close()
cmd.Wait()
}()
fmt.Fprintln(pager, "Hello, 世界")
}
func runPager() (*exec.Cmd, io.WriteCloser) {
pager := os.Getenv("PAGER")
if pager == "" {
pager = "more"
}
cmd := exec.Command(pager)
out, err := cmd.StdinPipe()
if err != nil {
log.Fatal(err)
}
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
if err := cmd.Start(); err != nil {
log.Fatal(err)
}
return cmd, out
}

Related

Reading os.OpenFile in Golang while still being written?

I have code that is writing to a logfile while executing a system command. E.g.
logfile, err := os.OpenFile(THIS_LOG_FILE, os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0600)
if err != nil {
return err
}
cmd.Stderr = logfile
cmd.Stdout = logfile
go func() {
err := cmd.Run()
if err != nil {
// WANT TO LOG ERROR HERE
}
}()
At the "// WANT TO LOG" line, I'd like to output the content to the standard logger, in addition to the previously assigned logfile destination. Is there a way to capture this in memory? Or should I just write everything to an in-memory buffer and flush at the end?
To clarify, in capturing the output of the command in memory, I can parse it and take action in the running program (handling errors/etc). When I write to the log file, that information is lost.
My issue is that, theoretically, I could read that back in from the file I just wrote, but that seems wasteful (and prone to failure if the command failed).
If I understand correctly, you want to write the content of stdout/stderror to a file while executing a shell command.
Since stdout and stderror are implemented the ReadCloser interface, you can merge them by io.MultiReader and perform io.Copy from source to destination.
The following snippet implements the pipeline
package main
import (
"io"
"log"
"os"
"os/exec"
)
func main() {
// prepare the command
cmd := exec.Command("your-shell-command.sh")
// get the stdout and stderr stream
erc, err := cmd.StderrPipe()
if err != nil {
log.Fatalln("Failed to get stderr reader: ", err)
}
orc, err := cmd.StdoutPipe()
if err != nil {
log.Fatalln("Failed to get stdout reader: ", err)
}
// combine stdout and stderror ReadCloser
rc := io.MultiReader(erc, orc)
// Prepare the writer
f, err := os.Create("output.log")
if err != nil {
log.Fatalln("Failed to create file")
}
defer f.Close()
// Command.Start starts a new go routine
if err := cmd.Start(); err != nil {
log.Println("Failed to start the command")
}
// add the TeeReader.
var buf bytes.Buffer
tr := io.TeeReader(rc, &buf)
if _, err := io.Copy(f, tr); err != nil {
logger.Fatalf("Failed to stream to file: %s", err)
}
if err := cmd.Wait(); err != nil {
log.Println("Failed to wait the command to execute: ", err)
}
}

continuously reading from exec.Cmd output

Guys I am trying pick new lines as they come from command output, but always I end up doing it synchronous way (I have to wait until script is finished). I tired to use fsnotify but it is working only with regular files, do you have any idea how it can be done ?
package main
import (
"fmt"
"os/exec"
"bytes"
"os"
)
func main() {
cmd := exec.Command("scripts/long_script")
output := new(bytes.Buffer)
cmd.Stdout = output
cmd.Stderr = output
if err := cmd.Start(); err != nil{ // after Start program is continued and script is executing in background
fmt.Printf("Failed to start " + err.Error())
os.Exit(1)
}
fmt.Printf(" Before WAIT %s \n", output.String()) // script is writing but nothing can be read from output
cmd.Wait()
fmt.Printf(" After Wait %s \n", output.String()) // if we wait to finish execution, we can read all output
}
You should use os.StdoutPipe()
func main() {
for i := 10; i < 20; i++ {
go printName(`My name is Bob, I am ` + strconv.Itoa(i) + ` years old`)
// Adding delay so as to see incremental output
time.Sleep(60 * time.Millisecond)
}
// Adding delay so as to let program complete
// Please use channels or wait groups
time.Sleep(100 * time.Millisecond)
}
func printName(jString string) {
cmd := exec.Command("echo", "-n", jString)
cmdReader, err := cmd.StdoutPipe()
if err != nil {
log.Fatal(err)
}
scanner := bufio.NewScanner(cmdReader)
go func() {
for scanner.Scan() {
fmt.Println(scanner.Text())
}
}()
if err := cmd.Start(); err != nil {
log.Fatal(err)
}
if err := cmd.Wait(); err != nil {
log.Fatal(err)
}
}
sources that helped me:
nathanleclaire.com
blog.kowalczyk.info
eventually I managed to do it with []bytes
stdout, err := cmd.StdoutPipe()
buff := make([]byte,10)
var n int
for err == nil {
n,err = stdout.Read(buff)
if n > 0{
fmt.Printf("taken %d chars %s",n,string(buff[:n]))
}
}
cmd.Wait()
if cmd.ProcessState.Success() {. // ProcessState is set after Wait
fmt.Println("Script success")
} else {
fmt.Println("Script failed")
}

Pipe to exec'ed process

My Go application outputs some amounts of text data and I need to pipe it to some external command (e.g. less). I haven't find any way to pipe this data to syscall.Exec'ed process.
As a workaround I write that text data to a temporary file and then use that file as an argument to less:
package main
import (
"io/ioutil"
"log"
"os"
"os/exec"
"syscall"
)
func main() {
content := []byte("temporary file's content")
tmpfile, err := ioutil.TempFile("", "example")
if err != nil {
log.Fatal(err)
}
defer os.Remove(tmpfile.Name()) // Never going to happen!
if _, err := tmpfile.Write(content); err != nil {
log.Fatal(err)
}
if err := tmpfile.Close(); err != nil {
log.Fatal(err)
}
binary, err := exec.LookPath("less")
if err != nil {
log.Fatal(err)
}
args := []string{"less", tmpfile.Name()}
if err := syscall.Exec(binary, args, os.Environ()); err != nil {
log.Fatal(err)
}
}
It works but leaves a temporary file on a file system, because syscall.Exec replaces the current Go process with another (less) one and deferred os.Remove won't run. Such behaviour is not desirable.
Is there any way to pipe some data to an external process without leaving any artefacts?
You should be using os/exec to build an exec.Cmd to execute, then you could supply any io.Reader you want as the stdin for the command.
From the example in the documentation:
cmd := exec.Command("tr", "a-z", "A-Z")
cmd.Stdin = strings.NewReader("some input")
var out bytes.Buffer
cmd.Stdout = &out
err := cmd.Run()
if err != nil {
log.Fatal(err)
}
fmt.Printf("in all caps: %q\n", out.String())
If you want to write directly to the command's stdin, then you call cmd.StdInPipe to get an io.WriteCloser you can write to.
If you really need to exec the process in place of your current one, you can simply remove the file before exec'ing, and provide that file descriptor as stdin for the program.
content := []byte("temporary file's content")
tmpfile, err := ioutil.TempFile("", "example")
if err != nil {
log.Fatal(err)
}
os.Remove(tmpfile.Name())
if _, err := tmpfile.Write(content); err != nil {
log.Fatal(err)
}
tmpfile.Seek(0, 0)
err = syscall.Dup2(int(tmpfile.Fd()), syscall.Stdin)
if err != nil {
log.Fatal(err)
}
binary, err := exec.LookPath("less")
if err != nil {
log.Fatal(err)
}
args := []string{"less"}
if err := syscall.Exec(binary, args, os.Environ()); err != nil {
log.Fatal(err)
}

Golang - Copy exec output to a buffer

I'm writing a function that exec's a program and returns stdout and stderr. It also has the option to display the output to the console. I'm clearly not waiting on something, as if I run the function twice in a row, the outputs are different. Here's a sample program, replace the dir var with a dir with a lot of files to fill up the buffers:
func main() {
dir := "SOMEDIRECTORYWITHALOTOFFILES"
out, err := run("ls -l "+dir, true)
if err != nil {
log.Fatalf("run returned %s", err)
}
log.Printf("Out: %s", out)
out2, err := run("ls -l "+dir, false)
if err != nil {
log.Fatalf("run returned %s", err)
}
log.Printf("Out2: %s", out2)
if out != out2 {
log.Fatalf("Out mismatch")
}
}
func run(cmd string, displayOutput bool) (string, error) {
var command *exec.Cmd
command = exec.Command("/bin/sh", "-c", cmd)
var output bytes.Buffer
stdout, err := command.StdoutPipe()
if err != nil {
return "", fmt.Errorf("Unable to setup stdout for command: %v", err)
}
go func() {
if displayOutput == true {
w := io.MultiWriter(os.Stdout, &output)
io.Copy(w, stdout)
} else {
output.ReadFrom(stdout)
}
}()
stderr, err := command.StderrPipe()
if err != nil {
return "", fmt.Errorf("Unable to setup stderr for command: %v", err)
}
go func() {
if displayOutput == true {
w := io.MultiWriter(os.Stderr, &output)
io.Copy(w, stderr)
} else {
output.ReadFrom(stderr)
}
}()
err = command.Run()
if err != nil {
return "", err
}
return output.String(), nil
}
Here is a simplified and working revision of your example. Note that the test command was swapped out so that I could test within Windows and that your error checks have been omitted only for brevity.
The key change is that a sync.WaitGroup is preventing the run function from printing the output and returning until the goroutine has indicated that it's finished.
func main() {
dir := "c:\\windows\\system32"
command1 := exec.Command("cmd", "/C", "dir", "/s", dir)
command2 := exec.Command("cmd", "/C", "dir", "/s", dir)
out1, _ := run(command1)
out2, _ := run(command2)
log.Printf("Length [%d] vs [%d]\n", len(out1), len(out2))
}
func run(cmd *exec.Cmd) (string, error) {
var output bytes.Buffer
var waitGroup sync.WaitGroup
stdout, _ := cmd.StdoutPipe()
writer := io.MultiWriter(os.Stdout, &output)
waitGroup.Add(1)
go func() {
defer waitGroup.Done()
io.Copy(writer, stdout)
}()
cmd.Run()
waitGroup.Wait()
return output.String(), nil
}
I see some problems:
You should be waiting for the goroutines to finish (e.g., using
sync.WaitGroup).
You're accessing output concurrently in two
goroutines, which is not safe.
You could collect stdout and stderr in two separate buffers and return them separately, if that works for what you're trying to do.

Interact with external application from within code

I need to be able to run an external application and interact with it as though I was manually running it from the command-line. All the examples I find only deal with running the program and capturing the output.
Below is a very simple example that I hope illustrates what I am trying to accomplish.
package main
import (
"fmt"
"log"
"os/exec"
)
func main() {
cmd := exec.Command("rm", "-i", "somefile.txt")
out, err := cmd.CombinedOutput()
if err != nil {
log.Fatal(err)
}
if string(out) == "Remove file 'somefile.txt'?" {
// send the response 'y' back to the rm process
}
// program completes normally...
}
I've tried to tweak various examples that I've found to accomplish this with zero success. It seems that even though 'rm' is waiting for a response, Go closes the process.
Any examples, articles, or advice you can provide would be greatly appreciated. Many thanks in advance.
You have two possibilities. First is to use ReadLine() but that works only if application output is full lines, and you can wait for \n. This is not the case with rm, so you have to develop a custom SplitFunction for Scanner. Both versions can be found below.
Please note that you can not use CombinedOutput, as it can not be Scanned. You have to use the pipes.
package main
import (
"bufio"
//"fmt"
"log"
"os/exec"
)
func main() {
cmd := exec.Command("rm", "-i", "somefile.txt")
// Stdout + stderr
out, err := cmd.StderrPipe() // rm writes the prompt to err
if err != nil {
log.Fatal(err)
}
r := bufio.NewReader(out)
// Stdin
in, err := cmd.StdinPipe()
if err != nil {
log.Fatal(err)
}
defer in.Close()
// Start the command!
err = cmd.Start()
if err != nil {
log.Fatal(err)
}
line, _, err := r.ReadLine()
for err != nil {
if string(line) == "Remove file 'somefile.txt'?" {
in.Write([]byte("y\n"))
}
line, _, err = r.ReadLine()
}
// program completes normally...s
}
This is a second version with the scanner, and it uses both \n and ? as line delimiters:
package main
import (
"bufio"
"bytes"
"fmt"
"log"
"os/exec"
)
// Ugly hack, this is bufio.ScanLines with ? added as an other delimiter :D
func new_scanner(data []byte, atEOF bool) (advance int, token []byte, err error) {
if atEOF && len(data) == 0 {
return 0, nil, nil
}
if i := bytes.IndexByte(data, '\n'); i >= 0 {
// We have a full newline-terminated line.
fmt.Printf("nn\n")
return i + 1, data[0:i], nil
}
if i := bytes.IndexByte(data, '?'); i >= 0 {
// We have a full ?-terminated line.
return i + 1, data[0:i], nil
}
// If we're at EOF, we have a final, non-terminated line. Return it.
if atEOF {
return len(data), data, nil
}
// Request more data.
return 0, nil, nil
}
func main() {
cmd := exec.Command("rm", "-i", "somefile.txt")
// Stdout + stderr
out, err := cmd.StderrPipe() // Again, rm writes prompts to stderr
if err != nil {
log.Fatal(err)
}
scanner := bufio.NewScanner(out)
scanner.Split(new_scanner)
// Stdin
in, err := cmd.StdinPipe()
if err != nil {
log.Fatal(err)
}
defer in.Close()
// Start the command!
err = cmd.Start()
if err != nil {
log.Fatal(err)
}
// Start scanning
for scanner.Scan() {
line := scanner.Text()
if line == "rm: remove regular empty file ‘somefile.txt’" {
in.Write([]byte("y\n"))
}
}
// Report scanner's errors
if err := scanner.Err(); err != nil {
log.Fatal(err)
}
// program completes normally...s
}

Resources