I have a large buffer (buffer []byte) that I would like to print to stdout but pipe through a pager like less or more. Kind of like the man command. I don't want to write the buffer to tmp file first or make the user manually pipe the output to a pager on the command line.
I can find examples of how to pipe the output of one command to another, but nothing starting with an internal buffer.
Any ideas?
Thanks.
In order to pipe to a pager, you can do something like this:
package main
import (
"fmt"
"io"
"os"
"os/exec"
)
func main() {
// declare your pager
cmd := exec.Command("less")
// create a pipe (blocking)
r, stdin := io.Pipe()
// Set your i/o's
cmd.Stdin = r
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
// Create a blocking chan, Run the pager and unblock once it is finished
c := make(chan struct{})
go func() {
defer close(c)
cmd.Run()
}()
// Pass anything to your pipe
fmt.Fprintf(stdin, "hello world\n")
// Close stdin (result in pager to exit)
stdin.Close()
// Wait for the pager to be finished
<-c
}
Sounds like what you need is an Encoder. Are you using a package for the pager? If so you might want to look for an Encoder in the package, or create your own if one isn't provided.
Here is an example of how you could use a JSON Encoder to achieve something similar to what you're trying to do:
b := []byte(`{ ... some json object ... }`)
json_encoder := json.NewEncoder(os.Stdout)
json_encoder.Encode(b)
In this example, the JSON encoder accepts the []byte and does all the work to encode it into a JSON document and write to the provided io.writer. If you're using a package and it doesn't provide an encoder, you can get ideas on how to write one by looking into the JSON Encoder source code to create your own.
Related
How to read/manipulate the input from a connection that is passed to a command stdin?
For example, given the following code.
c, _ := net.Dial("tcp", somehost)
cmd := exec.Command("/bin/sh")
cmd.Stdin, cmd.Stdout, cmd.Stderr = c, c, c
cmd.Run()
How would it be possible to reverse the string from the connection before it is passed to the cmd.Stdin or how could I parse the string and not pass it on to cmd.Stdin?
Ive considered reading from the connection with bufio and then passing it to Command second argument, the params, but I was hoping for a better solution that does not require me to handle all the different cases for args input in a command, but instead just passing it on to Stdin after analysing the input
Ok since you mentioned in comments that "my real issue is how to intercept the input from the connection, parse it and parse it to the Stdin of the command. Seems when I do cmd.Run() I block and hence cant really continously parse"
Here is how I will do it:
import (
"io"
"os"
"os/exec"
)
func main() {
//All errors are not checked
cmd := exec.Command("/bin/sh")
cmdStdin, _ := cmd.StdinPipe()
go func() {
defer cmdStdin.Close()
//here you will need to loop on reading the connection,
//for simplicity lets assume you do that & receive data
//let says you got ls from connection
cmdStdin.Write([]byte("ls\n"))
}()
cmdStdout, _ := cmd.StdoutPipe()
go io.Copy(os.Stdout, cmdStdout)
cmd.Run()
}
I started learning and playing around with Go to see what it is like to make some more complex console/cli type tools instead of using shells or Python. I want to execute commands and display the output. I figured out how to print the output like this:
out, err := exec.Command("pwd").Output()
print(string(out))
Is there a way to execute the commands and have it default to stdout like a shell script, or do I need to make a helper function for this?
Update: After getting IntelliJ and the Go plugin, I poked around in the Go source and agree there is currently no way to do with without a helper method.
It is not possible to reuse a Cmd object as per this comment in the exec.go source code:
// A Cmd cannot be reused after calling its Run, Output or CombinedOutput
// methods.
I did incorporate the stdout option into my own helper, including other options like shell integration. I will try turn that into open source if I can make it useful. An interesting first day of Go.
The solution
Actually, it is pretty easy. You can set the stdout of the command to os.Stdout and Bob's your uncle:
package main
import (
"os"
"os/exec"
)
func main() {
cmd := exec.Command("pwd")
cmd.Stdout = os.Stdout
err := cmd.Run()
if err != nil {
panic(err)
}
}
What's happening here?
By default, the output of a command is stored in a bytes.Buffer if cmd.Stdout is not set to another io.Writer. The call of cmd.Output() then runs the command and saves the output to said buffer.
Since os.Stdout implements io.Writer interface, we simply set cmd.Stdout to be os.Stdout. Now when .Run() is called, the output of the command gets written to the io.Writer defined in cmd.Stdout, which happens to be os.Stdout and the output gets written in the shell.
EDIT: As per comment, if all commands should write to os.Stdout, there of course is no way to prevent some helper. I'd do it like this:
package main
import (
"os"
"os/exec"
)
func CmdToStdout( c string ) (err error){
cmd := exec.Command(c)
cmd.Stdout = os.Stdout
err = cmd.Run()
return
}
func main() {
err := CmdToStdout("pwd")
if err != nil {
panic(err)
}
}
You have to create a helper if you need this often (and 5 lines looks too much). Based on the documentation this is a recommended way:
package main
import (
"fmt"
"log"
"os/exec"
)
func main() {
out, err := exec.Command("date").Output()
if err != nil {
log.Fatal(err)
}
fmt.Printf("The date is %s\n", out)
}
This question is similar to Golang - Copy Exec output to Log except it is concerned with the buffering of output from exec commands.
I have the following test program:
package main
import (
"fmt"
"log"
"os/exec"
)
func main() {
cmd := exec.Command("python", "inf_loop.py")
var out outstream
cmd.Stdout = out
if err := cmd.Start(); err != nil {
log.Fatal(err)
}
fmt.Println(cmd.Wait())
}
type outstream struct{}
func (out outstream) Write(p []byte) (int, error) {
fmt.Println(string(p))
return len(p), nil
}
inf_loop.py, which the above refers to, simply contains:
print "hello"
while True:
pass
The go program hangs when I run it and doesn't output anything, but if I use os.Stdout instead of out then it outputs "hello" before it hangs. Why is there a discrepancy between the two io.Writers and how can it be fixed?
Some more diagnostic information:
When the loop is removed from inf_loop.py then "hello" is output from both programs, as expected.
When using yes as the program instead of the python script and outputting len(p) in outstream.Write then there is output, and the output is usually 16384 or 32768. This indicates to me that this is a buffering issue, as I originally anticipated, but I still don't understand why the outstream structure is being blocked by buffering but os.Stdout isn't. One possibility is that the behaviour is the result of the way that exec passes the io.Writer directly to os.StartProcess if it is an os.File (see source for details), otherwise it creates an os.Pipe() between the process and the io.Writer, and this pipe may be causing the buffering. However, the operation and possible buffering of os.Pipe() is too low-level for me to investigate.
Python buffers stdout by default. Try this program:
import sys
print "hello"
sys.stdout.flush()
while True:
pass
or run Python with unbuffered stdout and stderr:
cmd := exec.Command("python", "-u", "foo.py")
Note the -u flag.
You see different results when using cmd.Stout = os.Stdout because Python uses line buffering when stdout is a terminal.
I'm new to golang and I'm having problem understanding go's io.Pipe. Is this similar to node.js' .pipe? And how should I use it? Is it possible to use it with 1 read file and a write file?
Thank in advance guys.
No, they are not precisely similar. io.Copy(dat io.Writer, src io.Reader) is quite enough to read and write files, like this:
input := bufio.NewReader(os.Stdin)
output := bufio.NewWriter(os.Stdout) // buffer output like C stdlib
io.Copy(output, input) // copy entire file
output.Flush()
io.Pipe() (*PipeReader, *PipeWriter) will produce piped Reader and Writer for you when you have not them but code expect them, like this:
type id struct{
name string
age int
}
payload := id{"John", 25}
requestBody, jsonPayload := io.Pipe()
request := http.NewRequest("POST". "http://www.example.com", requestBody) // NewRequest expect io.Reader
encoder := json.NewEncoder(jsonPayload) // NewEncoder expect io.Writer
err := encoder.Encode(payload)
response, err := client.Do(request)
In Python, for example, I can do the following:
realout = sys.stdout
sys.stdout = StringIO.StringIO()
some_function() # prints to stdout get captured in the StringIO object
result = sys.stdout.getvalue()
sys.stdout = realout
Can you do this in Go?
I agree you should use the fmt.Fprint functions if you can manage it. However, if you don't control the code whose output you're capturing, you may not have that option.
Mostafa's answer works, but if you want to do it without a temporary file you can use os.Pipe. Here's an example that's equivalent to Mostafa's with some code inspired by Go's testing package.
package main
import (
"bytes"
"fmt"
"io"
"os"
)
func print() {
fmt.Println("output")
}
func main() {
old := os.Stdout // keep backup of the real stdout
r, w, _ := os.Pipe()
os.Stdout = w
print()
outC := make(chan string)
// copy the output in a separate goroutine so printing can't block indefinitely
go func() {
var buf bytes.Buffer
io.Copy(&buf, r)
outC <- buf.String()
}()
// back to normal state
w.Close()
os.Stdout = old // restoring the real stdout
out := <-outC
// reading our temp stdout
fmt.Println("previous output:")
fmt.Print(out)
}
This answer is similar to the previous ones but looks cleaner by using io/ioutil.
http://play.golang.org/p/fXpK0ZhXXf
package main
import (
"fmt"
"io/ioutil"
"os"
)
func main() {
rescueStdout := os.Stdout
r, w, _ := os.Pipe()
os.Stdout = w
fmt.Println("Hello, playground") // this gets captured
w.Close()
out, _ := ioutil.ReadAll(r)
os.Stdout = rescueStdout
fmt.Printf("Captured: %s", out) // prints: Captured: Hello, playground
}
I don't recommend this, but you can achieve it with altering os.Stdout. Since this variable is of type os.File, your temporary output should also be a file.
package main
import (
"fmt"
"io/ioutil"
"os"
"path/filepath"
)
func print() {
fmt.Println("output")
}
func main() {
// setting stdout to a file
fname := filepath.Join(os.TempDir(), "stdout")
fmt.Println("stdout is now set to", fname)
old := os.Stdout // keep backup of the real stdout
temp, _ := os.Create(fname) // create temp file
os.Stdout = temp
print()
// back to normal state
temp.Close()
os.Stdout = old // restoring the real stdout
// reading our temp stdout
fmt.Println("previous output:")
out, _ := ioutil.ReadFile(fname)
fmt.Print(string(out))
}
I don't recommend because this is too much hacking, and not very idiomatic in Go. I suggest passing an io.Writer to the functions and writing outputs to that. This is the better way to do almost the same thing.
package main
import (
"bytes"
"fmt"
"io"
"os"
)
func print(w io.Writer) {
fmt.Fprintln(w, "output")
}
func main() {
fmt.Println("print with byes.Buffer:")
var b bytes.Buffer
print(&b)
fmt.Print(b.String())
fmt.Println("print with os.Stdout:")
print(os.Stdout)
}
I think the whole idea is not advisable (race condition) at all, but I guess one can mess with os.Stdout in a way similar/analogical to your example.
Even though the options listed above works, there is a clean approach in modern Go, that makes use of io.Pipe and io.Copy.
package main
import (
"bytes"
"fmt"
"io"
"os"
)
// Your function
func some_function(w *io.PipeWriter) {
defer w.Close()
// Fill pipe writer
fmt.Fprintln(w, "Hello World")
}
// main function
func main() {
// create a pipe reader and writer
pr, pw := io.Pipe()
// pass writer to function
go some_function(pw)
// custom buffer to get standard output of function
var b bytes.Buffer
// create a multi writer that is a combination of
// os.Stdout and variable byte buffer `b`
mw := io.MultiWriter(os.Stdout, &b)
// copies pipe reader content to standard output & custom buffer
_, err := io.Copy(mw, pr)
if err != nil {
if err != io.EOF {
panic(err)
}
}
// use variable
fmt.Println(b.String())
}
The above program works this way:
Create a pipe that gives a reader and writer. It means, if you write something into pipe writer, will be copied to pipe reader by go
Create a MultiWriter with os.Stdout and custom buffer b
some_function(as a go-routine) will write a string into pipe writer
io.Copy will then copy content from pipe reader into multi-writer
os.Stdout will receive the output as well as your custom buffer b
Use buffer b
io package comes with all batteries included to work with io.Reader and io.Writer. No need to use os package, unless files are involved.
Running snippet:
https://goplay.tools/snippet/3NcLVNmbEDd