Program should be able to get input from stdin on terminal, as follows:
echo foobar | program
However, in the source below for Program, the stdin read blocks if the pipe is omitted:
package main
import (
"fmt"
"os"
)
func main() {
b := make([]byte, 1024)
r := os.Stdin
n, e := r.Read(b)
if e != nil {
fmt.Printf("Err: %s\n", e)
}
fmt.Printf("Res: %s (%d)\n", b, n)
}
So how can Program detect whether something is being piped to it in this manner, and continue execution instead of blocking if not?
... and is it a good idea to do so?
os.Stdin is treated like a file and has permissions. When os.Stdin is open, the perms are 0600, when closed it's 0620.
This code works:
package main
import (
"fmt"
"os"
)
func main() {
stat, _ := os.Stdin.Stat()
fmt.Printf("stdin mode: %v\n", stat.Mode().Perm())
if stat.Mode().Perm() == 0600 {
fmt.Printf("stdin open\n")
return
}
fmt.Printf("stdin close\n")
}
It would be possible if you put the read in a goroutine and use select to do a non-blocking read from the channel in main, but you would then have a race condition where you could get different results depending on whether the data comes in quickly enough. What is the larger problem you're trying to solve?
Related
Currently, I'm using the following to format data from my npm script.
npm run startWin | while IFS= read -r line; do printf '%b\n' "$line"; done | less
It works, but my colleagues do not use Linux. So, I would like to implement while IFS= read -r line; do printf '%b\n' "$line"; done in Go, and use the binary in the pipe.
npm run startWin | magical-go-formater
What I tried
package main
import (
"fmt"
"io/ioutil"
"os"
"strings"
)
func main() {
fi, _ := os.Stdin.Stat() // get the FileInfo struct
if (fi.Mode() & os.ModeCharDevice) == 0 {
bytes, _ := ioutil.ReadAll(os.Stdin)
str := string(bytes)
arr := strings.Fields(str)
for _, v := range arr {
fmt.Println(v)
}
}
Currently the program silences any output from the text-stream.
You want to use bufio.Scanner for tail-type reads. IMHO the checks you're doing on os.Stdin are unnecessary, but YMMV.
See this answer for an example. ioutil.ReadAll() (now deprecated, just use io.ReadAll()) reads up to an error/EOF, but it is not a looping input - that's why you want bufio.Scanner.Scan().
Also - %b will convert any escape sequence in the text - e.g. any \n in a passed line will be rendered as a newline - do you need that? B/c go does not have an equivalent format specifier, AFAIK.
EDIT
So I think that, your ReadAll()-based approach would/could have worked...eventually. I am guessing that you were expecting behavior like you get with bufio.Scanner - the receiving process handles bytes as they are written (it's actually a polling operation - see the standard library source for Scan() to see the grimy details).
But ReadAll() buffers everything read and doesn't return until it finally gets either an error or an EOF. I hacked up an instrumented version of ReadAll() (this is an exact copy of the standard library source with just a little bit of additional instrumentation output), and you can see that it's reading as the bytes are written, but it just doesn't return and yield the contents until the writing process is finished, at which time it closes its end of the pipe (its open filehandle), which generates the EOF:
package main
import (
"fmt"
"io"
"os"
"time"
)
func main() {
// os.Stdin.SetReadDeadline(time.Now().Add(2 * time.Second))
b, err := readAll(os.Stdin)
if err != nil {
fmt.Println("ERROR: ", err.Error())
}
str := string(b)
fmt.Println(str)
}
func readAll(r io.Reader) ([]byte, error) {
b := make([]byte, 0, 512)
i := 0
for {
if len(b) == cap(b) {
// Add more capacity (let append pick how much).
b = append(b, 0)[:len(b)]
}
n, err := r.Read(b[len(b):cap(b)])
//fmt.Fprintf(os.Stderr, "READ %d - RECEIVED: \n%s\n", i, string(b[len(b):cap(b)]))
fmt.Fprintf(os.Stderr, "%s READ %d - RECEIVED %d BYTES\n", time.Now(), i, n)
i++
b = b[:len(b)+n]
if err != nil {
if err == io.EOF {
fmt.Fprintln(os.Stderr, "RECEIVED EOF")
err = nil
}
return b, err
}
}
}
I just hacked up a cheap script to generate the input, simulating something long-running and writing only at periodic intervals, how I'd imagine npm is behaving in your case:
#!/bin/sh
for x in 1 2 3 4 5 6 7 8 9 10
do
cat ./main.go
sleep 10
done
As a side note, I find reading the actual standard library code really helpful...or at least interesting in cases like this.
#Sandy Cash was helpful in stating to use Bufio. I don't know why, if what #Jim said is true, but Bufio worked out and ReadAll() didn't.
Thanks for the help.
The code:
package main
import (
"bufio"
"fmt"
"os"
"strings"
)
func main() {
scanner := bufio.NewScanner(os.Stdin)
for scanner.Scan() {
s := scanner.Text()
arr := strings.Split(s, `\n`)
for _, v := range arr {
fmt.Println(v)
}
}
}
I have a somewhat challenging situation where I need to write into a system command stdin the same stdout it outputs (in another running program), here's an example program that represents what I mean:
package main
import (
"bufio"
"fmt"
"math/rand"
"os"
)
func main() {
rand.Seed(time.Now().Unix())
var greetings []string = []string{"hi", "hola", "bonjour", "hallo", "whats up"}
var greeting string = greetings[rand.Intn(len(greetings))]
fmt.Println(greeting)
reader := bufio.NewReader(os.Stdin)
message, _ := reader.ReadString('\n')
if message == greeting+"\n" {
fmt.Println("nice to meet you!")
} else {
fmt.Println("oops!")
}
}
Since you greet with a random greeting, you have to read the stdout, send it to stdin and also capture if it was the correct answer or not. I've tried with stdinpipes but it freezes waiting for the stdin close since I think that only works for the start of the command run only, so for a running program it hasn't been working for me...
I appreciate any help!
EDIT
I wanted to add sort of what I was trying to do, I've tried without channels as well but it didn't seem to make a difference on the outcome, it just freezes waiting for stdin to close and I need to get first stdout before closing stdin since it consists of it:
package main
import (
"io"
"os/exec"
)
func main() {
cmd := exec.Command("./executable_program")
stdout, _ := cmd.StdoutPipe()
stdin, _ := cmd.StdinPipe()
var c chan []byte = make(chan []byte)
cmd.Start()
go func() {
b, _ := io.ReadAll(stdout)
c <- b
}()
stdin.Write(<-c)
stdin.Close()
cmd.Wait()
}
You can use a pipe to join the stdout to the stdin of the program that you execute:
package main
import (
"io"
"os/exec"
)
func main() {
r, w := io.Pipe()
cmd := exec.Command("<name-of-program-to-run>")
cmd.Stdin = r
cmd.Stdout = w
cmd.Run()
}
To see this in action, first let's prepare a test program to be executed by the program above. This test program simply prints a line to stdout, and then reads each line of stdin and prints it to stdout until stdin is closed.
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
fmt.Fprint(os.Stdout, "priming the pump!\n")
s := bufio.NewScanner(os.Stdin)
for s.Scan() {
line := s.Text()
fmt.Fprint(os.Stdout, line+"\n")
}
}
Then, we modify our initial program to print the bytes traversing through the pipe so we see what's going on.
package main
import (
"fmt"
"io"
"os/exec"
)
func main() {
r, w := io.Pipe()
sr := &readSpy{r: r}
wr := &writeSpy{w: w}
cmd := exec.Command("./test-program")
cmd.Stdin = sr
cmd.Stdout = wr
cmd.Run()
}
type readSpy struct {
r io.Reader
}
func (s *readSpy) Read(d []byte) (int, error) {
size, err := s.r.Read(d)
fmt.Println("readSpy read", string(d[:size]))
return size, err
}
type writeSpy struct {
w io.Writer
}
func (s *writeSpy) Write(d []byte) (int, error) {
size, err := s.w.Write(d)
fmt.Println("writeSpy wrote", string(d[:size]))
return size, err
}
Running the above, you will see the following getting printed in a infinite loop, which makes sense since the priming the pump! string is printed to stdout and fed right back to the stdin of the test program:
writeSpy wrote priming the pump!
readSpy read priming the pump!
...repeated forever...
I'm new to GO and I'm trying to write a small utility in which I would like to execute commands in a loop and read their output. The code works but only the first iteration produces an output. I guess the assignment of stdout in the first iteration somehow blocks the subsequent use of stdout. Can someone explain me how to get around this problem?
(I simplified the code where the IP Addresses come from. I read them from a file but that's not relevant for the Problem.)
package main
import (
"os/exec"
"bufio"
"fmt"
"os"
"strings"
)
var ip_addresses []string
func main() {
ip_addresses = append(ip_addresses,"/server:192.168.100.1")
ip_addresses = append(ip_addresses,"/server:192.168.100.2")
ip_addresses = append(ip_addresses,"/server:192.168.100.3")
for _, eachline := range ip_addresses {
if strings.HasPrefix(eachline, "#") != true {
c, b := exec.Command("query", "user", eachline), new(strings.Builder)
c.Stdout = b
c.Run()
print(b.String())
}
}
}
You should catch the *exec.ExitError exception and log it to prevent the loop from breaking.
for _, eachline := range ip_addresses {
cmd := exec.Command("ping", eachline)
stdout, err := cmd.Output()
if exit, ok := err.(*exec.ExitError); ok {
if status, ok := exit.Sys().(syscall.WaitStatus); ok {
log.Printf("Exit Status: %d", status.ExitStatus())
}
} else {
log.Fatal(err)
}
fmt.Print(string(stdout))
}
I'm trying to read from Stdin in Golang as I'm trying to implement a driver for Erlang. I have the following code:
package main
import (
"fmt"
"os"
"bufio"
"time"
)
func main() {
go func() {
stdout := bufio.NewWriter(os.Stdin)
p := []byte{121,100,125,'\n'}
stdout.Write(p)
}()
stdin := bufio.NewReader(os.Stdin)
values := make([]byte,4,4)
for{
fmt.Println("b")
if read_exact(stdin) > 0 {
stdin.Read(values)
fmt.Println("a")
give_func_write(values)
}else{
continue
}
}
}
func read_exact(r *bufio.Reader) int {
bits := make([]byte,3,3)
a,_ := r.Read(bits)
if a > 0 {
r.Reset(r)
return 1
}
return -1
}
func give_func_write(a []byte) bool {
fmt.Println("Yahu")
return true
}
However it seems that the give_func_write is never reached. I tried to start a goroutine to write to standard input after 2 seconds to test this.
What am I missing here?
Also the line r.Reset(r). Is this valid in go? What I tried to achieve is simply restart the reading from the beginning of the file. Is there a better way?
EDIT
After having played around I was able to find that the code is stuck at a,_ := r.Read(bits) in the read_exact function
I guess that I will need to have a protocol in which I send a \n to
make the input work and at the same time discard it when reading it
No, you don't. Stdin is line-buffered only if it's bound to terminal. You can run your program prog < /dev/zero or cat file | prog.
bufio.NewWriter(os.Stdin).Write(p)
You probably don't want to write to stdin. See "Writing to stdin and reading from stdout" for details.
Well, it's not particular clear for me what you're trying to achieve. I'm assuming, that you just want to read data from stdin by fixed-size chunks. Use io.ReadFull for this. Or if you want to use buffers, you can use Reader.Peek or Scanner to ensure, that specific number of bytes is available. I've changed your program to demonstrate the usage of io.ReadFull:
package main
import (
"fmt"
"io"
"time"
)
func main() {
input, output := io.Pipe()
go func() {
defer output.Close()
for _, m := range []byte("123456") {
output.Write([]byte{m})
time.Sleep(time.Second)
}
}()
message := make([]byte, 3)
_, err := io.ReadFull(input, message)
for err == nil {
fmt.Println(string(message))
_, err = io.ReadFull(input, message)
}
if err != io.EOF {
panic(err)
}
}
You can easily split it in two programs and test it that way. Just change input to os.Stdin.
In Python, for example, I can do the following:
realout = sys.stdout
sys.stdout = StringIO.StringIO()
some_function() # prints to stdout get captured in the StringIO object
result = sys.stdout.getvalue()
sys.stdout = realout
Can you do this in Go?
I agree you should use the fmt.Fprint functions if you can manage it. However, if you don't control the code whose output you're capturing, you may not have that option.
Mostafa's answer works, but if you want to do it without a temporary file you can use os.Pipe. Here's an example that's equivalent to Mostafa's with some code inspired by Go's testing package.
package main
import (
"bytes"
"fmt"
"io"
"os"
)
func print() {
fmt.Println("output")
}
func main() {
old := os.Stdout // keep backup of the real stdout
r, w, _ := os.Pipe()
os.Stdout = w
print()
outC := make(chan string)
// copy the output in a separate goroutine so printing can't block indefinitely
go func() {
var buf bytes.Buffer
io.Copy(&buf, r)
outC <- buf.String()
}()
// back to normal state
w.Close()
os.Stdout = old // restoring the real stdout
out := <-outC
// reading our temp stdout
fmt.Println("previous output:")
fmt.Print(out)
}
This answer is similar to the previous ones but looks cleaner by using io/ioutil.
http://play.golang.org/p/fXpK0ZhXXf
package main
import (
"fmt"
"io/ioutil"
"os"
)
func main() {
rescueStdout := os.Stdout
r, w, _ := os.Pipe()
os.Stdout = w
fmt.Println("Hello, playground") // this gets captured
w.Close()
out, _ := ioutil.ReadAll(r)
os.Stdout = rescueStdout
fmt.Printf("Captured: %s", out) // prints: Captured: Hello, playground
}
I don't recommend this, but you can achieve it with altering os.Stdout. Since this variable is of type os.File, your temporary output should also be a file.
package main
import (
"fmt"
"io/ioutil"
"os"
"path/filepath"
)
func print() {
fmt.Println("output")
}
func main() {
// setting stdout to a file
fname := filepath.Join(os.TempDir(), "stdout")
fmt.Println("stdout is now set to", fname)
old := os.Stdout // keep backup of the real stdout
temp, _ := os.Create(fname) // create temp file
os.Stdout = temp
print()
// back to normal state
temp.Close()
os.Stdout = old // restoring the real stdout
// reading our temp stdout
fmt.Println("previous output:")
out, _ := ioutil.ReadFile(fname)
fmt.Print(string(out))
}
I don't recommend because this is too much hacking, and not very idiomatic in Go. I suggest passing an io.Writer to the functions and writing outputs to that. This is the better way to do almost the same thing.
package main
import (
"bytes"
"fmt"
"io"
"os"
)
func print(w io.Writer) {
fmt.Fprintln(w, "output")
}
func main() {
fmt.Println("print with byes.Buffer:")
var b bytes.Buffer
print(&b)
fmt.Print(b.String())
fmt.Println("print with os.Stdout:")
print(os.Stdout)
}
I think the whole idea is not advisable (race condition) at all, but I guess one can mess with os.Stdout in a way similar/analogical to your example.
Even though the options listed above works, there is a clean approach in modern Go, that makes use of io.Pipe and io.Copy.
package main
import (
"bytes"
"fmt"
"io"
"os"
)
// Your function
func some_function(w *io.PipeWriter) {
defer w.Close()
// Fill pipe writer
fmt.Fprintln(w, "Hello World")
}
// main function
func main() {
// create a pipe reader and writer
pr, pw := io.Pipe()
// pass writer to function
go some_function(pw)
// custom buffer to get standard output of function
var b bytes.Buffer
// create a multi writer that is a combination of
// os.Stdout and variable byte buffer `b`
mw := io.MultiWriter(os.Stdout, &b)
// copies pipe reader content to standard output & custom buffer
_, err := io.Copy(mw, pr)
if err != nil {
if err != io.EOF {
panic(err)
}
}
// use variable
fmt.Println(b.String())
}
The above program works this way:
Create a pipe that gives a reader and writer. It means, if you write something into pipe writer, will be copied to pipe reader by go
Create a MultiWriter with os.Stdout and custom buffer b
some_function(as a go-routine) will write a string into pipe writer
io.Copy will then copy content from pipe reader into multi-writer
os.Stdout will receive the output as well as your custom buffer b
Use buffer b
io package comes with all batteries included to work with io.Reader and io.Writer. No need to use os package, unless files are involved.
Running snippet:
https://goplay.tools/snippet/3NcLVNmbEDd