I'm trying to read from Stdin in Golang as I'm trying to implement a driver for Erlang. I have the following code:
package main
import (
"fmt"
"os"
"bufio"
"time"
)
func main() {
go func() {
stdout := bufio.NewWriter(os.Stdin)
p := []byte{121,100,125,'\n'}
stdout.Write(p)
}()
stdin := bufio.NewReader(os.Stdin)
values := make([]byte,4,4)
for{
fmt.Println("b")
if read_exact(stdin) > 0 {
stdin.Read(values)
fmt.Println("a")
give_func_write(values)
}else{
continue
}
}
}
func read_exact(r *bufio.Reader) int {
bits := make([]byte,3,3)
a,_ := r.Read(bits)
if a > 0 {
r.Reset(r)
return 1
}
return -1
}
func give_func_write(a []byte) bool {
fmt.Println("Yahu")
return true
}
However it seems that the give_func_write is never reached. I tried to start a goroutine to write to standard input after 2 seconds to test this.
What am I missing here?
Also the line r.Reset(r). Is this valid in go? What I tried to achieve is simply restart the reading from the beginning of the file. Is there a better way?
EDIT
After having played around I was able to find that the code is stuck at a,_ := r.Read(bits) in the read_exact function
I guess that I will need to have a protocol in which I send a \n to
make the input work and at the same time discard it when reading it
No, you don't. Stdin is line-buffered only if it's bound to terminal. You can run your program prog < /dev/zero or cat file | prog.
bufio.NewWriter(os.Stdin).Write(p)
You probably don't want to write to stdin. See "Writing to stdin and reading from stdout" for details.
Well, it's not particular clear for me what you're trying to achieve. I'm assuming, that you just want to read data from stdin by fixed-size chunks. Use io.ReadFull for this. Or if you want to use buffers, you can use Reader.Peek or Scanner to ensure, that specific number of bytes is available. I've changed your program to demonstrate the usage of io.ReadFull:
package main
import (
"fmt"
"io"
"time"
)
func main() {
input, output := io.Pipe()
go func() {
defer output.Close()
for _, m := range []byte("123456") {
output.Write([]byte{m})
time.Sleep(time.Second)
}
}()
message := make([]byte, 3)
_, err := io.ReadFull(input, message)
for err == nil {
fmt.Println(string(message))
_, err = io.ReadFull(input, message)
}
if err != io.EOF {
panic(err)
}
}
You can easily split it in two programs and test it that way. Just change input to os.Stdin.
Related
Currently, I'm using the following to format data from my npm script.
npm run startWin | while IFS= read -r line; do printf '%b\n' "$line"; done | less
It works, but my colleagues do not use Linux. So, I would like to implement while IFS= read -r line; do printf '%b\n' "$line"; done in Go, and use the binary in the pipe.
npm run startWin | magical-go-formater
What I tried
package main
import (
"fmt"
"io/ioutil"
"os"
"strings"
)
func main() {
fi, _ := os.Stdin.Stat() // get the FileInfo struct
if (fi.Mode() & os.ModeCharDevice) == 0 {
bytes, _ := ioutil.ReadAll(os.Stdin)
str := string(bytes)
arr := strings.Fields(str)
for _, v := range arr {
fmt.Println(v)
}
}
Currently the program silences any output from the text-stream.
You want to use bufio.Scanner for tail-type reads. IMHO the checks you're doing on os.Stdin are unnecessary, but YMMV.
See this answer for an example. ioutil.ReadAll() (now deprecated, just use io.ReadAll()) reads up to an error/EOF, but it is not a looping input - that's why you want bufio.Scanner.Scan().
Also - %b will convert any escape sequence in the text - e.g. any \n in a passed line will be rendered as a newline - do you need that? B/c go does not have an equivalent format specifier, AFAIK.
EDIT
So I think that, your ReadAll()-based approach would/could have worked...eventually. I am guessing that you were expecting behavior like you get with bufio.Scanner - the receiving process handles bytes as they are written (it's actually a polling operation - see the standard library source for Scan() to see the grimy details).
But ReadAll() buffers everything read and doesn't return until it finally gets either an error or an EOF. I hacked up an instrumented version of ReadAll() (this is an exact copy of the standard library source with just a little bit of additional instrumentation output), and you can see that it's reading as the bytes are written, but it just doesn't return and yield the contents until the writing process is finished, at which time it closes its end of the pipe (its open filehandle), which generates the EOF:
package main
import (
"fmt"
"io"
"os"
"time"
)
func main() {
// os.Stdin.SetReadDeadline(time.Now().Add(2 * time.Second))
b, err := readAll(os.Stdin)
if err != nil {
fmt.Println("ERROR: ", err.Error())
}
str := string(b)
fmt.Println(str)
}
func readAll(r io.Reader) ([]byte, error) {
b := make([]byte, 0, 512)
i := 0
for {
if len(b) == cap(b) {
// Add more capacity (let append pick how much).
b = append(b, 0)[:len(b)]
}
n, err := r.Read(b[len(b):cap(b)])
//fmt.Fprintf(os.Stderr, "READ %d - RECEIVED: \n%s\n", i, string(b[len(b):cap(b)]))
fmt.Fprintf(os.Stderr, "%s READ %d - RECEIVED %d BYTES\n", time.Now(), i, n)
i++
b = b[:len(b)+n]
if err != nil {
if err == io.EOF {
fmt.Fprintln(os.Stderr, "RECEIVED EOF")
err = nil
}
return b, err
}
}
}
I just hacked up a cheap script to generate the input, simulating something long-running and writing only at periodic intervals, how I'd imagine npm is behaving in your case:
#!/bin/sh
for x in 1 2 3 4 5 6 7 8 9 10
do
cat ./main.go
sleep 10
done
As a side note, I find reading the actual standard library code really helpful...or at least interesting in cases like this.
#Sandy Cash was helpful in stating to use Bufio. I don't know why, if what #Jim said is true, but Bufio worked out and ReadAll() didn't.
Thanks for the help.
The code:
package main
import (
"bufio"
"fmt"
"os"
"strings"
)
func main() {
scanner := bufio.NewScanner(os.Stdin)
for scanner.Scan() {
s := scanner.Text()
arr := strings.Split(s, `\n`)
for _, v := range arr {
fmt.Println(v)
}
}
}
Program should be able to get input from stdin on terminal, as follows:
echo foobar | program
However, in the source below for Program, the stdin read blocks if the pipe is omitted:
package main
import (
"fmt"
"os"
)
func main() {
b := make([]byte, 1024)
r := os.Stdin
n, e := r.Read(b)
if e != nil {
fmt.Printf("Err: %s\n", e)
}
fmt.Printf("Res: %s (%d)\n", b, n)
}
So how can Program detect whether something is being piped to it in this manner, and continue execution instead of blocking if not?
... and is it a good idea to do so?
os.Stdin is treated like a file and has permissions. When os.Stdin is open, the perms are 0600, when closed it's 0620.
This code works:
package main
import (
"fmt"
"os"
)
func main() {
stat, _ := os.Stdin.Stat()
fmt.Printf("stdin mode: %v\n", stat.Mode().Perm())
if stat.Mode().Perm() == 0600 {
fmt.Printf("stdin open\n")
return
}
fmt.Printf("stdin close\n")
}
It would be possible if you put the read in a goroutine and use select to do a non-blocking read from the channel in main, but you would then have a race condition where you could get different results depending on whether the data comes in quickly enough. What is the larger problem you're trying to solve?
I want to do this:
Read a line from a text file.
Process the line.
Delete the line.
My first thought was to read the entire file into memory with ioutil.Readfile(),
but I'm not sure how to update the text file after the line has been processed,
and what happens if extra lines is added to the text file after it has been read into memory?
I normally write shell scripts and would do something like this:
while read -r line; do
echo "${line}"
sed -i 1d "${myList}"
done < "${myList}"
What is the best way to do this in Golang?
Use the bufio package.
Here's the basic syntax for opening a text file and looping through each line.
package main
import (
"bufio"
"fmt"
"os"
)
func main() {
// Open the file.
f, _ := os.Open("C:\\programs\\file.txt")
// Create a new Scanner for the file.
scanner := bufio.NewScanner(f)
// Loop over all lines in the file and print them.
for scanner.Scan() {
line := scanner.Text()
fmt.Println(line)
}
}
you have some options:
1- read file, process it, then write it back (you need to lock that file).
2- use binary file and invent (make use of) special data structure (like linked list) to optimize text processing (with line locking).
3- use ready made databases.
4- use Virtual filesystem inside your file, and treat each line like one file, see: https://github.com/lotrfan/vfs and https://github.com/blang/vfs
using file manager (like database server) solves the file locking dilemma.
and if the purpose of using file is one way communication which sender program just adds new line and receiver program just removes it, it is better to use os pipes (named pipe (FIFO)) or other interop methods.
see for Linux: Unix FIFO in go?
for Windows: https://github.com/natefinch/npipe
sample file writer:
package main
import (
"bufio"
"fmt"
"os"
"time"
)
func main() {
f, err := os.OpenFile("/tmp/file.txt", os.O_CREATE|os.O_WRONLY|os.O_APPEND, 0666)
if err != nil {
panic(err)
}
defer f.Close()
for i := 0; ; i++ {
w := bufio.NewWriter(f)
_, err := fmt.Fprintln(w, i)
if err != nil {
panic(err)
}
w.Flush() // Flush writes any buffered data to the underlying io.Writer.
f.Sync() // commit the current contents of the file to stable storage.
fmt.Println("write", i)
time.Sleep(500 * time.Millisecond)
}
}
sample file reader:
package main
import (
"fmt"
"os"
"time"
)
func main() {
f, err := os.OpenFile("/tmp/file.txt", os.O_RDWR, 0666)
if err != nil {
panic(err)
}
defer f.Close()
i := 0
for {
n, err := fmt.Fscanln(f, &i)
if n == 1 {
fmt.Println(i)
}
if err != nil {
fmt.Println(err)
return
}
time.Sleep(500 * time.Millisecond)
}
}
I am trying to end terminal input programmatically in 3 seconds and output the result.
My code is the following:
package main
import (
"bufio"
"fmt"
"os"
"time"
)
var (
result string
err error
)
func main() {
fmt.Println("Please input something, you have 3000 milliseconds")
go func() {
time.Sleep(time.Millisecond * 3000)
fmt.Println("It's time to break input and read what you have already typed")
fmt.Println("result")
fmt.Println(result)
}()
in := bufio.NewReader(os.Stdin)
result, err = in.ReadString('\n')
if err != nil {
fmt.Println(err)
}
}
The output:
Please input something, you have 3000 milliseconds
hello It's time to break input and read what you have already typed
result
I just printed hello and 3 seconds passed and the program should end the input and read my hello and give output:
result
hello
But I don't know how to provide this. Is it possible to end terminal input without user's intention and read the inputted value?
You can't timeout the read on stdin directly, so you need to create a timeout around receiving the result from the reading goroutine:
func getInput(input chan string) {
in := bufio.NewReader(os.Stdin)
result, err := in.ReadString('\n')
if err != nil {
log.Fatal(err)
}
input <- result
}
func main() {
input := make(chan string, 1)
go getInput(input)
select {
case i := <-input:
fmt.Println(i)
case <-time.After(3000 * time.Millisecond):
fmt.Println("timed out")
}
}
Let's say I have a program than outputs things on file descriptor 3; something like this:
package main
import "os"
func main() {
fd3 := os.NewFile(3, "fd3")
fd3.Write([]byte("FOOBAR\n"))
fd3.Close()
}
Now, I want to get the output sent to file descriptor 3 from a Go program:
package main
import (
"bufio"
"fmt"
"os/exec"
"os"
)
func main() {
cmd := exec.Command("./client")
cmd.Stderr = os.Stderr
fd3 := os.NewFile(3, "fd3")
defer fd3.Close()
cmd.ExtraFiles = []*os.File{fd3}
err := cmd.Start()
if err != nil {
panic(err)
}
go func() {
for {
reader := bufio.NewReader(fd3)
line, err := reader.ReadString('\n')
if err != nil {
panic(err)
}
fmt.Print(line)
}
}()
cmd.Wait()
fmt.Println("--- END ---")
}
But that does not work as it outputs the following error:
panic: read fd3: bad file descriptor
I don't understand what's wrong with my code. Anyone willing to help?
os.NewFile doesn't actually open a file descriptor. It's really an API to wrap a fd that was given to you.
look at the godoc: http://golang.org/pkg/os/#Create
(click the name Create, which currently points to this)
I think you want to call os.Create(name) and pass the fd to the child process
or potentiall os.Open / os.OpenFile if you need to set mode and stuff