This is an example from Go docs, that just hangs waiting for STDIN input:
scanner := bufio.NewScanner(os.Stdin)
for scanner.Scan() {
fmt.Println(scanner.Text())
}
if err := scanner.Err(); err != nil {
fmt.Fprintln(os.Stderr, "reading standard input:", err)
}
The same thing happens if you read an empty file.
What is the propper way to break hanging scanner?
The only solution that comes to mind is a periodic check to see if scanner has received new data.
There is a feeling that I'm missing some nonsense and the solution is actually obscenely simple and obvious.
You can't end (terminate) a Scanner.Scan() call.
You loop until Scanner.Scan() keeps returns true. Scanner.Scan() will keep returning true as long as lines (be them empty or not) are successfully read. It returns false if end of input is reached or reading fails.
So for you to exit the loop, you have to "transmit" end of input signal on your terminal. This can be done by pressing CTRL+D on unix systems, and CTRL+Z on Windows.
Of course you can modify the loop body, and add a condition (if) to terminate if a certain input is entered, e.g. exit:
for scanner.Scan() {
line := scanner.Text()
fmt.Println(line)
if line == "exit" {
break
}
}
Related
trying to figure out how to read from STDIN in a loop in Go / Golang and block while there is nothing there. I am trying to have a parent program execute this one as a child with a pipe to write to the child STDIN and read from child STDOUT. I get through the first iteration fine, but then EOF is returned every time on subsequent reads without blocking and loops infinitely. I have tried using bufio.Reader, bufio.Scanner, and fmt.Scan and I keep getting the EOF error and looping infinitely. I commented out the Scanner and Reader implementations to show you what I tried. Here is what I have so far:
// loop until we are told to shutdown
//scanner := bufio.NewScanner(os.Stdin)
//reader := bufio.NewReader(os.Stdin)
for {
// reads lines from STDIN
// WANT TO BLOCK HERE UNTIL SOMETHING SENT TO STDIN
//var line []byte
//if scanner.Scan() {
//line = scanner.Bytes()
//}
//line, err := reader.ReadString('\n')
line := ""
_, err := fmt.Scan(&line)
if err != nil {
log.Printf("Issue reading\n%s", err)
}
if line == "stop" {
break
}
}
No sure what to do here besides maybe use a pipe? Any help is appreciated.
Shouldn't be any more difficult than:
scanner := bufio.NewScanner(os.Stdin)
for scanner.Scan() {
line := scanner.Text()
if line == "stop" {
break
}
fmt.Printf("You entered: %s\n", line)
}
if err := scanner.Err(); err != nil {
panic(err)
}
I was trying to read in CSV file in Golang line by line with a for loop that required an if statement with a break to see if the error reading the file was EOF. I find this syntax rather unnecessary when I could in java for example read the line inside a while loop conditional and simultaneously check for the EOF error. I thought that declaring a variable inside of a for loop was possible and I know for sure that you can do this with if statements in Golang. Doing:
if v := 2; v > 1{
fmt.Println("2 is better than 1")
}
The first snippet of code I have here is what I know to work in my program.
reader := csv.NewReader(some_file)
for {
line, err := reader.Read()
if err == io.EOF {
break
}
//do data parsing from your line here
}
I do not know whether or not this second snippet is conceptually possible or just syntactically incorrect.
reader := csv.NewReader(some_file)
for line, err := reader.Read(); err != io.EOF {
//do data parsing from your line here
}
Would like some clarification/benefits/conventions of doing it one way over another, Thanks :)
It is conventional way to write simpler statements rather than lengthy, complex one, isn't it?
So, I consider the 1st version is more conventional way than the 2nd version. Moreover, the for loop in your 2nd version isn't in the right way. If you want to use that, then fix it like following or whatever you wish:
for line, err := reader.Read(); err != io.EOF; line, err = reader.Read() {
//do data parsing from your line here
}
I'm new to Go, I want to do a simple program that reads filename from user and display it's contents back to user. This is what I have so far:
fname := "D:\myfolder\file.txt"
f, err := os.Open(fname)
if err != nil {
fmt.Println(err)
}
var buff []byte
defer f.Close()
buff = make([]byte, 1024)
for {
n, err := f.Read(buff)
if n > 0 {
fmt.Println(string(buff[:n]))
}
if err == io.EOF {
break
}
}
but I get error:
The filename, directory name, or volume label syntax is incorrect.
I suspect the backslashes in fname is the reason. Try with double backslash (\\).
Put the filename in backquotes. This makes it a raw string literal. With raw string literals, no escape sequences such as \f will be processed.
fname := `D:\myfolder\file.txt`
You can also use the unix '/' path separators instead.
Does the job.
fname := "D:/myfolder/file.txt"
Congrats on learning Go! Though the question was about a specific error in the example, let's break it down line by line and learn a bit about some of the other issues that may be encountered:
fname := "D:\myfolder\file.txt"
Like C and many other languages, Go uses the backslash character for an "escape sequence". That is, certain characters that start with a backslash get translated into other characters that would be hard to see otherwise (eg. \t becomes a tab character, which may otherwise be indistinguishable from a space).
The fix is to use a raw string literal (use backticks instead of quotes) where no escape sequences are processed:
fname := `D:\myfolder\file.txt`
This fixes the initial error you were seeing by removing the invalid \m and \f escape sequences. A full list of escape sequences and more explanation can be found by reading the String Literals section of the Go spec.
f, err := os.Open(fname)
if err != nil {
fmt.Println(err)
}
The first line of this chunk is good, but it can be improved. If an error occurs, there is no reason for our program to continue executing since we couldn't even open the file, so we should both print it (probably to standard error) and exit, preferably with a non-zero exit status to indicate that something bad happened. Also, as a matter of good habit we probably want to close the file at the end of the function if opening it was successful. Putting it right below the Open call is conventional and makes it easier when someone else is reading your code. I would rewrite this as:
f, err := os.Open(fname)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(2)
// It is also common to replace these two lines with a call to log.Fatal
}
defer f.Close()
The last chunk is a bit complicated, and we could rewrite it in multiple ways. Right now it looks like this:
var buff []byte
defer f.Close()
buff = make([]byte, 1024)
for {
n, err := f.Read(buff)
if n > 0 {
fmt.Println(string(buff[:n]))
}
if err == io.EOF {
break
}
}
But we don't need to define our own buffering, because the standard library provides us with the bufio and bytes packages which can do this for us. In this case though, we probably don't need them because we can also replace the iteration with a call to io.Copy which does its own internal buffering. We could also use one of the other copy variants such as io.CopyBuffer if we wanted to use our own buffer. It's also missing some error handling, so we'll add that. Now this entire chunk becomes:
_, err := io.Copy(os.Stdout, f)
if err != nil {
fmt.Fprintf(os.Stderr, "Error reading from file: `%s'\n", err)
os.Exit(2)
}
// We're done!
I'm writing a program which opens a named pipe for reading, and then processes any lines written to this pipe:
err = syscall.Mkfifo("/tmp/myfifo", 0666)
if err != nil {
panic(err)
}
pipe, err := os.OpenFile("/tmp/myfifo", os.O_RDONLY, os.ModeNamedPipe)
if err != nil {
panic(err)
}
reader := bufio.NewReader(pipe)
scanner := bufio.NewScanner(reader)
for scanner.Scan() {
line := scanner.Text()
process(line)
}
This works fine as long as the writing process does not restart or for other reasons send an EOF. When this happens, the loop terminates (as expected from the specifications of Scanner).
However, I want to keep the pipe open to accept further writes. I could just reinitialize the scanner of course, but I believe this would create a race condition where the scanner might not be ready while a new process has begun writing to the pipe.
Are there any other options? Do I need to work directly with the File type instead?
From the bufio GoDoc:
Scan ... returns false when the scan stops, either by reaching the end of the input or an error.
So you could possibly leave the file open and read until EOF, then trigger scanner.Scan() again when the file has changed or at a regular interval (i.e. make a goroutine), and make sure the pipe variable doesn't go out of scope so you can reference it again.
If I understand your concern about a race condition correctly, this wouldn't be an issue (unless write and read operations must be synchronized) but when the scanner is re-initialized it will end up back at the beginning of the file.
I'm writing a service that has to stream output of a executed command both to parent and to log. When there is a long process, the problem is that cmd.StdoutPipe gives me a final (string) result.
Is it possible to give partial output of what is going on, like in shell
func main() {
cmd := exec.Command("sh", "-c", "some long runnig task")
stdout, _ := cmd.StdoutPipe()
cmd.Start()
scanner := bufio.NewScanner(stdout)
for scanner.Scan() {
m := scanner.Text()
fmt.Println(m)
log.Printf(m)
}
cmd.Wait()
}
P.S. Just to output would be:
cmd.Stdout = os.Stdout
But in my case it is not enough.
The code you posted works (with a reasonable command executed).
Here is a simple "some long running task" written in Go for you to call and test your code:
func main() {
fmt.Println("Child started.")
time.Sleep(time.Second*2)
fmt.Println("Tick...")
time.Sleep(time.Second*2)
fmt.Println("Child ended.")
}
Compile it and call it as your command. You will see the different lines appear immediately as written by the child process, "streamed".
Reasons why it may not work for you
The Scanner returned by bufio.NewScanner() reads whole lines and only returns something if a newline character is encountered (as defined by the bufio.ScanLines() function).
If the command you execute doesn't print newline characters, its output won't be returned immediately (only when newline character is printed, internal buffer is filled or the process ends).
Possible workarounds
If you have no guarantee that the child process prints newline characters but you still want to stream the output, you can't read whole lines. One solution is to read by words, or even read by characters (runes). You can achieve this by setting a different split function using the Scanner.Split() method:
scanner := bufio.NewScanner(stdout)
scanner.Split(bufio.ScanRunes)
The bufio.ScanRunes function reads the input by runes so Scanner.Scan() will return whenever a new rune is available.
Or reading manually without a Scanner (in this example byte-by-byte):
oneByte := make([]byte, 1)
for {
_, err := stdout.Read(oneByte)
if err != nil {
break
}
fmt.Printf("%c", oneByte[0])
}
Note that the above code would read runes that multiple bytes in UTF-8 encoding incorrectly. To read multi UTF-8-byte runes, we need a bigger buffer:
oneRune := make([]byte, utf8.UTFMax)
for {
count, err := stdout.Read(oneRune)
if err != nil {
break
}
fmt.Printf("%s", oneRune[:count])
}
Things to keep in mind
Processes have default buffers for standard output and for standard error (usually the size of a few KB). If a process writes to the standard output or standard error, it goes into the respective buffer. If this buffer gets full, further writes will block (in the child process). If you don't read the standard output and standard error of a child process, your child process may hang if the buffer is full.
So it is recommended to always read both the standard output and error of a child process. Even if you know that the command don't normally write to its standard error, if some error occurs, it will probably start dumping error messages to its standard error.
Edit: As Dave C mentions by default the standard output and error streams of the child process are discarded and will not cause a block / hang if not read. But still, by not reading the error stream you might miss a thing or two from the process.
I found good examples how to implement progress output in this article by Krzysztof Kowalczyk