Streaming commands output progress from Goroutine - go

Question

Streaming commands output progress question addresses the problem of printing progress of a long running command.
I tried to put the printing code within a goroutine but the scanner claims to have already hit the EOF immediately and the for block is never executed.
The bufio.scan code that gets executed on the first execution of the Scan() method is:
// We cannot generate a token with what we are holding.
// If we've already hit EOF or an I/O error, we are done.
if s.err != nil {
// Shut it down.
s.start = 0
s.end = 0
return false
}
And if I print s.err the output is EOF.
The code I'm trying to run is:
cmd := exec.Command("some", "command")
c := make(chan int, 1)
go func(cmd *exec.Cmd, c chan int) {
stdout, _ := cmd.StdoutPipe()
<-c
scanner := bufio.NewScanner(stdout)
for scanner.Scan() {
m := scanner.Text()
fmt.Println(m)
}
}(cmd, c)
cmd.Start()
c <- 1
cmd.Wait()
The idea is to start the Goroutine, get a hold of the cmd.stdout, wait that the cmd is started, and start processing its output.
The result is that the long command gets executed and the program waits for its completion, but nothing is printed to terminal.
Any idea why by the time scanner.Scan() is invoked for the first time the stdout has already reached EOF?

Answer 1

There are some problems:
The pipe is being closed before reading all data.
Always check for errors
Start cmd.Start() after c <- struct{}{} and use unbuffered channel c := make(chan struct{})
Two working sample codes:
1: Wait using channel then close the pipe after EOF using defer func() { c <- struct{}{} }(), like this working sample code:
package main
import (
"bufio"
"fmt"
"os/exec"
)
func main() {
cmd := exec.Command("Streamer")
c := make(chan struct{})
go run(cmd, c)
c <- struct{}{}
cmd.Start()
<-c
if err := cmd.Wait(); err != nil {
fmt.Println(err)
}
fmt.Println("done.")
}
func run(cmd *exec.Cmd, c chan struct{}) {
defer func() { c <- struct{}{} }()
stdout, err := cmd.StdoutPipe()
if err != nil {
panic(err)
}
<-c
scanner := bufio.NewScanner(stdout)
for scanner.Scan() {
m := scanner.Text()
fmt.Println(m)
}
fmt.Println("EOF")
}
2: Also you may Wait using sync.WaitGroup, like this working sample code:
package main
import (
"bufio"
"fmt"
"os/exec"
"sync"
)
var wg sync.WaitGroup
func main() {
cmd := exec.Command("Streamer")
c := make(chan struct{})
wg.Add(1)
go func(cmd *exec.Cmd, c chan struct{}) {
defer wg.Done()
stdout, err := cmd.StdoutPipe()
if err != nil {
panic(err)
}
<-c
scanner := bufio.NewScanner(stdout)
for scanner.Scan() {
m := scanner.Text()
fmt.Println(m)
}
}(cmd, c)
c <- struct{}{}
cmd.Start()
wg.Wait()
fmt.Println("done.")
}
And Streamer sample code (just for testing):
package main
import "fmt"
import "time"
func main() {
for i := 0; i < 10; i++ {
time.Sleep(1 * time.Second)
fmt.Println(i, ":", time.Now().UTC())
}
}
And see func (c *Cmd) StdoutPipe() (io.ReadCloser, error) Docs:
StdoutPipe returns a pipe that will be connected to the command's
standard output when the command starts.
Wait will close the pipe after seeing the command exit, so most
callers need not close the pipe themselves; however, an implication is
that it is incorrect to call Wait before all reads from the pipe have
completed. For the same reason, it is incorrect to call Run when using
StdoutPipe. See the example for idiomatic usage.

Answer 2

From godocs:
StdoutPipe returns a pipe that will be connected to the command's
standard output when the command starts.
Wait will close the pipe after seeing the command exit, so most
callers need not close the pipe themselves; however, an implication is
that it is incorrect to call Wait before all reads from the pipe have
completed.
You are calling Wait() immediately after starting the command. So the pipe gets closed as soon as the command completes, before making sure you have read all the data from the pipe. Try moving Wait() to your go routine after the scan loop.
go func(cmd *exec.Cmd, c chan int) {
stdout, _ := cmd.StdoutPipe()
<-c
scanner := bufio.NewScanner(stdout)
for scanner.Scan() {
m := scanner.Text()
fmt.Println(m)
}
cmd.Wait()
c <- 1
}(cmd, c)
cmd.Start()
c <- 1
// This is here so we don't exit the program early,
<-c
There's also a simpler way to do things, which is to just assign os.stdout as the cmd's stdout, causing the command to directly write to the os.stdout:
cmd := exec.Command("some", "command")
cmd.Stdout = os.Stdout
cmd.Run()