I have code and it works in the correct way, but it isn’t thread-safety https://play.golang.org/p/8EY3i1Uk_aO in these rows race happens here.
stdout := cmd.Stdout.(*bytes.Buffer).String()
stderr := cmd.Stderr.(*bytes.Buffer).String()
I rewrote it in this way
readout, _ := cmd.StdoutPipe()
readerr, _ := cmd.StderrPipe()
The link
https://play.golang.org/p/htbn2zXXeQk
I don’t like that MultiReader is used here and I cannot separate data stdout from stderr
r, _ := bufio.NewReader(io.MultiReader(readout, readerr)).ReadString('\n')
Also the second example doesn’t work (it is commented in the code). I expected stdout not to be empty (like here https://play.golang.org/p/8EY3i1Uk_aO)
How to make the logic like in the first example, but it should be thread-safety?
You have to pump cmd.Stdout and cmd.Stderr in separate goroutines until they are closed, for example, like you did with cmd.Stdin (but in other direction, of course). Otherwise there's a risk of deadlock - the process is blocked waiting to write to stdout/stderr, and your program is blocked waiting for the process to finish.
Or, like #JimB said, just assign string buffers to cmd.Stdout and cmd.Stderr, they will be filled as the process runs.
func invoke(cmd *exec.Cmd) (stdout string, stderr string, err error) {
stdoutbuf, stderrbuf := new(strings.Builder), new(strings.Builder)
cmd.Stdout = stdoutbuf
cmd.Stderr = stderrbuf
err = cmd.Start()
if err != nil {
return
}
err = cmd.Wait()
return stdoutbuf.String(), stderrbuf.String(), err
}
Live demo:
https://play.golang.org/p/hakSVNbqirB
I used the advice of #rusty, anyway the race
line `runner.go:264 ' is
append(normalizeEncoding(stdoutbuf.String()), normalizeEncoding(stderrbuf.String()), false)
solution:
create your own Writer wrapper
type lockedWriter struct {
sync.RWMutex
buf []byte
w io.Writer
}
func (w *lockedWriter) Write(b []byte) (n int, err error) {
w.Lock()
defer w.Unlock()
w.buf = append(w.buf, b...)
return w.w.Write(b)
}
func (w *lockedWriter) String() string {
w.RLock()
defer w.RUnlock()
return string(w.buf)
}
usage
stdoutbuf, stderrbuf := &lockedWriter{w: new(strings.Builder)}, &lockedWriter{w: new(strings.Builder)}
cmd.Stdout = stdoutbuf
cmd.Stderr = stderrbuf
Related
I'm using Go's ssh package to act an as SSH server, and I wanted to write some tests for that code with an SSH client implemented with the same package.
In the server-side code, I write stdout data to the ssh channel using channel.Write() and stderr data using channel.Stderr().Write(), like so:
// sample of server code
switch data.Type {
case "stdout":
channel.Write([]byte("stdout message!"))
case "stderr":
channel.Stderr().Write([]byte("stderr message!"))
}
In the client-side code, I read this data like so:
// sample of client code
conn, _ := gossh.Dial("tcp", "localhost:2222", config)
defer conn.Close()
session, _ := conn.NewSession()
defer session.Close()
stData := make([]byte, 100)
stderr, _ := session.StderrPipe()
stderr.Read(b)
stdout, _ := session.StdoutPipe()
stdoutData := make([]byte, 100)
stdout.Read(stdoutData)
This works fine when the code is actually running and communicating with a local ssh process. The problem is that when I'm the client, my read from the StderrPipe works as expected, but my read from StdoutPipe simply hangs. It appears that calls to channel.Write() send data somewhere other than the StdoutPipe, so my question is simply: where does that data go?
For reasons I still don't completely understand, it appears that JimB is correct and the key here is to set up goroutines that continuously read from stdout and stderr. I ended up doing something like this:
// provide pipes for two-way communication with the server
func setupIo(session *ssh.Session) (io.WriteCloser, chan []byte, chan []byte) {
stdout, err := session.StdoutPipe()
Expect(err).To(BeNil())
stderr, err := session.StderrPipe()
Expect(err).To(BeNil())
stdin, err := session.StdinPipe()
Expect(err).To(BeNil())
stdoutChan := make(chan []byte)
stderrChan := make(chan []byte)
go readPipe(stdout, stdoutChan)
go readPipe(stderr, stderrChan)
return stdin, stdoutChan, stderrChan
}
func readPipe(pipe io.Reader, outputChan chan []byte) {
b := make([]byte, 100)
for {
if n, err := pipe.Read(b); err != nil {
return
} else if n > 0 {
outputChan <- b[:n]
}
}
}
// later in the test...
_, stdoutChan, _ := setupIo(session)
output := <-stderrChan
// make assertions about output
// ...
I am writing a Test function that testing go program interacting with a command line program. That is
os.Stdio -> cmd.Stdin
cmd.Stdin -> os.Stdin
I could use pipe to connect those io, but I would have a log of the data passing the pipe.
I tried to use io.MultiWriter but it is not a os.file object and cannot assign to os.Stdout.
I have found some sample which use a lot of pipe and io.copy. but as io.copy is not interactive. How could I connect the Stdout to a io.MultiWriter with pipe?
logfile, err := os.Create("stdout.log")
r, w, _ := os.Pipe()
mw := io.MultiWriter(os.Stdout, logfile, w)
cmd.Stdin = r
os.Stdout = mw // <- error in this line
The error message like
cannot use mw (variable of type io.Writer) as type *os.File in assignment:
As an alternative solution, you can mimic the MultiWriter using a separate func to read what has been written to the stdout, capture it, then write it to the file and also to the original stdout.
package main
import (
"bufio"
"fmt"
"os"
"time"
)
func main() {
originalStdout := os.Stdout //Backup of the original stdout
r, w, _ := os.Pipe()
os.Stdout = w
//Use a separate goroutine for non-blocking
go func() {
f, _ := os.Create("stdout.log")
defer f.Close()
scanner := bufio.NewScanner(r)
for scanner.Scan() {
s := scanner.Text() + "\r\n"
f.WriteString(s)
originalStdout.WriteString(s)
}
}()
//Test
c := time.NewTicker(time.Second)
for {
select {
case <-c.C:
fmt.Println(time.Now())
fmt.Fprintln(os.Stderr, "This is on Stderr")
}
}
}
I figure out a work around with pipe, TeeReader and MultiWriter
The setup is a test function which test the interaction of go program interact with a python program though stdin and stdout.
main.stdout -> pipe -> TeeReader--> (client.stdin, MultiWriter(log, stdout))
client.stdout -> MultiWriter(pipe --> main.stdin, logfile, stdout )
I will try to add more explanation later
func Test_Interactive(t *testing.T) {
var tests = []struct {
file string
}{
{"Test_Client.py"},
}
for tc, tt := range tests {
fmt.Println("---------------------------------")
fmt.Printf("Test %d, Test Client:%v\n", tc+1, tt.file)
fmt.Println("---------------------------------")
// Define external program
client := exec.Command("python3", tt.file)
// Define log file
logfile, err := os.Create(tt.file + ".log")
if err != nil {
panic(err)
}
defer logfile.Close()
out := os.Stdout
defer func() { os.Stdout = out }() // Restore original Stdout
in := os.Stdin
defer func() { os.Stdin = in }() // Restore original Stdin
// Create pipe connect os.Stdout to client.Stdin
gr, gw, _ := os.Pipe()
// Connect os.Stdout to writer side of pipe
os.Stdout = gw
// Create MultiWriter to write to logfile and os.Stdout at the same time
gmw := io.MultiWriter(out, logfile)
// Create a tee reader read from reader side of the pipe and flow to the MultiWriter
// Repleace the cmd.Stdin with TeeReader
client.Stdin = io.TeeReader(gr, gmw)
// Create a pipe to connect client.Stdout to os.Stdin
cr, cw, _ := os.Pipe()
// Create MultWriter to client stdout
cmw := io.MultiWriter(cw, logfile, out)
client.Stdout = cmw
// Connect os stdin to another end of the pipe
os.Stdin = cr
// Start Client
client.Start()
// Start main
main()
// Check Testing program error
if err := client.Process.Release(); err != nil {
if exiterr, ok := err.(*exec.ExitError); ok {
if status, ok := exiterr.Sys().(syscall.WaitStatus); ok {
log.Printf("Exit Status: %d", status.ExitStatus())
t.Errorf("Tester return error\n")
}
} else {
log.Fatalf("cmd.Wait: %v", err)
t.Errorf("Tester return error\n")
}
}
}
}
I want to be able to seamlessly print the output from an unknown(user defined) command in go passed through a io.ReadCloser. The bufio.NewScanner reads the std out and prints the text correctly, however the color that the child process prints is not recorded and passed through the pipe(or I don't know how to access it).
I tried using execErr := syscall.Exec(binary, cmd.Args, os.Environ()) however since this takes over the go process, I can't get an array of processes to run.
// SpawnGroup spawns a group of processes
func SpawnGroup(cmds []*exec.Cmd) {
spawnWg := &sync.WaitGroup{}
spawnWg.Add(len(cmds))
defer spawnWg.Wait()
for _, cmd := range cmds {
go Spawn(cmd, spawnWg)
}
}
// Spawn spawn a child process
func Spawn(cmd *exec.Cmd, wg *sync.WaitGroup) {
defer wg.Done()
stdout, _ := cmd.StdoutPipe()
stderr, _ := cmd.StderrPipe()
err := cmd.Start()
if err != nil {
color.Red(err.Error())
}
scanner := bufio.NewScanner(stdout)
for scanner.Scan() {
fmt.Println(scanner.Text())
}
errScanner := bufio.NewScanner(stderr)
for errScanner.Scan() {
color.Red(scanner.Text())
}
cmd.Wait()
}
For example, if I try to run two commands passed in an array such as sls offline and vue-cli-service serve everything works as expected, but the logged output doesn't have the color. Both of these processes print some of their output to the command line with color. The exact array of commands will be unknown, so I need a way to print the exact output from them.
I was able to get this to work in node by using:
let cmd = await spawn(command, args, {
stdio: 'inherit',
shell: true,
...options
});
I haven't been able to track down how to do this in go.
Thanks for any advice or help, this is my first real dive into go, but it seems like an excellent language!
Changing the spawn code to assign the os.Stdout to the cmd.Stdout enabled the output to print with the correct colors.
func Spawn(cmd *exec.Cmd, wg *sync.WaitGroup) {
defer wg.Done()
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
err := cmd.Start()
if err != nil {
color.Red(err.Error())
}
defer cmd.Wait()
}```
I need to copy all the input provided by the user for the child process during its execution. I have tried to scan the cmd.Stdin for the copy of input but can't get it. Am I missing something here?
func main(){
cmd:= exec.Command("python", "-i")
cmd.Stdin = os.Stdin
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
buff := bufio.NewScanner(cmd.Stdin)
go func(){
for buff.Scan(){
fmt.Println(buff.Text())
}
}()
_ = cmd.Run()
}
I think you'll actually need to capture the input, and pass it to the subprocess...
func main(){
cmd := exec.Command("python", "-i")
stdin, err := cmd.StdinPipe()
if err != nil {
panic(err)
}
cmd.Stdout = os.Stdout
cmd.Stderr = os.Stderr
buff := bufio.NewScanner(os.Stdin)
go func(){
for buff.Scan(){
input := buff.Text()
fmt.Println(input)
io.WriteString(stdin, input)
}
}()
cmd.Start()
cmd.Wait()
}
I have an app called "myapp". That app simply writes to stderr.
The important bit is, I want to capture what is written in stderr and process it in real-time. How would I go about doing that?
I tried the code below. :
cmd := exec.Command("myapp") // this app prints lines to stderr
stderr, err := cmd.StderrPipe()
if err != nil {
log.Fatal(err)
}
if err := cmd.Start(); err != nil {
log.Fatal(err)
}
if b, err := ioutil.ReadAll(stderr); err == nil {
log.Println(string(b))
}
if err := cmd.Wait(); err != nil {
log.Fatal(err)
}
The code doesn't print out anyting. I suspect it's because ioutil.ReadAll() is not the proper func to call since it waits for EOF. How else would I read from the stderr pipe?
You can replace the command executed with anything that outputs to stdout or stderr like tail -f mylogfile. The point is, I want to process the lines as they are written to stdout.
StderrPipe returns a ReadCloser. You can use that to create a bufio.Scanner and then read lines one by one:
sc := bufio.NewScanner(stderr)
for sc.Scan() {
fmt.Printf("Line: %s\n", sc.Text());
}
Create a type that implements io.Writer and set that as the command's stderr writer.
type Processor struct{}
func (Processor) Write(b []byte) (int, error) {
// intercept data here
return os.Stdout.Write(b)
}
func main() {
cmd := exec.Command("mycommand")
cmd.Stderr = Processor{}
_ = cmd.Run()
}