There is a client and a server that communicates through stdio. I think I am basically confused about the stdin and stdout. I have some questions about the stdio.
Does the server read the request from stdin or stdout where the client writes to?
Does the server write response to stdin or stdout where the client can read?
Below is the code snippet of connection part on server side.
case "stdio":
log.Println("server: reading on stdin, writing on stdout")
<-jsonrpc2.NewConn(context.Background(), jsonrpc2.NewBufferedStream(stdrwc{}, jsonrpc2.VSCodeObjectCodec{}), handler, connOpt...).DisconnectNotify()
log.Println("connection closed")
return nil
type stdrwc struct{}
func (stdrwc) Read(p []byte) (int, error) {
return os.Stdin.Read(p)
}
func (stdrwc) Write(p []byte) (int, error) {
return os.Stdout.Write(p)
}
func (stdrwc) Close() error {
if err := os.Stdin.Close(); err != nil {
return err
}
return os.Stdout.Close()
}
It's hard to say what this program is doing (as there's just a portion of it). Looks like you have an implementation of ReadWriteCloser that reads from stdin and writes to stdout (and a portion of a switch statement).
Generally any program can read from stdin and write to stdout (and stderr). You can link stdout of one program to stdin of other program with a pipe (e.g. client | server), but that's unidirectional. In your case, it sounds like you want client's stdin to go to server's stdout and vice versa. In local development, Unix sockets are usually used for that, but you might be able to create a named pipe (with mkfifo) like shown here.
Also, it might be easier to start with a super simple toy program, that doesn't include jsonrpc2 and any other packages.
I hope that helps!
Related
I am attempting to write/print text to the screen from a Go program launched from another console/terminal application--a "door" program that launches from an old-school Bulletin Board System (BBS).
The BBS itself runs over a telnet connection, localhost:2323. And when launching my program, the BBS automatically adds the correct socket handle as an argument, which I can then read using Flag (it's an integer, like 236).
Obviously, in Linux, I'd just use fmt.Println("Hello World!") using os.Stdout... But on Windows, I need to somehow pipe/redirect the Go program's output to the provided socket.
Here's the function I started with:
func writeOut(fd int, buf []byte) bool {
for len(buf) > 0 {
n, err := syscall.Write(syscall.Handle(fd), buf)
if err != nil {
fmt.Println(err)
return false
}
buf = buf[n:]
}
return true
}
called from:
writeOut(socketInt, []byte("Writing to Windows socket..."))
The error returned is: The parameter is incorrect
What am I doing wrong, and how would this be accomplished in Go?
You can't arbitrarily pass file or socket handles to another process that isn't inheriting them from your process in the first place. Each process has its own unique set of handles. In POSIX inheriting socket handles is possible (albeit not recommended) but in Windows they simply cannot be inherited (see Are TCP SOCKET handles inheritable?).
You can redirect stdout to a TCP socket when calling CreateProcess though, so that when invoked, your program can indeed fmt.Println to stdout and the output would go straight to the socket:
func serveDoor(conn *net.TCPConn, name string, args ...string) {
defer conn.Close()
cmd := exec.Command(name, args...)
cmd.Stdin = conn
cmd.Stdout = conn
cmd.Stderr = conn
err := cmd.Run()
fmt.Println("door finished:", err)
}
(full gist)
Another solution is to use a pipe and pump it to the socket.
I encounter a weird behavior with exec.Wait() with a modified Stdin.
I'm just modifying Stdin in order to be able to duplicate its content, count the amount of data… but that's not the problem here.
I've made this stripped down program just to demonstrate the strange behavior :
with a modified Stdin, cmd.Wait() waits indefinitely… until I press "enter" or "^C"
with an unmodified Stdin (uncomment the line cmd.Stdin = os.Stdin), the program processes to the end flawlessly.
when I launch this program (with a modified Stdin) with Delve (dlv debug), the program processes to the end flawlessly !
I also added a time.Sleep of 30 seconds between cmd.Start() and cmd.Wait(), and then attached the program to Delve (dlv attach PID). When I enter continue, cmd.Wait() waits indefinitely… until I press "enter" or "^C"
I tested these behaviours with go1.11 and go1.12
package main
import (
"fmt"
"os"
"os/exec"
)
type Splitter struct {
f *os.File
fd int
}
func NewSplitter(f *os.File) *Splitter {
return &Splitter{f, int(f.Fd())}
}
func (s *Splitter) Close() error {
return s.f.Close()
}
func (s *Splitter) Read(p []byte) (int, error) {
return s.f.Read(p)
}
func (s *Splitter) Write(p []byte) (int, error) {
return s.f.Write(p)
}
func main() {
var cmd *exec.Cmd
cmd = exec.Command("cat", "foobarfile")
cmd.Stdin = NewSplitter(os.Stdin)
//cmd.Stdin = os.Stdin
cmd.Stdout = NewSplitter(os.Stdout)
cmd.Stderr = NewSplitter(os.Stderr)
cmd.Start()
cmd.Wait()
fmt.Println("done")
}
Is there something I'm doing wrong ?
Thanks for your help.
This program duplicates the content as you asked. You can although try the commented part as well. And the comments are self - explanatory, I hope it explains your query.
package main
import (
"io"
"log"
"os"
"os/exec"
)
func main() {
// Execute cat command w/ arguments
// cmd := exec.Command("cat", "hello.txt")
// Execute cat command w/o arguments
cmd := exec.Command("cat")
// Attach STDOUT stream
stdout, err := cmd.StdoutPipe()
if err != nil {
log.Println(err)
}
// Attach STDIN stream
stdin, err := cmd.StdinPipe()
if err != nil {
log.Println(err)
}
// Attach STDERR stream
stderr, err := cmd.StderrPipe()
if err != nil {
log.Println(err)
}
// Spawn go-routine to copy os's stdin to command's stdin
go io.Copy(stdin, os.Stdin)
// Spawn go-routine to copy command's stdout to os's stdout
go io.Copy(os.Stdout, stdout)
// Spawn go-routine to copy command's stderr to os's stderr
go io.Copy(os.Stderr, stderr)
// Run() under the hood calls Start() and Wait()
cmd.Run()
// Note: The PIPES above will be closed automatically after Wait sees the command exit.
// A caller need only call Close to force the pipe to close sooner.
log.Println("Command complete")
}
You are replacing the process file descriptors, which are normally *os.File, with other Go types. In order for stdin to act like a stream, the os/exec package needs to launch a goroutine to copy the data between the io.Reader and the process. This is documented in the os/exec package:
// Otherwise, during the execution of the command a separate
// goroutine reads from Stdin and delivers that data to the command
// over a pipe. In this case, Wait does not complete until the goroutine
// stops copying, either because it has reached the end of Stdin
// (EOF or a read error) or because writing to the pipe returned an error.
If you look at the stack trace from your program, you'll see that it is waiting for the io goroutines to complete in Wait():
goroutine 1 [chan receive]:
os/exec.(*Cmd).Wait(0xc000076000, 0x0, 0x0)
/usr/local/go/src/os/exec/exec.go:510 +0x125
main.main()
Because you are now in control of the data stream, it is up to you to close it as necessary. If Stdin is not necessary here, then don't assign it at all. If it is going to be used, then you must Close() it to have Wait() return.
Another option is to ensure that you are using an *os.File, which the easiest method is to use the StdinPipe, StdoutPipe and StderrPipe methods, which in turn use os.Pipe(). This way ensures that the process is dealing only with *os.File, and not with other Go types.
The godocs say:
"EOF is the error returned by Read when no more input is available. Functions should return EOF only to signal a graceful end of input. If the EOF occurs unexpectedly in a structured data stream, the appropriate error is either ErrUnexpectedEOF or some other error giving more detail."
and in my program I would love love to get ErrUnexpectedEOF, but I don't. I always get just EOF even when the client terminates in the middle of sending a file.
so, i'm stuck. Successful files that transfer 100% of the way thru send EOF at end. And files that quit 50% thru with error also send EOF as the error. I do not know the total size of the file before the transfer.
Is there any more information I could get from the net.TCPConn to know was this a full file or not?
TCP is a streaming protocol; a stream of bytes. That's why file transfer protocols invented/implemented upon TCP, like FTP (File Transfer Protocol).
But (not recommended/thought experiment) if you can send your files in chunks that does not contain a certain character/byte (like base64 + \n as that certain character), then you can use that certain character as delimiter.
Now if we handle our connections using a function similar to this:
func (srv *Server) handler(conn net.Conn) {
//...
defer srv.closeConnection(conn, nil)
reader := bufio.NewReader(conn)
writer := bufio.NewWriter(conn)
//...
conn.SetDeadline(time.Now().Add(conf.ClientTimeout))
for {
select {
case <-conf.QuitSignal:
return
default:
line, err := reader.ReadBytes('\n')
if err != nil {
return
}
//...
}
conn.SetDeadline(time.Now().Add(conf.ClientTimeout))
}
}
func (srv *Server) closeConnection(conn net.Conn, err error) {
conn.Close()
//...
}
From the documentation of Reader.ReadBytes we read ReadBytes returns err != nil if and only if the returned data does not end in delimiter. This way we can recognize a broken chunk.
I have a function which returns the Reader end of an io.Pipe and kicks off a go-routine which writes data to the Writer end of it, and then closes the pipe.
func GetPipeReader() io.ReadCloser {
r, w := io.Pipe()
go func() {
_, err := io.CopyN(w, SomeReaderOfSize(N), N)
w.CloseWithError(err)
}()
return r
}
func main() {
var buf bytes.Buffer
io.Copy(&buf, GetPipeReader())
println("got", buf.Len(), "bytes")
}
https://play.golang.org/p/OAijIwmtRr
This seems to always work in my testing, in that I get all the data I wrote. But the API docs are a bit worrying to me:
func Pipe() (*PipeReader, *PipeWriter)
Pipe creates a synchronous in-memory pipe. [...] Reads on one end are
matched with writes on the other, [...] there is no internal
buffering.
func (w *PipeWriter) CloseWithError(err error) error
CloseWithError closes the writer; subsequent reads from the read half
of the pipe will return no bytes and the error err, or EOF if err is
nil.
What I want to know is, what are the possible race conditions here? Is is plausible that my go-routine will write a bunch of data and then close the pipe before I can read it all?
Do I need to use a channel for some signalling on when to close? What can go wrong, basically.
No, there are no race conditions. As the documentation mentions, reads on one end are matched with writes on the other. So, when CloseWithError() is reached, it means every Write has successfully completed and been matched with a corresponding Read - so the other end must have read everything there was to read.
This question is similar to Golang - Copy Exec output to Log except it is concerned with the buffering of output from exec commands.
I have the following test program:
package main
import (
"fmt"
"log"
"os/exec"
)
func main() {
cmd := exec.Command("python", "inf_loop.py")
var out outstream
cmd.Stdout = out
if err := cmd.Start(); err != nil {
log.Fatal(err)
}
fmt.Println(cmd.Wait())
}
type outstream struct{}
func (out outstream) Write(p []byte) (int, error) {
fmt.Println(string(p))
return len(p), nil
}
inf_loop.py, which the above refers to, simply contains:
print "hello"
while True:
pass
The go program hangs when I run it and doesn't output anything, but if I use os.Stdout instead of out then it outputs "hello" before it hangs. Why is there a discrepancy between the two io.Writers and how can it be fixed?
Some more diagnostic information:
When the loop is removed from inf_loop.py then "hello" is output from both programs, as expected.
When using yes as the program instead of the python script and outputting len(p) in outstream.Write then there is output, and the output is usually 16384 or 32768. This indicates to me that this is a buffering issue, as I originally anticipated, but I still don't understand why the outstream structure is being blocked by buffering but os.Stdout isn't. One possibility is that the behaviour is the result of the way that exec passes the io.Writer directly to os.StartProcess if it is an os.File (see source for details), otherwise it creates an os.Pipe() between the process and the io.Writer, and this pipe may be causing the buffering. However, the operation and possible buffering of os.Pipe() is too low-level for me to investigate.
Python buffers stdout by default. Try this program:
import sys
print "hello"
sys.stdout.flush()
while True:
pass
or run Python with unbuffered stdout and stderr:
cmd := exec.Command("python", "-u", "foo.py")
Note the -u flag.
You see different results when using cmd.Stout = os.Stdout because Python uses line buffering when stdout is a terminal.