Using ConPTY to spawn a python3 process.
The python process read one line and output the line length.
It should output 3001. But output 511 in cmd, output 2510 in MSYS2.
It looks like there is a line buffer limit in ConPTY. How to disable it?
package main
import (
"bufio"
"context"
"fmt"
"strings"
"github.com/UserExistsError/conpty"
)
func main() {
cpty, err := conpty.Start("python3 -c \"import sys; s = sys.stdin.readline(); print(len(s))\"")
if err != nil {
fmt.Printf("Failed to spawn a pty: %v", err)
return
}
defer cpty.Close()
go func() {
cpty.Write([]byte(strings.Repeat("A", 3000) + "\r\n"))
scanner := bufio.NewScanner(cpty)
for scanner.Scan() {
// Should output 3001.
// But output 511 in cmd, output 2510 in MSYS2.
fmt.Printf("Output: %s\n", scanner.Text())
}
if err := scanner.Err(); err != nil {
fmt.Println(err)
}
}()
if _, err := cpty.Wait(context.Background()); err != nil {
fmt.Printf("Error: %v", err)
}
}
Related
I'm studying networks, and I'm doing a tcp server with Go. One of the challenges I'm studying is to send binaries or strings by a socket client to a server, save the server response to a txt, and compare it to the original data that was sent.
The problem is that the binaries do not arrive completely on the server.
Server
package main
import (
"fmt"
"io"
"log"
"net"
)
func main() {
l, err := net.Listen("tcp", ":8000")
if nil != err {
log.Println(err)
}
defer l.Close()
for {
conn, err := l.Accept()
if nil != err {
log.Println(err)
continue
}
defer conn.Close()
go ConnHandler(conn)
}
}
func ConnHandler(conn net.Conn) {
recvBuf := make([]byte, 4096)
for {
n, err := conn.Read(recvBuf)
if nil != err {
if io.EOF == err {
log.Println(err)
return
}
log.Println(err)
return
}
if 0 < n {
data := recvBuf[:n]
fmt.Println(string(data))
}
}
}
Client
package main
import (
"fmt"
"log"
"net"
)
func main() {
conn, err := net.Dial("tcp", ":8000")
if nil != err {
log.Println(err)
}
var s string
fmt.Scanln(&s)
conn.Write([]byte(s))
conn.Close()
}
I'm generating the binaries using the command on linux:
head -c100000 /dev/urandom > binary_message.txt
I run the server:
./server > result.txt
And I send this data by the client using:
./client < binary_data.txt
In the end the file binary_data.txt have 98KB but the result .txt only has 0KB.
The problem is with scanning the binary from input. You didn't see it because the errors were ignored and not printed or otherwise handled. fmt.Scanln returns an error (so does the Write function). You should always check for possible errors happening.
I rewrote the client to load the file from disk itself as I don't think using stdin is a good fit for binary data.
package main
import (
"flag"
"io"
"log"
"net"
"os"
)
var fileName = flag.String("file", "", "file to send")
func main() {
flag.Parse()
conn, err := net.Dial("tcp", ":8000")
if nil != err {
log.Println(err)
}
defer conn.Close()
f, err := os.Open(*fileName)
if err != nil {
log.Println(err)
return
}
defer f.Close()
b := make([]byte, 1024)
for {
n, err := f.Read(b)
if err != nil {
if err == io.EOF {
log.Println("Done sending")
return
}
log.Println(err)
return
}
if _, err := conn.Write(b[:n]); err != nil {
log.Println(err)
return
}
}
}
You can use it with:
go run . -file=binary_message.txt
or if you have built the binary:
./client -file=binary_message.txt
I suggest you do the same for the server. Open a file for writing and write the binary data into that file. Use a flag to pass in the filename to write to. That will be cleaner than piping stdout to a file.
I am trying to call an external command (e.g., seq 10) and take its output, process the output then print out the processed results. But the following code does not work. Could you please let me know how to make it work?
// vim: set noexpandtab tabstop=2:
package main
import (
"bufio"
"io"
"os"
"log"
"os/exec"
)
func main() {
cmd := exec.Command("seq", "10")
stdin := bufio.NewReader(cmd.Stdout)
err := cmd.Run()
if err != nil {
log.Fatalf("cmd.Run() failed with %s\n", err)
}
for {
line, err := stdin.ReadBytes('\n')
if err == io.EOF {
if len(line) == 0 { break }
} else {
if err != nil { log.Fatal(err) }
line = line[:(len(line)-1)]
}
os.Stdout.Write(line)
os.Stdout.Write([]byte{'\n'})
}
}
$ $ go run main.go
# command-line-arguments
./main.go:15:30: cannot use cmd.Stdout (type io.Writer) as type io.Reader in argument to bufio.NewReader:
io.Writer does not implement io.Reader (missing Read method)
EDIT: I also tried this. But it also has error. Could anybody show me a working example.
// vim: set noexpandtab tabstop=2:
package main
import (
"bufio"
"io"
"os"
"log"
"os/exec"
)
func main() {
cmd := exec.Command("seq", "10")
stdout, err := cmd.StdoutPipe()
if err != nil { log.Fatal(err) }
stdin := bufio.NewReader(stdout)
err = cmd.Run()
if err != nil {
log.Fatalf("cmd.Run() failed with %s\n", err)
}
for {
line, err := stdin.ReadBytes('\n')
if err == io.EOF {
if len(line) == 0 { break }
} else {
if err != nil { log.Fatal(err) }
line = line[:(len(line)-1)]
}
os.Stdout.Write(line)
os.Stdout.Write([]byte{'\n'})
}
}
Another method (and a cleaner one) is to use bufio.Scanner which handles \n (or any other delimiter) automatically. Another advantage is that this method doesn't have race issues (been there, done that):
package main
import (
"bufio"
"fmt"
"log"
"os/exec"
)
func main() {
cmd := exec.Command("seq", "10")
stdout, err := cmd.StdoutPipe()
if err != nil {
log.Fatal(err)
}
err = cmd.Start()
if err != nil {
log.Fatalf("cmd.Start() failed with %s\n", err)
}
stdin := bufio.NewScanner(stdout)
for stdin.Scan() {
fmt.Println(stdin.Text())
}
cmd.Wait()
}
stdin.Scan() returns false on EOF, which is given once the process exits. cmd.Wait() will close StdoutPipe, and you can read err.(exec.ExitError).ExitCode() to get the exit code (if exited non-zero).
You need to pipe the standard out to the reader using .StdoutPipe(), you also need to use exec.Command(..).Start() to read incrementally (.Run() waits for the process to exit).
Here is the working code:
// vim: set noexpandtab tabstop=2:
package main
import (
"bufio"
"fmt"
"io"
"log"
"os"
"os/exec"
)
func main() {
cmd := exec.Command("seq", "10")
cmdStdOut, err := cmd.StdoutPipe()
cmdStdErr, err := cmd.StderrPipe()
defer cmdStdOut.Close()
if err != nil {
log.Fatalf("command failed with %s\n", err)
}
stdoutReader := bufio.NewReader(cmdStdOut)
stderrReader := bufio.NewReader(cmdStdErr)
err = cmd.Start()
if err != nil {
log.Fatalf("cmd.Run() failed with %s\n", err)
}
// Read stdout
for {
line, err := stdoutReader.ReadBytes('\n')
if err == io.EOF {
if len(line) == 0 {
break
}
} else {
if err != nil {
log.Fatal(err)
}
line = line[:(len(line) - 1)]
}
os.Stdout.Write(line)
os.Stdout.Write([]byte{'\n'})
}
// Read stderr
for {
line, err := stderrReader.ReadBytes('\n')
if err == io.EOF {
if len(line) == 0 {
break
}
} else {
if err != nil {
log.Fatal(err)
}
line = line[:(len(line) - 1)]
}
os.Stderr.Write(line)
os.Stderr.Write([]byte{'\n'})
}
cmd.Wait()
fmt.Println(cmd.ProcessState.ExitCode())
}
I've been trying to get the exec stdoutpipe from ffmpeg and write it into a different file. However, it stalls and doesn't finish executing the command.
package main
import (
"bytes"
"io"
"io/ioutil"
"log"
"os"
"os/exec"
)
func stdinfill(stdin io.WriteCloser) {
fi, err := ioutil.ReadFile("music.ogg")
if err != nil {
log.Fatal(err)
}
io.Copy(stdin, bytes.NewReader(fi))
}
func main() {
runcommand()
}
func runcommand() {
cmd := exec.Command("ffmpeg", "-i", "pipe:0", "-f", "mp3", "pipe:1")
stdin, err := cmd.StdinPipe()
if err != nil {
log.Fatal(err)
}
stdout, err := cmd.StdoutPipe()
if err != nil {
log.Fatal(err)
}
cmd.Stderr = os.Stderr
err = cmd.Start()
if err != nil {
log.Fatal(err)
}
stdinfill(stdin)
fo, err := os.Create("output.mp3")
if err != nil {
log.Fatal(err)
}
io.Copy(fo, stdout)
defer fo.Close()
err = cmd.Wait()
if err != nil {
log.Fatal(err)
}
}
does anyone have any ideas? It starts running ffmpeg but just stalls.
Running stdinfill in a goroutine fixed the issue.
Guys I am trying pick new lines as they come from command output, but always I end up doing it synchronous way (I have to wait until script is finished). I tired to use fsnotify but it is working only with regular files, do you have any idea how it can be done ?
package main
import (
"fmt"
"os/exec"
"bytes"
"os"
)
func main() {
cmd := exec.Command("scripts/long_script")
output := new(bytes.Buffer)
cmd.Stdout = output
cmd.Stderr = output
if err := cmd.Start(); err != nil{ // after Start program is continued and script is executing in background
fmt.Printf("Failed to start " + err.Error())
os.Exit(1)
}
fmt.Printf(" Before WAIT %s \n", output.String()) // script is writing but nothing can be read from output
cmd.Wait()
fmt.Printf(" After Wait %s \n", output.String()) // if we wait to finish execution, we can read all output
}
You should use os.StdoutPipe()
func main() {
for i := 10; i < 20; i++ {
go printName(`My name is Bob, I am ` + strconv.Itoa(i) + ` years old`)
// Adding delay so as to see incremental output
time.Sleep(60 * time.Millisecond)
}
// Adding delay so as to let program complete
// Please use channels or wait groups
time.Sleep(100 * time.Millisecond)
}
func printName(jString string) {
cmd := exec.Command("echo", "-n", jString)
cmdReader, err := cmd.StdoutPipe()
if err != nil {
log.Fatal(err)
}
scanner := bufio.NewScanner(cmdReader)
go func() {
for scanner.Scan() {
fmt.Println(scanner.Text())
}
}()
if err := cmd.Start(); err != nil {
log.Fatal(err)
}
if err := cmd.Wait(); err != nil {
log.Fatal(err)
}
}
sources that helped me:
nathanleclaire.com
blog.kowalczyk.info
eventually I managed to do it with []bytes
stdout, err := cmd.StdoutPipe()
buff := make([]byte,10)
var n int
for err == nil {
n,err = stdout.Read(buff)
if n > 0{
fmt.Printf("taken %d chars %s",n,string(buff[:n]))
}
}
cmd.Wait()
if cmd.ProcessState.Success() {. // ProcessState is set after Wait
fmt.Println("Script success")
} else {
fmt.Println("Script failed")
}
I need to be able to run an external application and interact with it as though I was manually running it from the command-line. All the examples I find only deal with running the program and capturing the output.
Below is a very simple example that I hope illustrates what I am trying to accomplish.
package main
import (
"fmt"
"log"
"os/exec"
)
func main() {
cmd := exec.Command("rm", "-i", "somefile.txt")
out, err := cmd.CombinedOutput()
if err != nil {
log.Fatal(err)
}
if string(out) == "Remove file 'somefile.txt'?" {
// send the response 'y' back to the rm process
}
// program completes normally...
}
I've tried to tweak various examples that I've found to accomplish this with zero success. It seems that even though 'rm' is waiting for a response, Go closes the process.
Any examples, articles, or advice you can provide would be greatly appreciated. Many thanks in advance.
You have two possibilities. First is to use ReadLine() but that works only if application output is full lines, and you can wait for \n. This is not the case with rm, so you have to develop a custom SplitFunction for Scanner. Both versions can be found below.
Please note that you can not use CombinedOutput, as it can not be Scanned. You have to use the pipes.
package main
import (
"bufio"
//"fmt"
"log"
"os/exec"
)
func main() {
cmd := exec.Command("rm", "-i", "somefile.txt")
// Stdout + stderr
out, err := cmd.StderrPipe() // rm writes the prompt to err
if err != nil {
log.Fatal(err)
}
r := bufio.NewReader(out)
// Stdin
in, err := cmd.StdinPipe()
if err != nil {
log.Fatal(err)
}
defer in.Close()
// Start the command!
err = cmd.Start()
if err != nil {
log.Fatal(err)
}
line, _, err := r.ReadLine()
for err != nil {
if string(line) == "Remove file 'somefile.txt'?" {
in.Write([]byte("y\n"))
}
line, _, err = r.ReadLine()
}
// program completes normally...s
}
This is a second version with the scanner, and it uses both \n and ? as line delimiters:
package main
import (
"bufio"
"bytes"
"fmt"
"log"
"os/exec"
)
// Ugly hack, this is bufio.ScanLines with ? added as an other delimiter :D
func new_scanner(data []byte, atEOF bool) (advance int, token []byte, err error) {
if atEOF && len(data) == 0 {
return 0, nil, nil
}
if i := bytes.IndexByte(data, '\n'); i >= 0 {
// We have a full newline-terminated line.
fmt.Printf("nn\n")
return i + 1, data[0:i], nil
}
if i := bytes.IndexByte(data, '?'); i >= 0 {
// We have a full ?-terminated line.
return i + 1, data[0:i], nil
}
// If we're at EOF, we have a final, non-terminated line. Return it.
if atEOF {
return len(data), data, nil
}
// Request more data.
return 0, nil, nil
}
func main() {
cmd := exec.Command("rm", "-i", "somefile.txt")
// Stdout + stderr
out, err := cmd.StderrPipe() // Again, rm writes prompts to stderr
if err != nil {
log.Fatal(err)
}
scanner := bufio.NewScanner(out)
scanner.Split(new_scanner)
// Stdin
in, err := cmd.StdinPipe()
if err != nil {
log.Fatal(err)
}
defer in.Close()
// Start the command!
err = cmd.Start()
if err != nil {
log.Fatal(err)
}
// Start scanning
for scanner.Scan() {
line := scanner.Text()
if line == "rm: remove regular empty file ‘somefile.txt’" {
in.Write([]byte("y\n"))
}
}
// Report scanner's errors
if err := scanner.Err(); err != nil {
log.Fatal(err)
}
// program completes normally...s
}