How to “postpone” reading from STDIN - go

In this minimal working example I'm trying to do the following:
Prompt user for password
Unmarshal JSON either from files specified as arguments or from STDIN
Here's the source code:
package main
import (
"encoding/json"
"fmt"
"io"
"log"
"os"
"syscall"
"golang.org/x/crypto/ssh/terminal"
)
const correctPassword = "secret"
func main() {
args := os.Args[1:]
var passwd string
for {
passwd = promptPassword()
if passwd == correctPassword {
log.Println("Correct password! Begin processing...")
break
}
log.Println("Incorrect password!")
}
if len(args) == 0 { // Read from stdin
log.Println("Reading from stdin")
dec := json.NewDecoder(os.Stdin)
for {
var v interface{}
if err := dec.Decode(&v); err == io.EOF {
break
} else if err != nil {
log.Fatal(err)
}
log.Printf("%#v", v)
}
}
for _, fileName := range args {
log.Println("Reading from", fileName)
f, err := os.Open(fileName)
if err != nil {
log.Println(err)
continue
}
defer f.Close()
dec := json.NewDecoder(f)
for {
var v interface{}
if err := dec.Decode(&v); err == io.EOF {
break
} else if err != nil {
log.Fatal(err)
}
log.Printf("%#v", v)
}
}
}
func promptPassword() (passwd string) {
for {
fmt.Fprintln(os.Stderr, "Enter password:")
b, _ := terminal.ReadPassword(int(syscall.Stdin))
passwd = string(b)
if passwd != "" {
break
}
}
return passwd
}
Everything works all right except when already prepared data is piped or redirected (e.g. go run main.go < mydata.json, or echo 42 | go run main.go, etc).
When I pipe or redirect some data to the program, the data gets processed by the password prompt, not the JSON decoder part. Is there any way to at first prompt for the password, and only after process the incoming data?
I was trying to detect if there's any data in STDIN to read it and store in some temporary bytes slice, but I can't find how to close/truncate the STDIN, so it won't read data twice.

Without any changes to your program, you can include password in stdin before json, eg (bash): {echo pass; cat data.json; } | goprog, or cat pass.txt data.json | goprog
For better method for password passing (eg environment or file descriptor) look at sshpass: https://linux.die.net/man/1/sshpass
You can also buffer all stdin, and reuse its content later (via io.Reader)
Redesign your application logic to function which accept io.Reader as source of data to unmarshall.
In main() pass os.Stdin to mentioned function if there is no file argument on command line, otherwise (try to) open file and pass it to unmarshalling function.
Note: for deciding whether to print prompt or not you may use isatty like function, which tells if stdin is interactive: https://github.com/mattn/go-isatty

You wont be able because the shell will close the stdin file descriptor once it has finished to write the content.
Once stdin is closed you cant re open it, this is controlled by the shell.
Check this program, to test this behavior
package main
import (
"fmt"
"io"
"os"
)
func main() {
io.Copy(os.Stdout, os.Stdin)
fmt.Println("done")
some := make([]byte, 100)
_, err := os.Stdin.Read(some)
fmt.Println(err)
fmt.Println(string(some))
}
The output will be
$ echo "some" | go run main.go
some
done
EOF

Related

Go command line wrapper with flexible output (stdout/file/network)

Below is a command line wrapper which can parse user input command line string to Go exec.Command(). Here is why I want to write a wrapper on it:
exec.Command can only access command parameters 1 by 1, but I want to feed the shell command line as a whole
I want to run all commands in parallel(for me access multiple urls in parallel and retreive the data) - this is in exeCmd(cmdline string, output string, wg sync.WaitGroup)
I want to choose where the output data goes: stdout, local file or network - I defined a map which maps cmdline to output
Here is my code
package main
import (
"fmt"
"log"
"os"
"os/exec"
"strings"
"sync"
)
// command line parser , generate exec.Command
// cmd is same command line as running in shell(remove single quote)
func GenCmd(cmdline string) *exec.Cmd {
fmt.Println("orgin command is ", cmdline)
// splitting head => g++ parts => rest of the command
parts := strings.Fields(cmdline)
// loopArr(parts)
head := parts[0]
parts = parts[1:len(parts)]
// exec cmd & collect output
cmd := exec.Command(head, parts...)
fmt.Printf("Generated cmdline : %s\n", cmd)
return cmd
}
func exeCmd(cmdline string, output string, wg *sync.WaitGroup) {
fmt.Println("Start execCmd() ")
cmd := GenCmd(cmdline)
// check if assigned output file
if output != "" {
f, err := os.Create(output)
if err != nil {
log.Fatal(err)
}
defer f.Close()
cmd.Stdout = f // set stdout to short-response.json
err = cmd.Run()
if err != nil {
log.Fatal(err)
}
} else {
out, err := cmd.Output()
if err != nil {
fmt.Printf("%s", err)
}
fmt.Printf("%s", out)
}
wg.Done() // signal to waitgroup this goroutine complete
}
func main() {
x := make(map[string]string)
x["echo newline >> foo.o"] = ""
x["echo newline >> f1.o"] = "cmd1.txt"
cmdCnt := len(x)
wg := new(sync.WaitGroup)
wg.Add(cmdCnt)
for cmd, output := range x {
go exeCmd(cmd, output, wg) // empty string output to stdout
}
wg.Wait()
}
Go Playground for code above
My question is :
Is there a more decent way of doing this ? any exsiting go package already doing this ?
(better to have) Can someone help on network output part, write the output to another host

Bufio scan function is waiting for sometime for the last buffer (tailed operation) in GO

I am new to golang so please review the code and suggest any changes required.
So the problem statement goes below,
We have a file whose contents are in binary and are encrypted. The only way to read that contents if by using a custom utility say (named decode_it).. The command just accepts filename like below
decode_it filename.d
Now what I have to do is live monitoring the output of the decode_it utility in GO. I have written the code which is working great but somehow it is not able to process the latest tailed output (it is waiting for some amount of time for reading the last latest chunk before more data comes in ). s.Scan() is the function which is not returning the latest changes in output of that utility. I have another terminal side by side so I know that a line is appended or not. The GO Scan() function only scans when another chunk is appended at the end.
Please help. Suggest any changes required and also if possible you can suggest any other alternative approach for this.
Output of utility is - These are huge and come in seconds
1589261318 493023 8=DECODE|9=59|10=053|34=1991|35=0|49=TEST|52=20200512-05:28:38|56=TEST|57=ADMIN|
1589261368 538427 8=DECODE|9=59|10=054|34=1992|35=0|49=TEST|52=20200512-05:29:28|56=TEST|57=ADMIN|
1589261418 579765 8=DECODE|9=59|10=046|34=1993|35=0|49=TEST|52=20200512-05:30:18|56=TEST|57=ADMIN|
1589261468 627052 8=DECODE|9=59|10=047|34=1994|35=0|49=TEST|52=20200512-05:31:08|56=TEST|57=ADMIN|
1589261518 680570 8=DECODE|9=59|10=053|34=1995|35=0|49=TEST|52=20200512-05:31:58|56=TEST|57=ADMIN|
1589261568 722516 8=DECODE|9=59|10=054|34=1996|35=0|49=TEST|52=20200512-05:32:48|56=TEST|57=ADMIN|
1589261618 766070 8=DECODE|9=59|10=055|34=1997|35=0|49=TEST|52=20200512-05:33:38|56=TEST|57=ADMIN|
1589261668 807964 8=DECODE|9=59|10=056|34=1998|35=0|49=TEST|52=20200512-05:34:28|56=TEST|57=ADMIN|
1589261718 853464 8=DECODE|9=59|10=057|34=1999|35=0|49=TEST|52=20200512-05:35:18|56=TEST|57=ADMIN|
1589261768 898758 8=DECODE|9=59|10=031|34=2000|35=0|49=TEST|52=20200512-05:36:08|56=TEST|57=ADMIN|
1589261818 948236 8=DECODE|9=59|10=037|34=2001|35=0|49=TEST|52=20200512-05:36:58|56=TEST|57=ADMIN|
1589261868 995181 8=DECODE|9=59|10=038|34=2002|35=0|49=TEST|52=20200512-05:37:48|56=TEST|57=ADMIN|
1589261918 36727 8=DECODE|9=59|10=039|34=2003|35=0|49=TEST|52=20200512-05:38:38|56=TEST|57=ADMIN|
1589261968 91253 8=DECODE|9=59|10=040|34=2004|35=0|49=TEST|52=20200512-05:39:28|56=TEST|57=ADMIN|
1589262018 129336 8=DECODE|9=59|10=032|34=2005|35=0|49=TEST|52=20200512-05:40:18|56=TEST|57=ADMIN|
1589262068 173247 8=DECODE|9=59|10=033|34=2006|35=0|49=TEST|52=20200512-05:41:08|56=TEST|57=ADMIN|
1589262118 214993 8=DECODE|9=59|10=039|34=2007|35=0|49=TEST|52=20200512-05:41:58|56=TEST|57=ADMIN|
1589262168 256754 8=DECODE|9=59|10=040|34=2008|35=0|49=TEST|52=20200512-05:42:48|56=TEST|57=ADMIN|
1589262218 299908 8=DECODE|9=59|10=041|34=2009|35=0|49=TEST|52=20200512-05:43:38|56=TEST|57=ADMIN|
1589262268 345560 8=DECODE|9=59|10=033|34=2010|35=0|49=TEST|52=20200512-05:44:28|56=TEST|57=ADMIN|
1589262318 392894 8=DECODE|9=59|10=034|34=2011|35=0|49=TEST|52=20200512-05:45:18|56=TEST|57=ADMIN|
1589262368 439936 8=DECODE|9=59|10=035|34=2012|35=0|49=TEST|52=20200512-05:46:08|56=TEST|57=ADMIN|
1589262418 484959 8=DECODE|9=59|10=041|34=2013|35=0|49=TEST|52=20200512-05:46:58|56=TEST|57=ADMIN|
1589262468 531136 8=DECODE|9=59|10=042|34=2014|35=0|49=TEST|52=20200512-05:47:48|56=TEST|57=ADMIN|
1589262518 577190 8=DECODE|9=59|10=043|34=2015|35=0|49=TEST|52=20200512-05:48:38|56=TEST|57=ADMIN|
1589262568 621673 8=DECODE|9=59|10=044|34=2016|35=0|49=TEST|52=20200512-05:49:28|56=TEST|57=ADMIN|
1589262618 661569 8=DECODE|9=59|10=036|34=2017|35=0|49=TEST|52=20200512-05:50:18|56=TEST|57=ADMIN|
1589262668 704912 8=DECODE|9=59|10=037|34=2018|35=0|49=TEST|52=20200512-05:51:08|56=TEST|57=ADMIN|
1589262718 751844 8=DECODE|9=59|10=043|34=2019|35=0|49=TEST|52=20200512-05:51:58|56=TEST|57=ADMIN|
1589262768 792980 8=DECODE|9=59|10=035|34=2020|35=0|49=TEST|52=20200512-05:52:48|56=TEST|57=ADMIN|
1589262818 840365 8=DECODE|9=59|10=036|34=2021|35=0|49=TEST|52=20200512-05:53:38|56=TEST|57=ADMIN|
1589262868 879185 8=DECODE|9=59|10=037|34=2022|35=0|49=TEST|52=20200512-05:54:28|56=TEST|57=ADMIN|
1589262918 925163 8=DECODE|9=59|10=038|34=2023|35=0|49=TEST|52=20200512-05:55:18|56=TEST|57=ADMIN|
1589262968 961584 8=DECODE|9=59|10=039|34=2024|35=0|49=TEST|52=20200512-05:56:08|56=TEST|57=ADMIN|
1589263018 10120 8=DECODE|9=59|10=045|34=2025|35=0|49=TEST|52=20200512-05:56:58|56=TEST|57=ADMIN|
1589263068 53127 8=DECODE|9=59|10=046|34=2026|35=0|49=TEST|52=20200512-05:57:48|56=TEST|57=ADMIN|
1589263118 92960 8=DECODE|9=59|10=047|34=2027|35=0|49=TEST|52=20200512-05:58:38|56=TEST|57=ADMIN|
1589263168 134768 8=DECODE|9=59|10=048|34=2028|35=0|49=TEST|52=20200512-05:59:28|56=TEST|57=ADMIN|
1589263218 180362 8=DECODE|9=59|10=035|34=2029|35=0|49=TEST|52=20200512-06:00:18|56=TEST|57=ADMIN|
1589263268 220070 8=DECODE|9=59|10=027|34=2030|35=0|49=TEST|52=20200512-06:01:08|56=TEST|57=ADMIN|
1589263318 269426 8=DECODE|9=59|10=033|34=2031|35=0|49=TEST|52=20200512-06:01:58|56=TEST|57=ADMIN|
1589263368 309432 8=DECODE|9=59|10=034|34=2032|35=0|49=TEST|52=20200512-06:02:48|56=TEST|57=ADMIN|
1589263418 356561 8=DECODE|9=59|10=035|34=2033|35=0|49=TEST|52=20200512-06:03:38|56=TEST|57=ADMIN|
Code -
package main
import (
"bytes"
"bufio"
"io"
"log"
"os/exec"
"fmt"
)
// dropCRLR drops a terminal \r from the data.
func dropCRLR(data []byte) []byte {
if len(data) > 0 && data[len(data)-1] == '\r' {
return data[0 : len(data)-1]
}
return data
}
func newLineSplitFunc(data []byte, atEOF bool) (advance int, token []byte, err error) {
if atEOF && len(data) == 0 {
return 0, nil, nil
}
if i := bytes.IndexByte(data, '\n'); i >= 0 {
// We have a full newline-terminated line.
return i + 1, dropCRLR(data[0:i]), nil
}
// If we're at EOF, we have a final, non-terminated line. Return it.
if atEOF {
return len(data), dropCRLR(data), nil
}
// Request more data.
// fmt.Println("Returning 0,nil,nil")
return 0, nil, nil
}
func main() {
cmd := exec.Command("decode_it", "filename.d", "4", "1")
var out io.Reader
{
stdout, err := cmd.StdoutPipe()
if err != nil {
log.Fatal(err)
}
stderr, err := cmd.StderrPipe()
if err != nil {
log.Fatal(err)
}
out = io.MultiReader(stdout, stderr)
}
if err := cmd.Start(); err != nil {
log.Fatal(err)
}
// Make a new channel which will be used to ensure we get all output
done := make(chan struct{})
go func() {
// defer cmd.Process.Kill()
s := bufio.NewScanner(out)
s.Split(newLineSplitFunc)
for s.Scan() {
fmt.Println("---- " + s.Text())
}
if s.Err() != nil {
fmt.Printf("error: %s\n", s.Err())
}
}()
// Wait for all output to be processed
<-done
// Wait for the command to finish
if err := cmd.Wait(); err != nil{
fmt.Println("Error: " + string(err.Error()))
}
// if out closes, cmd closed.
log.Println("all done")
}
Also, Since scan() is taking a lot of time and goes into a loop from which I am not able to break as well. Please help for that too..
try something like this one, i fixed some issues and make it more simple:
package main
import (
"bufio"
"fmt"
"io"
"log"
"os/exec"
)
func main() {
var err error
// change to your command
cmd := exec.Command("sh", "test.sh")
var out io.Reader
{
var stdout, stderr io.ReadCloser
stdout, err = cmd.StdoutPipe()
if err != nil {
log.Fatal(err)
}
stderr, err = cmd.StderrPipe()
if err != nil {
log.Fatal(err)
}
out = io.MultiReader(stdout, stderr)
}
if err = cmd.Start(); err != nil {
log.Fatal(err)
}
scanner := bufio.NewScanner(out)
for scanner.Scan() {
fmt.Println("---- " + scanner.Text())
}
if err = scanner.Err(); err != nil {
fmt.Printf("error: %v\n", err)
}
log.Println("all done")
}
test.sh that i used in test:
#!/bin/bash
while [[ 1 = 1 ]]; do
echo 1
sleep 1
done
:)
I tried resolving the above issue using stdbuf -
cmd := exec.Command("stdbuf", "-o0", "-e0", "decode_it", FILEPATH, "4", "1")
Reference link - STDIO Buffering
When programs write to stdout they write with line bufferring. If they are writing to something else, then they use fully buffered mode. golang exec.Command seems to end up using fully buffered mode so using stdbuf forces no buffering.

How to read from device when stdin is pipe

So I have a Go program that reads from STDIN as such below. I want the username and password to be entered from keyboard or device but the string slice can be passed using pipe. If I run the command as below:
echo "Hello World" | go run main.go
os.Stdin will be set to read from pipes and never the keyboard. Is there a way that I can change os.Stdin FileMode as such that it will be reading from device, i.e. keyboard for username and password?
I tried using os.Stdin.Chmod(FileMode) but received this error:
chmod /dev/stdin: invalid argument
func main() {
var n = []string{}
scanner := bufio.NewScanner(os.Stdin)
fmt.Println("Please type anything with Newline Separated, empty line signals termination")
for scanner.Scan() {
h := scanner.Text()
if h == "" {
break
}
n = append(n, h)
}
if err := scanner.Err(); err != nil {
fmt.Printf("Error in reading from STDIN: %v\n", err)
}
reader := bufio.NewReader(os.Stdin)
os.Stdout.WriteString("Username: ")
username, err := reader.ReadString('\n')
if err != nil {
fmt.Printf("Unable to read username: %v\n", err)
}
username = strings.TrimSpace(username)
os.Stdout.WriteString("Password: ")
bytePassword, _ := terminal.ReadPassword(int(os.Stdin.Fd()))
password := string(bytePassword)
os.Stdout.WriteString("\n")
}
Probably scanf could help, check this example:
https://play.golang.org/p/tteQNl0trJp
package main
import (
"fmt"
)
func main() {
fmt.Println("Enter your name")
var name string
fmt.Scanf("%s", &name)
fmt.Printf("name = %s\n", name)
}
Something a little more elaborated to check if there is something to read from stdin and if not prompt the user:
https://play.golang.org/p/7qeAQ5UNhdQ
package main
import (
"bufio"
"fmt"
"log"
"os"
)
func main() {
// check if there is somethinig to read on STDIN
stat, _ := os.Stdin.Stat()
if (stat.Mode() & os.ModeCharDevice) == 0 {
var stdin []byte
scanner := bufio.NewScanner(os.Stdin)
for scanner.Scan() {
stdin = append(stdin, scanner.Bytes()...)
}
if err := scanner.Err(); err != nil {
log.Fatal(err)
}
fmt.Printf("stdin = %s\n", stdin)
} else {
fmt.Println("Enter your name")
var name string
fmt.Scanf("%s", &name)
fmt.Printf("name = %s\n", name)
}
}
You can instead read from /dev/tty as this is always the terminal (if the program runs on a terminal). This is portable only to Unix-like systems (Linux, BSD, macOS, etc) and won't work on Windows.
// +build !windows
tty, err := os.Open("/dev/tty")
if err != nil {
log.Fatalf("can't open /dev/tty: %s", err)
}
scanner := bufio.NewScanner(tty)
// as you were ...

Read from console while piping commands in Go

I need a way to read input from console twice (1 -from cat outupt, 2 - user inputs password), like this:
cat ./test_data/test.txt | app account import
My current code skips password input:
reader := bufio.NewReader(os.Stdin)
raw, err := ioutil.ReadAll(reader)
if err != nil {
return cli.NewExitError(err.Error(), 1)
}
wlt, err := wallet.Deserialize(string(raw))
if err != nil {
return cli.NewExitError(err.Error(), 1)
}
fmt.Print("Enter password: ")
pass := ""
fmt.Fscanln(reader, &pass)
Also tried to read password with Scanln - doesn't works.
Note:
cat (and piping at all) can't be used with user input, as shell redirects inputs totally.
So the most simple solutions are:
to pass filename as argument
redirect manually app account import < ./test.txt
Read file and password separately (and don't show password), try this:
package main
import (
"fmt"
"io/ioutil"
"log"
"github.com/howeyc/gopass"
)
func main() {
b, err := ioutil.ReadFile("./test_data/test.txt") // just pass the file name
if err != nil {
fmt.Print(err)
}
fmt.Println(string(b))
fmt.Print("Password: ")
pass, err := gopass.GetPasswd()
if err != nil {
log.Fatalln(err)
}
fmt.Println(string(pass))
}
and go get github.com/howeyc/gopass first.

Determine if Stdin has data with Go

Is there a way to check if the input stream (os.Stdin) has data?
The post Read from initial stdin in GO? shows how to read the data, but unfortunately blocks if no data is piped into the stdin.
os.Stdin is like any other "file", so you can check it's size:
package main
import (
"fmt"
"os"
)
func main() {
file := os.Stdin
fi, err := file.Stat()
if err != nil {
fmt.Println("file.Stat()", err)
}
size := fi.Size()
if size > 0 {
fmt.Printf("%v bytes available in Stdin\n", size)
} else {
fmt.Println("Stdin is empty")
}
}
I built this as a "pipe" executable, here is how it works:
$ ./pipe
Stdin is empty
$ echo test | ./pipe
5 bytes available in Stdin
This seems to be reliable solution and works even with sleep/delayed data via pipe.
https://coderwall.com/p/zyxyeg/golang-having-fun-with-os-stdin-and-shell-pipes
package main
import (
"os"
"fmt"
)
func main() {
fi, err := os.Stdin.Stat()
if err != nil {
panic(err)
}
if fi.Mode() & os.ModeNamedPipe == 0 {
fmt.Println("no pipe :(")
} else {
fmt.Println("hi pipe!")
}
}

Resources