I have this function which I use to log:
func formattedLog(prefix, m string, color int) {
fmt.Printf("\033[%dm%s", color, DateTimeFormat)
fmt.Printf("▶ %s: %s\033[%dm\n", prefix, m, int(Black))
}
I want to save my log output in some file:
f, err := os.OpenFile("../../../go-logs.txt", os.O_WRONLY|os.O_CREATE|os.O_APPEND, 0666)
if err != nil {
log.Fatal("error opening logs file", err)
}
defer f.Close()
//set output of logs to f
log.SetOutput(f)
log.Println("This is a test log entry") // <====This logs in file
but when I call my function, which uses fmt.Printf it doesn't log in the file go-logs.txt:
formattedErr("ERR", msg, err.Error(), int(Red))
is there anyway to setoutput also for fmt.Printf
fmt.Printf() documents that it writes to the standard output:
Printf formats according to a format specifier and writes to standard output.
So there is no fmt.SetOutput() to redirect that to your file.
But note that the standard output is a variable in the os package:
Stdin, Stdout, and Stderr are open Files pointing to the standard input, standard output, and standard error file descriptors.
Note that the Go runtime writes to standard error for panics and crashes; closing Stderr may cause those messages to go elsewhere, perhaps to a file opened later.
var (
Stdin = NewFile(uintptr(syscall.Stdin), "/dev/stdin")
Stdout = NewFile(uintptr(syscall.Stdout), "/dev/stdout")
Stderr = NewFile(uintptr(syscall.Stderr), "/dev/stderr")
)
And you are allowed to set your own os.File to os.Stdout. Although it's not a good idea to use the same os.File for a logger and to also set it to os.Stdout, access to its File.Write() method would not be synchronized between the fmt package and the logger.
Best would be to use a log.Logger everywhere (whose output you properly set, so log messages would properly be serialized).
Related
I am using Delve to debug and having issues with the best way to handle stdin/stdout.
The first problem is that I cannot read the console. I have a function that takes use onput from the console:
func readConsole() string {
reader := bufio.NewReader(os.Stdin)
entry, err := reader.ReadString('\n')
if err != nil {
tlog.Fatal(fmt.Errorf("readConsole(): Error reading console input. %v", err))
}
entry = strings.Replace(entry, "\n", "", -1)
return entry
}
The following "bad file descriptor" error is returned by ReadString():
F0208 21:03:56.574021 429026 configurator.go:81] readConsole(): Error reading console input. read /dev/stdin: bad file descriptor
The second problem is that fmt.Printf() works when I just run the app, but if I am stepping through source code fmt.Printf() does not display anything.
I get that dlv is competing for input and output via the console, but not sure how to manage the competing requirements.
I want to capture all stdout and stderr messages, and parse the data and print them in my desired format. How do I do this in go?
You can use cmd.CombinedOutput or cmd.Output:
out, err := exec.Command("ls", "-al").CombinedOutput()
//or
out, err := exec.Command("ls", "-al").Output()
This question is similar to Golang - Copy Exec output to Log except it is concerned with the buffering of output from exec commands.
I have the following test program:
package main
import (
"fmt"
"log"
"os/exec"
)
func main() {
cmd := exec.Command("python", "inf_loop.py")
var out outstream
cmd.Stdout = out
if err := cmd.Start(); err != nil {
log.Fatal(err)
}
fmt.Println(cmd.Wait())
}
type outstream struct{}
func (out outstream) Write(p []byte) (int, error) {
fmt.Println(string(p))
return len(p), nil
}
inf_loop.py, which the above refers to, simply contains:
print "hello"
while True:
pass
The go program hangs when I run it and doesn't output anything, but if I use os.Stdout instead of out then it outputs "hello" before it hangs. Why is there a discrepancy between the two io.Writers and how can it be fixed?
Some more diagnostic information:
When the loop is removed from inf_loop.py then "hello" is output from both programs, as expected.
When using yes as the program instead of the python script and outputting len(p) in outstream.Write then there is output, and the output is usually 16384 or 32768. This indicates to me that this is a buffering issue, as I originally anticipated, but I still don't understand why the outstream structure is being blocked by buffering but os.Stdout isn't. One possibility is that the behaviour is the result of the way that exec passes the io.Writer directly to os.StartProcess if it is an os.File (see source for details), otherwise it creates an os.Pipe() between the process and the io.Writer, and this pipe may be causing the buffering. However, the operation and possible buffering of os.Pipe() is too low-level for me to investigate.
Python buffers stdout by default. Try this program:
import sys
print "hello"
sys.stdout.flush()
while True:
pass
or run Python with unbuffered stdout and stderr:
cmd := exec.Command("python", "-u", "foo.py")
Note the -u flag.
You see different results when using cmd.Stout = os.Stdout because Python uses line buffering when stdout is a terminal.
I'm writing a service that has to stream output of a executed command both to parent and to log. When there is a long process, the problem is that cmd.StdoutPipe gives me a final (string) result.
Is it possible to give partial output of what is going on, like in shell
func main() {
cmd := exec.Command("sh", "-c", "some long runnig task")
stdout, _ := cmd.StdoutPipe()
cmd.Start()
scanner := bufio.NewScanner(stdout)
for scanner.Scan() {
m := scanner.Text()
fmt.Println(m)
log.Printf(m)
}
cmd.Wait()
}
P.S. Just to output would be:
cmd.Stdout = os.Stdout
But in my case it is not enough.
The code you posted works (with a reasonable command executed).
Here is a simple "some long running task" written in Go for you to call and test your code:
func main() {
fmt.Println("Child started.")
time.Sleep(time.Second*2)
fmt.Println("Tick...")
time.Sleep(time.Second*2)
fmt.Println("Child ended.")
}
Compile it and call it as your command. You will see the different lines appear immediately as written by the child process, "streamed".
Reasons why it may not work for you
The Scanner returned by bufio.NewScanner() reads whole lines and only returns something if a newline character is encountered (as defined by the bufio.ScanLines() function).
If the command you execute doesn't print newline characters, its output won't be returned immediately (only when newline character is printed, internal buffer is filled or the process ends).
Possible workarounds
If you have no guarantee that the child process prints newline characters but you still want to stream the output, you can't read whole lines. One solution is to read by words, or even read by characters (runes). You can achieve this by setting a different split function using the Scanner.Split() method:
scanner := bufio.NewScanner(stdout)
scanner.Split(bufio.ScanRunes)
The bufio.ScanRunes function reads the input by runes so Scanner.Scan() will return whenever a new rune is available.
Or reading manually without a Scanner (in this example byte-by-byte):
oneByte := make([]byte, 1)
for {
_, err := stdout.Read(oneByte)
if err != nil {
break
}
fmt.Printf("%c", oneByte[0])
}
Note that the above code would read runes that multiple bytes in UTF-8 encoding incorrectly. To read multi UTF-8-byte runes, we need a bigger buffer:
oneRune := make([]byte, utf8.UTFMax)
for {
count, err := stdout.Read(oneRune)
if err != nil {
break
}
fmt.Printf("%s", oneRune[:count])
}
Things to keep in mind
Processes have default buffers for standard output and for standard error (usually the size of a few KB). If a process writes to the standard output or standard error, it goes into the respective buffer. If this buffer gets full, further writes will block (in the child process). If you don't read the standard output and standard error of a child process, your child process may hang if the buffer is full.
So it is recommended to always read both the standard output and error of a child process. Even if you know that the command don't normally write to its standard error, if some error occurs, it will probably start dumping error messages to its standard error.
Edit: As Dave C mentions by default the standard output and error streams of the child process are discarded and will not cause a block / hang if not read. But still, by not reading the error stream you might miss a thing or two from the process.
I found good examples how to implement progress output in this article by Krzysztof Kowalczyk
I execute process with Go and write output to file (log file)
cmd := exec.Command(path)
cmd.Dir = dir
t := time.Now()
t1 := t.Format("20060102-150405")
fs, err := os.Create(dir + "/var/log/" + t1 + ".std")
if err == nil {
cmd.Stdout = fs
}
I wish to rotate logs and change log file daily
http://golang.org/pkg/os/exec/
// Stdout and Stderr specify the process's standard output and error.
//
// If either is nil, Run connects the corresponding file descriptor
// to the null device (os.DevNull).
//
// If Stdout and Stderr are the same writer, at most one
// goroutine at a time will call Write.
Stdout io.Writer
Stderr io.Writer
Is it safe to change cmd.Stdout variable daily from arbitary goroutine or I have to implement goroutine that will copy from Stdout to another file and switch files?
It is safe to change those variables directly. However, if you change them once the command has actually been run then they will have no effect on the actual running child process. To rotate the output of the running process "live" you will have to implement that in the process itself, or pipe everything through the parent and use a goroutine as you suggest.