I execute process with Go and write output to file (log file)
cmd := exec.Command(path)
cmd.Dir = dir
t := time.Now()
t1 := t.Format("20060102-150405")
fs, err := os.Create(dir + "/var/log/" + t1 + ".std")
if err == nil {
cmd.Stdout = fs
}
I wish to rotate logs and change log file daily
http://golang.org/pkg/os/exec/
// Stdout and Stderr specify the process's standard output and error.
//
// If either is nil, Run connects the corresponding file descriptor
// to the null device (os.DevNull).
//
// If Stdout and Stderr are the same writer, at most one
// goroutine at a time will call Write.
Stdout io.Writer
Stderr io.Writer
Is it safe to change cmd.Stdout variable daily from arbitary goroutine or I have to implement goroutine that will copy from Stdout to another file and switch files?
It is safe to change those variables directly. However, if you change them once the command has actually been run then they will have no effect on the actual running child process. To rotate the output of the running process "live" you will have to implement that in the process itself, or pipe everything through the parent and use a goroutine as you suggest.
Related
I have this function which I use to log:
func formattedLog(prefix, m string, color int) {
fmt.Printf("\033[%dm%s", color, DateTimeFormat)
fmt.Printf("▶ %s: %s\033[%dm\n", prefix, m, int(Black))
}
I want to save my log output in some file:
f, err := os.OpenFile("../../../go-logs.txt", os.O_WRONLY|os.O_CREATE|os.O_APPEND, 0666)
if err != nil {
log.Fatal("error opening logs file", err)
}
defer f.Close()
//set output of logs to f
log.SetOutput(f)
log.Println("This is a test log entry") // <====This logs in file
but when I call my function, which uses fmt.Printf it doesn't log in the file go-logs.txt:
formattedErr("ERR", msg, err.Error(), int(Red))
is there anyway to setoutput also for fmt.Printf
fmt.Printf() documents that it writes to the standard output:
Printf formats according to a format specifier and writes to standard output.
So there is no fmt.SetOutput() to redirect that to your file.
But note that the standard output is a variable in the os package:
Stdin, Stdout, and Stderr are open Files pointing to the standard input, standard output, and standard error file descriptors.
Note that the Go runtime writes to standard error for panics and crashes; closing Stderr may cause those messages to go elsewhere, perhaps to a file opened later.
var (
Stdin = NewFile(uintptr(syscall.Stdin), "/dev/stdin")
Stdout = NewFile(uintptr(syscall.Stdout), "/dev/stdout")
Stderr = NewFile(uintptr(syscall.Stderr), "/dev/stderr")
)
And you are allowed to set your own os.File to os.Stdout. Although it's not a good idea to use the same os.File for a logger and to also set it to os.Stdout, access to its File.Write() method would not be synchronized between the fmt package and the logger.
Best would be to use a log.Logger everywhere (whose output you properly set, so log messages would properly be serialized).
I have a script that dumps quite a bit of text into STDOUT when run. I'm trying to execute this script and write the output to a file without holding the entire buffer in memory at one time. (We're talking many megabytes of text that this script outputs at one time.)
The following works, but because I'm doing this across several goroutines, my memory consumption shoots up to > 5GB which I would really like to avoid:
var out bytes.Buffer
cmd := exec.Command("/path/to/script/binary", "arg1", "arg2")
cmd.Stdout = &out
err := cmd.Run()
if err != nil {
log.Fatal(err)
}
out.WriteTo(io) // io is the writer connected to the new file
Ideally as out fills up, I want to be emptying it into my target file to keep memory usage low. I've tried changing this to:
cmd := exec.Command("/path/to/script/binary", "arg1", "arg2")
cmd.Start()
stdout, _ := cmd.StdoutPipe()
r := *bufio.NewReader(stdout)
r.WriteTo(io)
cmd.Wait()
However when I print out these variables stdout is <nil>, r is {[0 0 0 0 0...]}, and r.WriteTo panics: invalid memory address or nil pointer dereference.
Is it possible to write the output of cmd as it is generated to keep memory usage down? Thanks!
Why don't you just write to a file directly?
file, _ := os.Create("/some/file")
cmd.Stdout = file
Or use your io thing (that's a terrible name for a variable, by the way, since it's a) the name of a standard library package, b) ambiguous--what does it mean?)
cmd.Stdout = io
Hi guys for some reason Wait() when I execute mysql command hangs for ever, does anyone know why?
Here is my code.
// Import imports data into Local database
func (x MySQL) Import(data string, opt LocalDb) {
var stderr bytes.Buffer
cmd := exec.Command("mysql", x.importOptions(opt)...)
// Set < pipe variable
stdin, err := cmd.StdinPipe()
errChk(err)
cmd.Stderr = &stderr
cmd.Start()
// Write data to pipe
io.WriteString(stdin, data)
fmt.Println("Importing " + x.DB + " to localhost...")
// Log mysql error
if err := cmd.Wait(); err != nil {
log.Fatal(stderr.String())
} else {
fmt.Println("Importing complete")
}
}
This function accomplishes everything and mysql imports the data into database but it never return from Wait() just freezes there even though is completed.
The problem is that you haven't closed the stdin pipe. MySQL will remain active until it is.
The fix is thus simple:
// Write data to pipe
io.WriteString(stdin, data)
stdin.Close()
fmt.Println("Importing " + x.DB + " to localhost...")
The fact that StdinPipe() acts in this way is documented as such:
StdinPipe returns a pipe that will be connected to the command's standard input when the command starts. The pipe will be closed automatically after Wait sees the command exit. A caller need only call Close to force the pipe to close sooner. For example, if the command being run will not exit until standard input is closed, the caller must close the pipe.
I'm writing a service that has to stream output of a executed command both to parent and to log. When there is a long process, the problem is that cmd.StdoutPipe gives me a final (string) result.
Is it possible to give partial output of what is going on, like in shell
func main() {
cmd := exec.Command("sh", "-c", "some long runnig task")
stdout, _ := cmd.StdoutPipe()
cmd.Start()
scanner := bufio.NewScanner(stdout)
for scanner.Scan() {
m := scanner.Text()
fmt.Println(m)
log.Printf(m)
}
cmd.Wait()
}
P.S. Just to output would be:
cmd.Stdout = os.Stdout
But in my case it is not enough.
The code you posted works (with a reasonable command executed).
Here is a simple "some long running task" written in Go for you to call and test your code:
func main() {
fmt.Println("Child started.")
time.Sleep(time.Second*2)
fmt.Println("Tick...")
time.Sleep(time.Second*2)
fmt.Println("Child ended.")
}
Compile it and call it as your command. You will see the different lines appear immediately as written by the child process, "streamed".
Reasons why it may not work for you
The Scanner returned by bufio.NewScanner() reads whole lines and only returns something if a newline character is encountered (as defined by the bufio.ScanLines() function).
If the command you execute doesn't print newline characters, its output won't be returned immediately (only when newline character is printed, internal buffer is filled or the process ends).
Possible workarounds
If you have no guarantee that the child process prints newline characters but you still want to stream the output, you can't read whole lines. One solution is to read by words, or even read by characters (runes). You can achieve this by setting a different split function using the Scanner.Split() method:
scanner := bufio.NewScanner(stdout)
scanner.Split(bufio.ScanRunes)
The bufio.ScanRunes function reads the input by runes so Scanner.Scan() will return whenever a new rune is available.
Or reading manually without a Scanner (in this example byte-by-byte):
oneByte := make([]byte, 1)
for {
_, err := stdout.Read(oneByte)
if err != nil {
break
}
fmt.Printf("%c", oneByte[0])
}
Note that the above code would read runes that multiple bytes in UTF-8 encoding incorrectly. To read multi UTF-8-byte runes, we need a bigger buffer:
oneRune := make([]byte, utf8.UTFMax)
for {
count, err := stdout.Read(oneRune)
if err != nil {
break
}
fmt.Printf("%s", oneRune[:count])
}
Things to keep in mind
Processes have default buffers for standard output and for standard error (usually the size of a few KB). If a process writes to the standard output or standard error, it goes into the respective buffer. If this buffer gets full, further writes will block (in the child process). If you don't read the standard output and standard error of a child process, your child process may hang if the buffer is full.
So it is recommended to always read both the standard output and error of a child process. Even if you know that the command don't normally write to its standard error, if some error occurs, it will probably start dumping error messages to its standard error.
Edit: As Dave C mentions by default the standard output and error streams of the child process are discarded and will not cause a block / hang if not read. But still, by not reading the error stream you might miss a thing or two from the process.
I found good examples how to implement progress output in this article by Krzysztof Kowalczyk
I want to execute perforce command line "p4" from Go to do the login job. "p4 login" require user to input password.
How can I run a program that requires user's input in Go?
The following code doesn't work.
err = exec.Command(p4cmd, "login").Run()
if err != nil {
log.Fatal(err)
}
// To run any system commands. EX: Cloud Foundry CLI commands: `CF login`
cmd := exec.Command("cf", "login")
// Sets standard output to cmd.stdout writer
cmd.Stdout = os.Stdout
// Sets standard input to cmd.stdin reader
cmd.Stdin = os.Stdin
cmd.Run()
From the os/exec.Command docs:
// Stdin specifies the process's standard input. If Stdin is
// nil, the process reads from the null device (os.DevNull).
Stdin io.Reader
Set the command's Stdin field before executing it.