My Go application outputs some amounts of text data and I need to pipe it to some external command (e.g. less). I haven't find any way to pipe this data to syscall.Exec'ed process.
As a workaround I write that text data to a temporary file and then use that file as an argument to less:
package main
import (
"io/ioutil"
"log"
"os"
"os/exec"
"syscall"
)
func main() {
content := []byte("temporary file's content")
tmpfile, err := ioutil.TempFile("", "example")
if err != nil {
log.Fatal(err)
}
defer os.Remove(tmpfile.Name()) // Never going to happen!
if _, err := tmpfile.Write(content); err != nil {
log.Fatal(err)
}
if err := tmpfile.Close(); err != nil {
log.Fatal(err)
}
binary, err := exec.LookPath("less")
if err != nil {
log.Fatal(err)
}
args := []string{"less", tmpfile.Name()}
if err := syscall.Exec(binary, args, os.Environ()); err != nil {
log.Fatal(err)
}
}
It works but leaves a temporary file on a file system, because syscall.Exec replaces the current Go process with another (less) one and deferred os.Remove won't run. Such behaviour is not desirable.
Is there any way to pipe some data to an external process without leaving any artefacts?
You should be using os/exec to build an exec.Cmd to execute, then you could supply any io.Reader you want as the stdin for the command.
From the example in the documentation:
cmd := exec.Command("tr", "a-z", "A-Z")
cmd.Stdin = strings.NewReader("some input")
var out bytes.Buffer
cmd.Stdout = &out
err := cmd.Run()
if err != nil {
log.Fatal(err)
}
fmt.Printf("in all caps: %q\n", out.String())
If you want to write directly to the command's stdin, then you call cmd.StdInPipe to get an io.WriteCloser you can write to.
If you really need to exec the process in place of your current one, you can simply remove the file before exec'ing, and provide that file descriptor as stdin for the program.
content := []byte("temporary file's content")
tmpfile, err := ioutil.TempFile("", "example")
if err != nil {
log.Fatal(err)
}
os.Remove(tmpfile.Name())
if _, err := tmpfile.Write(content); err != nil {
log.Fatal(err)
}
tmpfile.Seek(0, 0)
err = syscall.Dup2(int(tmpfile.Fd()), syscall.Stdin)
if err != nil {
log.Fatal(err)
}
binary, err := exec.LookPath("less")
if err != nil {
log.Fatal(err)
}
args := []string{"less"}
if err := syscall.Exec(binary, args, os.Environ()); err != nil {
log.Fatal(err)
}
Related
I have code that is writing to a logfile while executing a system command. E.g.
logfile, err := os.OpenFile(THIS_LOG_FILE, os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0600)
if err != nil {
return err
}
cmd.Stderr = logfile
cmd.Stdout = logfile
go func() {
err := cmd.Run()
if err != nil {
// WANT TO LOG ERROR HERE
}
}()
At the "// WANT TO LOG" line, I'd like to output the content to the standard logger, in addition to the previously assigned logfile destination. Is there a way to capture this in memory? Or should I just write everything to an in-memory buffer and flush at the end?
To clarify, in capturing the output of the command in memory, I can parse it and take action in the running program (handling errors/etc). When I write to the log file, that information is lost.
My issue is that, theoretically, I could read that back in from the file I just wrote, but that seems wasteful (and prone to failure if the command failed).
If I understand correctly, you want to write the content of stdout/stderror to a file while executing a shell command.
Since stdout and stderror are implemented the ReadCloser interface, you can merge them by io.MultiReader and perform io.Copy from source to destination.
The following snippet implements the pipeline
package main
import (
"io"
"log"
"os"
"os/exec"
)
func main() {
// prepare the command
cmd := exec.Command("your-shell-command.sh")
// get the stdout and stderr stream
erc, err := cmd.StderrPipe()
if err != nil {
log.Fatalln("Failed to get stderr reader: ", err)
}
orc, err := cmd.StdoutPipe()
if err != nil {
log.Fatalln("Failed to get stdout reader: ", err)
}
// combine stdout and stderror ReadCloser
rc := io.MultiReader(erc, orc)
// Prepare the writer
f, err := os.Create("output.log")
if err != nil {
log.Fatalln("Failed to create file")
}
defer f.Close()
// Command.Start starts a new go routine
if err := cmd.Start(); err != nil {
log.Println("Failed to start the command")
}
// add the TeeReader.
var buf bytes.Buffer
tr := io.TeeReader(rc, &buf)
if _, err := io.Copy(f, tr); err != nil {
logger.Fatalf("Failed to stream to file: %s", err)
}
if err := cmd.Wait(); err != nil {
log.Println("Failed to wait the command to execute: ", err)
}
}
I have been testing code using io.Pipe to tar and gunzip files into a tar ball and then unzipping using the tar utility. The follow code passes, however the untaring process keeps getting
error:
tar: Truncated input file (needed 1050624 bytes, only 0 available)
tar: Error exit delayed from previous errors.
This issue is really driving me crazy. It has been two weeks. I really need help debugging.
Thanks.
Development enviroment: go version go1.9 darwin/amd64
package main
import (
"archive/tar"
"compress/gzip"
"fmt"
"io"
"log"
"os"
"path/filepath"
"testing"
)
func testTarGzipPipe2(t *testing.T) {
src := "/path/to/file/folder"
pr, pw := io.Pipe()
gzipWriter := gzip.NewWriter(pw)
defer gzipWriter.Close()
tarWriter := tar.NewWriter(gzipWriter)
defer tarWriter.Close()
status := make(chan bool)
go func() {
defer pr.Close()
// tar to local disk
tarFile, err := os.OpenFile("/path/to/tar/ball/test.tar.gz", os.O_RDWR|os.O_CREATE, 0755)
if err != nil {
log.Fatal(err)
}
defer tarFile.Close()
if _, err := io.Copy(tarFile, pr); err != nil {
log.Fatal(err)
}
status <- true
}()
err := filepath.Walk(src, func(path string, info os.FileInfo, err error) error {
if err != nil {
return err
}
header, err := tar.FileInfoHeader(info, info.Name())
if err != nil {
return err
}
// header.Name = strings.TrimPrefix(strings.Replace(path, src, "", -1), string(filepath.Separator))
if err := tarWriter.WriteHeader(header); err != nil {
return err
}
if info.Mode().IsDir() {
return nil
}
fmt.Println(path)
f, err := os.Open(path)
if err != nil {
return err
}
defer f.Close()
if _, err := io.Copy(tarWriter, f); err != nil {
return err
}
return nil
})
if err != nil {
log.Fatal(err)
}
pw.Close()
<-status
}
You are closing the pipe before the deferred Close calls on the gzipWriter and tarWriter. There's no error, because you're not checking the error on either of those close calls. You need to close the tarWriter, then the gzipWriter, then the PipeWriter, in that order.
However, there's no reason for the pipe at all in this code, and you can remove the goroutine and the associated coordination altogether if you write directly to the file.
tarFile, err := os.OpenFile("/tmp/test.tar.gz", os.O_RDWR|os.O_CREATE, 0644)
if err != nil {
log.Fatal(err)
}
defer tarFile.Close()
gzipWriter := gzip.NewWriter(tarFile)
defer gzipWriter.Close()
tarWriter := tar.NewWriter(gzipWriter)
defer tarWriter.Close()
I have a simple scp function that is just a wrapper over the scp cli tool.
type credential struct {
username string
password string
host string
port string
}
func scpFile(filepath, destpath string, c *credential) error {
cmd := exec.Command("scp", filepath, c.username+"#"+c.host+":"+destpath)
if err := cmd.Run(); err != nil {
return err
}
fmt.Println("done")
return nil
}
This works just fine now I want to add the capability of putting in a password the SSH if scp needs it. This is what I came up with
func scpFile(filepath, destpath string, c *credential) error {
cmd := exec.Command("scp", filepath, c.username+"#"+c.host+":"+destpath)
stdin, err := cmd.StdinPipe()
if err != nil {
return err
}
defer stdin.Close()
if err := cmd.Start(); err != nil {
return err
}
io.WriteString(stdin, c.password+"\n")
cmd.Wait()
fmt.Println("done")
return nil
}
This does not work as the password prompt just hangs there. I tried adding a 1 second sleep before I re write to stdin thinking maybe I was writing the password to fast but did not make a difference.
So I was able to find a work around by instead of trying to send the password to stdin I create a ssh session and scp a file through the ssh session. Here is the new scpFile function:
func scpFile(filePath, destinationPath string, session *ssh.Session) error {
defer session.Close()
f, err := os.Open(filePath)
if err != nil {
return err
}
defer f.Close()
s, err := f.Stat()
if err != nil {
return err
}
go func() {
w, _ := session.StdinPipe()
defer w.Close()
fmt.Fprintf(w, "C%#o %d %s\n", s.Mode().Perm(), s.Size(), path.Base(filePath))
io.Copy(w, f)
fmt.Fprint(w, "\x00")
}()
cmd := fmt.Sprintf("scp -t %s", destinationPath)
if err := session.Run(cmd); err != nil {
return err
}
return nil
}
This could probably be made better but the main idea is there
I'm trying to download a remote file over ssh
The following approach works fine on shell
ssh hostname "tar cz /opt/local/folder" > folder.tar.gz
However the same approach on golang giving some difference in output artifact size. For example the same folders with pure shell produce artifact gz file 179B and same with go script 178B.
I assume that something has been missed from io.Reader or session got closed earlier. Kindly ask you guys to help.
Here is the example of my script:
func executeCmd(cmd, hostname string, config *ssh.ClientConfig, path string) error {
conn, _ := ssh.Dial("tcp", hostname+":22", config)
session, err := conn.NewSession()
if err != nil {
panic("Failed to create session: " + err.Error())
}
r, _ := session.StdoutPipe()
scanner := bufio.NewScanner(r)
go func() {
defer session.Close()
name := fmt.Sprintf("%s/backup_folder_%v.tar.gz", path, time.Now().Unix())
file, err := os.OpenFile(name, os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0644)
if err != nil {
panic(err)
}
defer file.Close()
for scanner.Scan() {
fmt.Println(scanner.Bytes())
if err := scanner.Err(); err != nil {
fmt.Println(err)
}
if _, err = file.Write(scanner.Bytes()); err != nil {
log.Fatal(err)
}
}
}()
if err := session.Run(cmd); err != nil {
fmt.Println(err.Error())
panic("Failed to run: " + err.Error())
}
return nil
}
Thanks!
bufio.Scanner is for newline delimited text. According to the documentation, the scanner will remove the newline characters, stripping any 10s out of your binary file.
You don't need a goroutine to do the copy, because you can use session.Start to start the process asynchronously.
You probably don't need to use bufio either. You should be using io.Copy to copy the file, which has an internal buffer already on top of any buffering already done in the ssh client itself. If an additional buffer is needed for performance, wrap the session output in a bufio.Reader
Finally, you return an error value, so use it rather than panic'ing on regular error conditions.
conn, err := ssh.Dial("tcp", hostname+":22", config)
if err != nil {
return err
}
session, err := conn.NewSession()
if err != nil {
return err
}
defer session.Close()
r, err := session.StdoutPipe()
if err != nil {
return err
}
name := fmt.Sprintf("%s/backup_folder_%v.tar.gz", path, time.Now().Unix())
file, err := os.OpenFile(name, os.O_APPEND|os.O_WRONLY|os.O_CREATE, 0644)
if err != nil {
return err
}
defer file.Close()
if err := session.Start(cmd); err != nil {
return err
}
n, err := io.Copy(file, r)
if err != nil {
return err
}
if err := session.Wait(); err != nil {
return err
}
return nil
You can try doing something like this:
r, _ := session.StdoutPipe()
reader := bufio.NewReader(r)
go func() {
defer session.Close()
// open file etc
// 10 is the number of bytes you'd like to copy in one write operation
p := make([]byte, 10)
for {
n, err := reader.Read(p)
if err == io.EOF {
break
}
if err != nil {
log.Fatal("err", err)
}
if _, err = file.Write(p[:n]); err != nil {
log.Fatal(err)
}
}
}()
Make sure your goroutines are synchronized properly so output is completeky written to the file.
I've written the following code to tar a file, code works but strangely if I untar the archive the file permissions are gone so I can't read it unless I then chmod the file:
package main
import (
"archive/tar"
"io/ioutil"
"log"
"os"
)
func main() {
c, err := os.Create("/path/to/tar/file/test.tar")
if err != nil {
log.Fatalln(err)
}
tw := tar.NewWriter(c)
f, err := os.Open("sample.txt")
if err != nil {
log.Fatalln(err)
}
fi, err := f.Stat()
if err != nil {
log.Fatalln(err)
}
hdr := &tar.Header{Name: f.Name(),
Size: fi.Size(),
}
if err := tw.WriteHeader(hdr); err != nil {
log.Fatalln(err)
}
r, err := ioutil.ReadFile("sample.txt")
if err != nil {
log.Fatalln(err)
}
if _, err := tw.Write(r); err != nil {
log.Fatalln(err)
}
if err := tw.Close(); err != nil {
log.Fatalln(err)
}
}
Any idea what I'm doing wrong?
You're not preserving the original permissions of the file. You're manually creating a header, and specifying only the name and size. Instead, use tar.FileInfoHeader to build the header.
package main
import (
"archive/tar"
"io/ioutil"
"log"
"os"
)
func main() {
c, err := os.Create("/path/to/tar/file/test.tar")
if err != nil {
log.Fatalln(err)
}
tw := tar.NewWriter(c)
f, err := os.Open("sample.txt")
if err != nil {
log.Fatalln(err)
}
fi, err := f.Stat()
if err != nil {
log.Fatalln(err)
}
// create header from FileInfo
hdr, err := tar.FileInfoHeader(fi, "")
if err != nil {
log.Fatalln(err)
}
if err := tw.WriteHeader(hdr); err != nil {
log.Fatalln(err)
}
// instead of reading the whole file into memory, prefer io.Copy
r, err := io.Copy(tw, f)
if err != nil {
log.Fatalln(err)
}
log.Printf("Wrote %d bytes\n", r)
}
Also note that I used io.Copy to copy data from the file (an io.Reader) to the tar writer (an io.Writer). This will work much better for larger files.
Also - pay special attention to this note from the docs:
Because os.FileInfo's Name method returns only the base name of the file it describes, it may be necessary to modify the Name field of the returned header to provide the full path name of the file.
In this simple example, you're just using sample.txt so you shouldn't run into trouble. If you wanted to preserve a directory structure in your tar, you may have to modify the Name field in the header.