There's a library that exports a file but I'd like to capture the contents of the file. I'd like to pass a writer to the library and be able to read what the writer wrote to the file. Eventually i want to augment the library to skip writing this file.
Is this possible with io.Copy or io.Pipe?
The library code creates a *File and uses this handle as an io.Writer.
I tried using io.Copy but only 0 bytes were read.
func TestFileCopy(t *testing.T) {
codeFile, err := os.Create("test.txt")
if err != nil {
t.Error(err)
}
defer codeFile.Close()
codeFile.WriteString("Hello World")
n, err := io.Copy(os.Stdout, codeFile)
if err != nil {
t.Error(err)
}
log.Println(n, "bytes")
}
If you want to capture the bytes as they are written, use an io.MultiWriter with a bytes.Buffer as the second writer.
var buf bytes.Buffer
w := io.MultiWriter(codeFile, &buf)
or to see the file on stdout as it's written:
w := io.MultiWriter(codeFile, os.Stdout)
Otherwise, if you want to re-read the same file, you need to seek back to the start after writing:
codeFile.Seek(0, 0)
Related
I wrote a piece of code with a IPC purpose. The expected behaviour is that the code reads the content from the named-pipe and prints the string (with the Send("log", buff.String())). First I open the named-pipe 'reader' inside the goroutine, while the reader is open I send a signal that the data can be written to the named-pipe (with the Send("datarequest", "")). Here is the code:
var wg sync.WaitGroup
wg.Add(1)
go func() {
//reader part
file, err := os.OpenFile("tmp/"+os.Args[1], os.O_RDONLY, os.ModeNamedPipe)
if err != nil {
Send("error", err.Error())
}
var buff bytes.Buffer
_, err = io.Copy(&buff, file)
Send("log", buff.String())
if err != nil {
Send("error", err.Error())
}
wg.Done()
}()
Send("datarequest", "")
wg.Wait()
And here is the code which executes when the signal is send:
//writer part
file, err := os.OpenFile("tmp/" + execID, os.O_WRONLY, 0777)
if err != nil {
c <- "[error] error opening file: " + err.Error()
}
bytedata, _ := json.Marshal(moduleParameters)
file.Write(bytedata)
So the behaviour I get it that the code blocks indefinitely when I try to copy it. I really don't know why this happens. When I test it with cat in the terminal I do get the intended result so my question is how do I get the same result with code?
Edit
The execID is the same as os.Args[1]
The writer should close the file after it's done sending using file.Close(). Note that file.Close() may return error.
I want to copy a os.Stdin string to a buffer, to check for a user inputted text (e.g. "hibye") and put an if statement against it.
My current code just handles simple stdin stdiout copy operations (no buffer):
func interact(c net.Conn) {
// Read from Reader and write to Writer until EOF()
copy := func(r io.ReadCloser, w io.WriteCloser) {
defer func() {
r.Close()
w.Close()
}()
n, err := io.Copy(w, r)
if err != nil {
log.Printf("[%s]: ERROR: %s\n", c.RemoteAddr(), err)
log.Println(n)
}
}
go copy(c, os.Stdout)
go copy(os.Stdin, c)
}
Question: What is the most efficient way to implement a use of a buffer to have control over the passed strings?
bad example (failed attempt):
buf := make([]byte, 1024)
go copy (os.Stdin, []byte(buf))
if buf == "hibye" {
do stuff
}
I am using os.Pipes() in my program, but for some reason it gives a bad file descriptor error each time i try to write or read data from it.
Is there some thing I am doing wrong?
Below is the code
package main
import (
"fmt"
"os"
)
func main() {
writer, reader, err := os.Pipe()
if err != nil {
fmt.Println(err)
}
_,err= writer.Write([]byte("hello"))
if err != nil {
fmt.Println(err)
}
var data []byte
_, err = reader.Read(data)
if err != nil {
fmt.Println(err)
}
fmt.Println(string(data))
}
output :
write |0: Invalid argument
read |1: Invalid argument
You are using an os.Pipe, which returns a pair of FIFO connected files from the os. This is different than an io.Pipe which is implemented in Go.
The invalid argument errors are because you are reading and writing to the wrong files. The signature of os.Pipe is
func Pipe() (r *File, w *File, err error)
which shows that the returns values are in the order "reader, writer, error".
and io.Pipe:
func Pipe() (*PipeReader, *PipeWriter)
Also returning in the order "reader, writer"
When you check the error from the os.Pipe function, you are only printing the value. If there was an error, the files are invalid. You need to return or exit on that error.
Pipes are also blocking (though an os.Pipe has a small, hard coded buffer), so you need to read and write asynchronously. If you swapped this for an io.Pipe it would deadlock immediately. Dispatch the Read method inside a goroutine and wait for it to complete.
Finally, you are reading into a nil slice, which will read nothing. You need to allocate space to read into, and you need to record the number of bytes read to know how much of the buffer is used.
A more correct version of your example would look like:
reader, writer, err := os.Pipe()
if err != nil {
log.Fatal(err)
}
var wg sync.WaitGroup
wg.Add(1)
go func() {
defer wg.Done()
data := make([]byte, 1024)
n, err = reader.Read(data)
if n > 0 {
fmt.Println(string(data[:n]))
}
if err != nil && err != io.EOF {
fmt.Println(err)
}
}()
_, err = writer.Write([]byte("hello"))
if err != nil {
fmt.Println(err)
}
wg.Wait()
I am trying to stream out bytes of a zip file using io.Pipe() function in golang. I am using pipe reader to read the bytes of each file in the zip and then stream those out and use the pipe writer to write the bytes in the response object.
func main() {
r, w := io.Pipe()
// go routine to make the write/read non-blocking
go func() {
defer w.Close()
bytes, err := ReadBytesforEachFileFromTheZip()
err := json.NewEncoder(w).Encode(bytes)
handleErr(err)
}()
This is not a working implementation but a structure of what I am trying to achieve. I don't want to use ioutil.ReadAll since the file is going to be very large and Pipe() will help me avoid bringing all the data into memory. Can someone help with a working implementation using io.Pipe() ?
I made it work using golang io.Pipe().The Pipewriter writes byte to the pipe in chunks and the pipeReader reader from the other end. The reason for using a go-routine is to have a non-blocking write operation while simultaneous reads happen form the pipe.
Note: It's important to close the pipe writer (w.Close()) to send EOF on the stream otherwise it will not close the stream.
func DownloadZip() ([]byte, error) {
r, w := io.Pipe()
defer r.Close()
defer w.Close()
zip, err := os.Stat("temp.zip")
if err != nil{
return nil, err
}
go func(){
f, err := os.Open(zip.Name())
if err != nil {
return
}
buf := make([]byte, 1024)
for {
chunk, err := f.Read(buf)
if err != nil && err != io.EOF {
panic(err)
}
if chunk == 0 {
break
}
if _, err := w.Write(buf[:chunk]); err != nil{
return
}
}
w.Close()
}()
body, err := ioutil.ReadAll(r)
if err != nil {
return nil, err
}
return body, nil
}
Please let me know if someone has another way of doing it.
I've modified the official documentation example for the zlib package to use an opened file rather than a set of hardcoded bytes (code below).
The code reads in the contents of a source text file and compresses it with the zlib package. I then try to read back the compressed file and print its decompressed contents into stdout.
The code doesn't error, but it also doesn't do what I expect it to do; which is to display the decompressed file contents into stdout.
Also: is there another way of displaying this information, rather than using io.Copy?
package main
import (
"compress/zlib"
"io"
"log"
"os"
)
func main() {
var err error
// This defends against an error preventing `defer` from being called
// As log.Fatal otherwise calls `os.Exit`
defer func() {
if err != nil {
log.Fatalln("\nDeferred log: \n", err)
}
}()
src, err := os.Open("source.txt")
if err != nil {
return
}
defer src.Close()
dest, err := os.Create("new.txt")
if err != nil {
return
}
defer dest.Close()
zdest := zlib.NewWriter(dest)
defer zdest.Close()
if _, err := io.Copy(zdest, src); err != nil {
return
}
n, err := os.Open("new.txt")
if err != nil {
return
}
r, err := zlib.NewReader(n)
if err != nil {
return
}
defer r.Close()
io.Copy(os.Stdout, r)
err = os.Remove("new.txt")
if err != nil {
return
}
}
Your defer func doesn't do anything, because you're shadowing the err variable on every new assignment. If you want a defer to run, return from a separate function, and call log.Fatal after the return statement.
As for why you're not seeing any output, it's because you're deferring all the Close calls. The zlib.Writer isn't flushed until after the function exits, and neither is the destination file. Call Close() explicitly where you need it.
zdest := zlib.NewWriter(dest)
if _, err := io.Copy(zdest, src); err != nil {
log.Fatal(err)
}
zdest.Close()
dest.Close()
I think you messed up the code logic with all this defer stuff and your "trick" err checking.
Files are definitively written when flushed or closed. You just copy into new.txt without closing it before opening it to read it.
Defering the closing of the file is neat inside a function which has multiple exits: It makes sure the file is closed once the function is left. But your main requires the new.txt to be closed after the copy, before re-opening it. So don't defer the close here.
BTW: Your defense against log.Fatal terminating the code without calling your defers is, well, at least strange. The files are all put into some proper state by the OS, there is absolutely no need to complicate the stuff like this.
Check the error from the second Copy:
2015/12/22 19:00:33
Deferred log:
unexpected EOF
exit status 1
The thing is, you need to close zdest immediately after you've done writing. Close it after the first Copy and it works.
I would have suggested to use io.MultiWriter.
In this way you read only once from src. Not much gain for small files but is faster for bigger files.
w := io.MultiWriter(dest, os.Stdout)