I'm new to golang and I'm having problem understanding go's io.Pipe. Is this similar to node.js' .pipe? And how should I use it? Is it possible to use it with 1 read file and a write file?
Thank in advance guys.
No, they are not precisely similar. io.Copy(dat io.Writer, src io.Reader) is quite enough to read and write files, like this:
input := bufio.NewReader(os.Stdin)
output := bufio.NewWriter(os.Stdout) // buffer output like C stdlib
io.Copy(output, input) // copy entire file
output.Flush()
io.Pipe() (*PipeReader, *PipeWriter) will produce piped Reader and Writer for you when you have not them but code expect them, like this:
type id struct{
name string
age int
}
payload := id{"John", 25}
requestBody, jsonPayload := io.Pipe()
request := http.NewRequest("POST". "http://www.example.com", requestBody) // NewRequest expect io.Reader
encoder := json.NewEncoder(jsonPayload) // NewEncoder expect io.Writer
err := encoder.Encode(payload)
response, err := client.Do(request)
Related
I am making a go program where I need to write a gob to a file. I used the .String() method to convert the gob to a string.
var network bytes.Buffer
encoder := gob.NewEncoder(&network)
_ = encoder.Encode(valueToEncode)
gobString := network.String()
then I will write the gob to a file, and later I will retrieve it and send it to this program:
var filebytes = []byte(file) //i think that these two lines are the issue
network := bytes.NewBuffer(filebytes)
decoder := gob.NewDecoder(network)
var decoded interface{}
_ := decoder.Decode(&decoded)
but when i run this, it gives me this error:
gob: encoded unsigned integer out of range
I think the issue is with the first two lines of the decoder program. So what should I put to properly decode the gob?
EDIT:
What I want is a .UnString() method for the gobString. How can i achieve that?
The encoding/gob generates binary data from Go values. The result is not for textual representation, so you should not treat it as a string, but as a series of bytes, e.g. []byte.
That said, do not use Buffer.String() but rather Buffer.Bytes() if you must obtain the encoded data.
Here's an example encoding and decoding a string value using encoding/gob:
// ENCODE
var network bytes.Buffer
encoder := gob.NewEncoder(&network)
valueToEncode := "Hello, 世界"
if err := encoder.Encode(valueToEncode); err != nil {
panic(err)
}
gobData := network.Bytes() // Save / serialize this byte slice
// DECODE
network2 := bytes.NewBuffer(gobData)
decoder := gob.NewDecoder(network2)
var decoded string
if err := decoder.Decode(&decoded); err != nil {
panic(err)
}
fmt.PrintTln(decoded)
Output (try it on the Go Playground):
Hello, 世界
Also note that if you intend to write the encoded data into a network connection or a file, you don't need bytes.Buffer, you can directly encode to those. If you must use bytes.Buffer, you may also use its Buffer.WriteTo() method to write its contents into an io.Writer (such as a file or network connection).
What's the best way to broadcast values from a stream to two network destination simultaneously? Here is simplified code:
func main() {
resp, _ := http.Get("http://origin.com/image.jpeg")
var buf bytes.Buffer
// tee, when read, writes to &buf
respBodyTee := io.TeeReader(resp.Body, &buf)
sendToClient(respBodyTee)
uploadeToServer(&buf)
}
A stream cannot be read twice, so TeeReader is used to populate &buf with whatever is read from respo.Body.
However, the functions (sendToClient and uploadToServer) will run synchronously while I'd like to them to make their work concurrently.
Solution that I have on mind is to pass a channel to sendToClient that will populate channel with bytes already sent to client. Later have uploadToServer read from the same channel. Something along these lines:
func main() {
resp, _ := http.Get("http://origin.com/image.jpeg")
ch := make(chan byte)
go sendToClient(respBodyTee, ch) // pass 'ch' for writing and run in a goroutine
uploadeToServer(ch) // will read from 'ch' (synchronous)
}
I'm new to Go and am not sure if the above is the right direction.
in your scenario, it's better to have 2 independent byte streams for 2 network calls. If they rely on one source stream, when sendToClient stalls uploadeToServer will hang. channel wouldn't solve the problem above but introducing locking overhead.
you can try io.MultiWriter to make 2 independent byte streams
var buf1, buf2 bytes.Buffer
mw := io.MultiWriter(&buf1, &buf2)
if _, err := io.Copy(mw, r.Body); err != nil {
...
}
go sendToClient(buf1)
go uploadeToServer(buf2)
...
I am building a Go application that takes an http.Response object and saves it (response headers and body) to a redis hash. When the application receives an http.Response.Body that is not gzipped, I want to gzip it before saving it to the cache.
My confusion stems from my inability to make clear sense of Go's io interfaces, and how to negotiate between http.Response.Body's io.ReadCloser and the gzip Writer. I imagine there is an elegant, streaming solution here, but I can't quite get it to work.
If you've already determined the body is uncompressed, and if you need a []byte of the compressed data (instead of for example already having an io.Writer you could write to, e.g. if you wanted to save the body to a file then you'd want to stream into the file not into a buffer) then something like this should work:
func getCompressedBody(r *http.Response) ([]byte, error) {
var buf bytes.Buffer
gz := gzip.NewWriter(&buf)
if _, err := io.Copy(gz, r.Body); err != nil {
return nil, err
}
err := gz.Close()
return buf.Bytes(), err
}
(this is just an example and would probably be in-line instead of as a function; if you wanted it as a fuction then it should probably take an io.Reader instead of an *http.Response).
I'm trying, just for fun, to connect a gzip Writer directly to a gzip Reader, so I could write to the Writer and read from the Reader on the fly. I expected to read exactly what I wrote. I'm using gzip but I'd like to use this method also with crypto/aes, I suppose it should work very similar and it could be used with other reader/writers like jpeg, png...
This is my best option, that is not working, but I hope you can see what I mean: http://play.golang.org/p/7qdUi9wwG7
package main
import (
"bytes"
"compress/gzip"
"fmt"
)
func main() {
s := []byte("Hello world!")
fmt.Printf("%s\n", s)
var b bytes.Buffer
gz := gzip.NewWriter(&b)
ungz, err := gzip.NewReader(&b)
fmt.Println("err: ", err)
gz.Write(s)
gz.Flush()
uncomp := make([]byte, 100)
n, err2 := ungz.Read(uncomp)
fmt.Println("err2: ", err2)
fmt.Println("n: ", n)
uncomp = uncomp[:n]
fmt.Printf("%s\n", uncomp)
}
It seems that gzip.NewReader(&b) is trying to read immediately and a EOF is returned.
You'll need to do two things to make it work
Use an io.Pipe to connect the reader and writer together - you can't read and write from the same buffer
Run the reading and writing in seperate goroutines. Because the first thing that gzip does is attempt to read the header you'll get a deadlock unless you have another go routine attemting to write it.
Here is what that looks like
Playground
func main() {
s := []byte("Hello world!")
fmt.Printf("%s\n", s)
in, out := io.Pipe()
gz := gzip.NewWriter(out)
go func() {
ungz, err := gzip.NewReader(in)
fmt.Println("err: ", err)
uncomp := make([]byte, 100)
n, err2 := ungz.Read(uncomp)
fmt.Println("err2: ", err2)
fmt.Println("n: ", n)
uncomp = uncomp[:n]
fmt.Printf("%s\n", uncomp)
}()
gz.Write(s)
gz.Flush()
}
Use a pipe. For example,
Package io
func Pipe
func Pipe() (*PipeReader, *PipeWriter)
Pipe creates a synchronous in-memory pipe. It can be used to connect
code expecting an io.Reader with code expecting an io.Writer. Reads on
one end are matched with writes on the other, copying data directly
between the two; there is no internal buffering. It is safe to call
Read and Write in parallel with each other or with Close. Close will
complete once pending I/O is done. Parallel calls to Read, and
parallel calls to Write, are also safe: the individual calls will be
gated sequentially.
I have an io.ReadCloser object (from an http.Response object).
What's the most efficient way to convert the entire stream to a string object?
EDIT:
Since 1.10, strings.Builder exists. Example:
buf := new(strings.Builder)
n, err := io.Copy(buf, r)
// check errors
fmt.Println(buf.String())
OUTDATED INFORMATION BELOW
The short answer is that it it will not be efficient because converting to a string requires doing a complete copy of the byte array. Here is the proper (non-efficient) way to do what you want:
buf := new(bytes.Buffer)
buf.ReadFrom(yourReader)
s := buf.String() // Does a complete copy of the bytes in the buffer.
This copy is done as a protection mechanism. Strings are immutable. If you could convert a []byte to a string, you could change the contents of the string. However, go allows you to disable the type safety mechanisms using the unsafe package. Use the unsafe package at your own risk. Hopefully the name alone is a good enough warning. Here is how I would do it using unsafe:
buf := new(bytes.Buffer)
buf.ReadFrom(yourReader)
b := buf.Bytes()
s := *(*string)(unsafe.Pointer(&b))
There we go, you have now efficiently converted your byte array to a string. Really, all this does is trick the type system into calling it a string. There are a couple caveats to this method:
There are no guarantees this will work in all go compilers. While this works with the plan-9 gc compiler, it relies on "implementation details" not mentioned in the official spec. You can not even guarantee that this will work on all architectures or not be changed in gc. In other words, this is a bad idea.
That string is mutable! If you make any calls on that buffer it will change the string. Be very careful.
My advice is to stick to the official method. Doing a copy is not that expensive and it is not worth the evils of unsafe. If the string is too large to do a copy, you should not be making it into a string.
Answers so far haven't addressed the "entire stream" part of the question. I think the good way to do this is ioutil.ReadAll. With your io.ReaderCloser named rc, I would write,
Go >= v1.16
if b, err := io.ReadAll(rc); err == nil {
return string(b)
} ...
Go <= v1.15
if b, err := ioutil.ReadAll(rc); err == nil {
return string(b)
} ...
data, _ := ioutil.ReadAll(response.Body)
fmt.Println(string(data))
func copyToString(r io.Reader) (res string, err error) {
var sb strings.Builder
if _, err = io.Copy(&sb, r); err == nil {
res = sb.String()
}
return
}
The most efficient way would be to always use []byte instead of string.
In case you need to print data received from the io.ReadCloser, the fmt package can handle []byte, but it isn't efficient because the fmt implementation will internally convert []byte to string. In order to avoid this conversion, you can implement the fmt.Formatter interface for a type like type ByteSlice []byte.
var b bytes.Buffer
b.ReadFrom(r)
// b.String()
I like the bytes.Buffer struct. I see it has ReadFrom and String methods. I've used it with a []byte but not an io.Reader.