I want to copy a os.Stdin string to a buffer, to check for a user inputted text (e.g. "hibye") and put an if statement against it.
My current code just handles simple stdin stdiout copy operations (no buffer):
func interact(c net.Conn) {
// Read from Reader and write to Writer until EOF()
copy := func(r io.ReadCloser, w io.WriteCloser) {
defer func() {
r.Close()
w.Close()
}()
n, err := io.Copy(w, r)
if err != nil {
log.Printf("[%s]: ERROR: %s\n", c.RemoteAddr(), err)
log.Println(n)
}
}
go copy(c, os.Stdout)
go copy(os.Stdin, c)
}
Question: What is the most efficient way to implement a use of a buffer to have control over the passed strings?
bad example (failed attempt):
buf := make([]byte, 1024)
go copy (os.Stdin, []byte(buf))
if buf == "hibye" {
do stuff
}
Related
func handleClient(p2, p1 net.Conn)
streamCopy := func(dst io.Writer, src io.ReadCloser) {
if _, err := generic.Copy(dst, src); err != nil {
log.Println(err, "in:", p1.RemoteAddr(), "out:", fmt.Sprint(p2.RemoteAddr(), "(", p2.ID(), ")"))
}
p1.Close()
p2.Close()
}
//p1 p2 they all net.conn
//here i want read p1's data do something
go streamCopy(p1, p2)
streamCopy(p2, p1)
}
//for generic.Copy src code:
func Copy(dst io.Writer, src io.Reader) (written int64, err error) {
if wt, ok := src.(io.WriterTo); ok {
log.Println("io.WriterTo")
return wt.WriteTo(dst)
}
if rt, ok := dst.(io.ReaderFrom); ok {
log.Println("io.ReaderFrom")
return rt.ReadFrom(src)
}
buf := make([]byte, bufSize)
return io.CopyBuffer(dst, src, buf)
I want to do two things:
1.p1's data must be read before they are copy change.
2.then copy the stream,like reverse proxy, I try to use
buf := bytes.Buffer{}
tee := io.TeeReader(p1, buf),
I can read the data, but the stream copy fails, I hope there is a way to read the buffer without affecting p1...so i don't know
I'm trying to download and decrypt HLS streams by using io.ReadFull to process the data in chunks to conserve memory:
Irrelevant parts of code has been left out for simplicity.
func main() {
f, _ := os.Create(out.ts)
for _, v := range mediaPlaylist {
resp, _ := http.Get(v.URI)
for {
r, err := decryptHLS(key, iv, resp.Body)
if err != nil && err == io.EOF {
break
else if err != nil && err != io.ErrUnexpectedEOF {
panic(err)
}
io.Copy(f, r)
}
}
}
func decryptHLS(key []byte, iv []byte, r io.Reader) (io.Reader, error) {
block, _ := aes.NewCipher(key)
buf := make([]byte, 8192)
mode := cipher.NewCBCDecrypter(block, iv)
n, err := io.ReadFull(r, buf)
if err != nil && err != io.ErrUnexpectedEOF {
return nil, err
}
mode.CryptBlocks(buf, buf)
return bytes.NewReader(buf[:n]), err
}
At first this seems to work as file size is correct and no errors during download,
but the video is corrupted. Not completely as the file is still recognized as a video, but image and sound is distorted.
If I change the code to use ioutil.ReadAll instead, the final video files will no longer be corrupted:
func main() {
f, _ := os.Create(out.ts)
for _, v := range mediaPlaylist {
resp, _ := http.Get(v.URI)
segment, _ := ioutil.ReadAll(resp.Body)
r, _ := decryptHLS(key, iv, &segment)
io.Copy(f, r)
}
}
func decryptHLS(key []byte, iv []byte, s *[]byte) io.Reader {
block, _ := aes.NewCipher(key)
mode := cipher.NewCBCDecrypter(block, iv)
mode.CryptBlocks(*s, *s)
return bytes.NewReader(*s)
}
Any ideas why it works correctly when reading the entire segment into memory, and not when using io.ReadFull and processing it in chunks?
Internally, CBCDecrypter makes a copy of your iv, so subsequent blocks start with the initial IV rather than the one that's been mutated by previous decryptions.
Create the decrypter once, and you should be able to keep re-using it to decrypt block by block (assuming the block size is a multiple of the block size expected by this crypto algorithm).
If I'm opening a file inside a for loop and will be finished with it at the end of that iteration, should I call Close immediately or trick Defer using a closure?
I have a series of filenames being read in from a chan string which have data to be copied into a zipfile. This is all being processed in a go func.
go func(fnames <-chan string, zipfilename string) {
f, _ := os.Create(zipfilename) // ignore error handling for this example
defer f.Close()
zf := zip.NewWriter(f)
defer zf.Close()
for fname := range fnames {
r, _ := os.Open(fname)
w, _ := zf.Create(r.Name())
io.Copy(w, r)
w.Close()
r.Close()
}(files, "some name.zip")
Inside my for loop, would it be more idiomatic Go to write:
for fname := range fnames {
func(){
r, _ := os.Open(fname)
defer r.Close()
w, _ := zf.Create(r.Name())
defer w.Close()
io.Copy(w, r)
}()
}
or should I continue with my code as-written?
You should be checking your errors. I know this is meant to just be an example, but in this case it is important. If all you do is defer Close(), you can't actually check if there was an error during defer.
The way I would write this is to create a helper function:
func copyFileToZip(zf *zip.Writer, filename string) error {
r, err := os.Open(filename)
if err != nil {
return err
}
defer r.Close()
w, err := zf.Create(r.Name())
if err != nil {
return err
}
defer w.Close()
_, err = io.Copy(w, r)
if err != nil {
return err
}
return w.Close()
}
Once you add in all that error handling, the function is big enough to make it a named function. It also has the added benefit of checking the error when closing the writer. Checking the reader's error is unnecessary since that won't affect if the data was written.
I'm trying to write some convenience wrapper funcs that base64 encodes and decodes byte slices. (Can't understand why this is not conveniently provided in the stdlib.)
However this code (in playground):
func b64encode(b []byte) []byte {
encodedData := &bytes.Buffer{}
encoder := base64.NewEncoder(base64.URLEncoding, encodedData)
defer encoder.Close()
encoder.Write(b)
return encodedData.Bytes()
}
func b64decode(b []byte) ([]byte, error) {
dec := base64.NewDecoder(base64.URLEncoding, bytes.NewReader(b))
buf := &bytes.Buffer{}
_, err := io.Copy(buf, dec)
if err != nil {
return nil, err
}
return buf.Bytes(), nil
}
func main() {
b := []byte("hello")
e := b64encode(b)
d, err := b64decode(e)
if err != nil {
log.Fatalf("could not decode: %s", err)
}
fmt.Println(string(d))
}
generates truncated output when I try to print it:
hel
What's going on?
The defer executes when the function ends. That is AFTER the return statement has been evaluated.
The following works: https://play.golang.org/p/sYn-W6fZh1
func b64encode(b []byte) []byte {
encodedData := &bytes.Buffer{}
encoder := base64.NewEncoder(base64.URLEncoding, encodedData)
encoder.Write(b)
encoder.Close()
return encodedData.Bytes()
}
That being said, if it really is all in memory, you can avoid creating an encoder entirely. Instead, you can do something like:
func b64encode(b []byte) []byte {
ret := make([]byte, base64.URLEncoding.EncodedLen(len(b)))
base64.URLEncoding.Encode(ret, b)
return ret
}
An added benefit of doing it this way it it is more efficient since it only needs to allocate once. It also allows you to no longer ignore errors in the Write and Close methods.
I am trying to stream out bytes of a zip file using io.Pipe() function in golang. I am using pipe reader to read the bytes of each file in the zip and then stream those out and use the pipe writer to write the bytes in the response object.
func main() {
r, w := io.Pipe()
// go routine to make the write/read non-blocking
go func() {
defer w.Close()
bytes, err := ReadBytesforEachFileFromTheZip()
err := json.NewEncoder(w).Encode(bytes)
handleErr(err)
}()
This is not a working implementation but a structure of what I am trying to achieve. I don't want to use ioutil.ReadAll since the file is going to be very large and Pipe() will help me avoid bringing all the data into memory. Can someone help with a working implementation using io.Pipe() ?
I made it work using golang io.Pipe().The Pipewriter writes byte to the pipe in chunks and the pipeReader reader from the other end. The reason for using a go-routine is to have a non-blocking write operation while simultaneous reads happen form the pipe.
Note: It's important to close the pipe writer (w.Close()) to send EOF on the stream otherwise it will not close the stream.
func DownloadZip() ([]byte, error) {
r, w := io.Pipe()
defer r.Close()
defer w.Close()
zip, err := os.Stat("temp.zip")
if err != nil{
return nil, err
}
go func(){
f, err := os.Open(zip.Name())
if err != nil {
return
}
buf := make([]byte, 1024)
for {
chunk, err := f.Read(buf)
if err != nil && err != io.EOF {
panic(err)
}
if chunk == 0 {
break
}
if _, err := w.Write(buf[:chunk]); err != nil{
return
}
}
w.Close()
}()
body, err := ioutil.ReadAll(r)
if err != nil {
return nil, err
}
return body, nil
}
Please let me know if someone has another way of doing it.