I'm trying to write some convenience wrapper funcs that base64 encodes and decodes byte slices. (Can't understand why this is not conveniently provided in the stdlib.)
However this code (in playground):
func b64encode(b []byte) []byte {
encodedData := &bytes.Buffer{}
encoder := base64.NewEncoder(base64.URLEncoding, encodedData)
defer encoder.Close()
encoder.Write(b)
return encodedData.Bytes()
}
func b64decode(b []byte) ([]byte, error) {
dec := base64.NewDecoder(base64.URLEncoding, bytes.NewReader(b))
buf := &bytes.Buffer{}
_, err := io.Copy(buf, dec)
if err != nil {
return nil, err
}
return buf.Bytes(), nil
}
func main() {
b := []byte("hello")
e := b64encode(b)
d, err := b64decode(e)
if err != nil {
log.Fatalf("could not decode: %s", err)
}
fmt.Println(string(d))
}
generates truncated output when I try to print it:
hel
What's going on?
The defer executes when the function ends. That is AFTER the return statement has been evaluated.
The following works: https://play.golang.org/p/sYn-W6fZh1
func b64encode(b []byte) []byte {
encodedData := &bytes.Buffer{}
encoder := base64.NewEncoder(base64.URLEncoding, encodedData)
encoder.Write(b)
encoder.Close()
return encodedData.Bytes()
}
That being said, if it really is all in memory, you can avoid creating an encoder entirely. Instead, you can do something like:
func b64encode(b []byte) []byte {
ret := make([]byte, base64.URLEncoding.EncodedLen(len(b)))
base64.URLEncoding.Encode(ret, b)
return ret
}
An added benefit of doing it this way it it is more efficient since it only needs to allocate once. It also allows you to no longer ignore errors in the Write and Close methods.
Related
func handleClient(p2, p1 net.Conn)
streamCopy := func(dst io.Writer, src io.ReadCloser) {
if _, err := generic.Copy(dst, src); err != nil {
log.Println(err, "in:", p1.RemoteAddr(), "out:", fmt.Sprint(p2.RemoteAddr(), "(", p2.ID(), ")"))
}
p1.Close()
p2.Close()
}
//p1 p2 they all net.conn
//here i want read p1's data do something
go streamCopy(p1, p2)
streamCopy(p2, p1)
}
//for generic.Copy src code:
func Copy(dst io.Writer, src io.Reader) (written int64, err error) {
if wt, ok := src.(io.WriterTo); ok {
log.Println("io.WriterTo")
return wt.WriteTo(dst)
}
if rt, ok := dst.(io.ReaderFrom); ok {
log.Println("io.ReaderFrom")
return rt.ReadFrom(src)
}
buf := make([]byte, bufSize)
return io.CopyBuffer(dst, src, buf)
I want to do two things:
1.p1's data must be read before they are copy change.
2.then copy the stream,like reverse proxy, I try to use
buf := bytes.Buffer{}
tee := io.TeeReader(p1, buf),
I can read the data, but the stream copy fails, I hope there is a way to read the buffer without affecting p1...so i don't know
I have a function which splits data and returns slice of subslices:
(buf []byte, lim int) [][]byte
Obviously I get an error if I do:
n, err = out.Write(split(buf[:n], 100))
The error:
cannot convert split(buf[:n], 100) (type [][]byte) to type []byte
How do I convert [][]byte to []byte?
Edit based on #Wishwa Perera: https://play.golang.org/p/nApPAYRV4ZW
Since you are splitting buf into chunks, you can pass them individually to Write by looping over the result of split.
for _, chunk := range split(buf[:n], 100) {
if _, err := out.Write(chunk); err != nil {
panic(err)
}
}
If out is a net.Conn as in your other question, then use net.Buffers to write the [][]byte.
b := net.Buffers(split(buf[:n], 100))
_, err := b.WriteTo(out)
if err != nil {
panic(err)
}
I'm trying to download and decrypt HLS streams by using io.ReadFull to process the data in chunks to conserve memory:
Irrelevant parts of code has been left out for simplicity.
func main() {
f, _ := os.Create(out.ts)
for _, v := range mediaPlaylist {
resp, _ := http.Get(v.URI)
for {
r, err := decryptHLS(key, iv, resp.Body)
if err != nil && err == io.EOF {
break
else if err != nil && err != io.ErrUnexpectedEOF {
panic(err)
}
io.Copy(f, r)
}
}
}
func decryptHLS(key []byte, iv []byte, r io.Reader) (io.Reader, error) {
block, _ := aes.NewCipher(key)
buf := make([]byte, 8192)
mode := cipher.NewCBCDecrypter(block, iv)
n, err := io.ReadFull(r, buf)
if err != nil && err != io.ErrUnexpectedEOF {
return nil, err
}
mode.CryptBlocks(buf, buf)
return bytes.NewReader(buf[:n]), err
}
At first this seems to work as file size is correct and no errors during download,
but the video is corrupted. Not completely as the file is still recognized as a video, but image and sound is distorted.
If I change the code to use ioutil.ReadAll instead, the final video files will no longer be corrupted:
func main() {
f, _ := os.Create(out.ts)
for _, v := range mediaPlaylist {
resp, _ := http.Get(v.URI)
segment, _ := ioutil.ReadAll(resp.Body)
r, _ := decryptHLS(key, iv, &segment)
io.Copy(f, r)
}
}
func decryptHLS(key []byte, iv []byte, s *[]byte) io.Reader {
block, _ := aes.NewCipher(key)
mode := cipher.NewCBCDecrypter(block, iv)
mode.CryptBlocks(*s, *s)
return bytes.NewReader(*s)
}
Any ideas why it works correctly when reading the entire segment into memory, and not when using io.ReadFull and processing it in chunks?
Internally, CBCDecrypter makes a copy of your iv, so subsequent blocks start with the initial IV rather than the one that's been mutated by previous decryptions.
Create the decrypter once, and you should be able to keep re-using it to decrypt block by block (assuming the block size is a multiple of the block size expected by this crypto algorithm).
I want to copy a os.Stdin string to a buffer, to check for a user inputted text (e.g. "hibye") and put an if statement against it.
My current code just handles simple stdin stdiout copy operations (no buffer):
func interact(c net.Conn) {
// Read from Reader and write to Writer until EOF()
copy := func(r io.ReadCloser, w io.WriteCloser) {
defer func() {
r.Close()
w.Close()
}()
n, err := io.Copy(w, r)
if err != nil {
log.Printf("[%s]: ERROR: %s\n", c.RemoteAddr(), err)
log.Println(n)
}
}
go copy(c, os.Stdout)
go copy(os.Stdin, c)
}
Question: What is the most efficient way to implement a use of a buffer to have control over the passed strings?
bad example (failed attempt):
buf := make([]byte, 1024)
go copy (os.Stdin, []byte(buf))
if buf == "hibye" {
do stuff
}
I am trying to read a buffered stream of signed 16 bit integers (wav format), but the bufio.Read method only accepts an array of bytes. My question is a 2-parter:
Can I preformat the byte stream into a buffered int16 array?
If I can't, whats the best way of post-processing the byte array into int16 array? My initial thought is to use tmp arrays and keep pushing/processing them, but I was curious if there was a more idiomatic way of doing this?
package main
import (
"bufio"
"io"
"log"
"os/exec"
)
func main() {
app := "someapp"
cmd := exec.Command(app)
stdout, err := cmd.StdoutPipe()
r := bufio.NewReader(stdout)
if err != nil {
log.Fatal(err)
}
if err := cmd.Start(); err != nil {
log.Fatal(err)
}
//"someapp" outputs signed 16bit integers (little endian))
buf := make([]byte, 0, 4*1024)
for {
n, err := r.Read(buf[:cap(buf)]) //r.Read only accepts type []byte
buf = buf[:n]
if n == 0 {
if err == nil {
continue
}
if err == io.EOF {
break
}
log.Fatal(err)
}
log.Printf("%x\n", buf)
//process buf here
if err != nil && err != io.EOF {
log.Fatal(err)
}
}
}
When working with IO, you always work with []bytes, there's no way to substitute that with []int16, or pre-format that as int16s, it's always a stream of bytes.
You can look at the encoding/binary package to decode this stream.
// to get the first uint16 as i
i := binary.LittleEndian.Uint16(buf[:2])
You can then iterate through the buf as needed.
You can also use binary.Read to read directly from the io.Reader.
var i uint16
for {
err := binary.Read(r, binary.LittleEndian, &i)
if err != nil {
log.Println(err)
break
}
fmt.Println(i)
}
It may worth noting the simplicity of what needs to be done. Each uint16 is created via:
func (littleEndian) Uint16(b []byte) uint16 {
return uint16(b[0]) | uint16(b[1])<<8
}
You can use encoding/binary.Read to fill an []int16 directly from your reader, although technically the answer to your first question is still no (check the source of binary.Read, it reads the data to a []byte first).