I have a function which splits data and returns slice of subslices:
(buf []byte, lim int) [][]byte
Obviously I get an error if I do:
n, err = out.Write(split(buf[:n], 100))
The error:
cannot convert split(buf[:n], 100) (type [][]byte) to type []byte
How do I convert [][]byte to []byte?
Edit based on #Wishwa Perera: https://play.golang.org/p/nApPAYRV4ZW
Since you are splitting buf into chunks, you can pass them individually to Write by looping over the result of split.
for _, chunk := range split(buf[:n], 100) {
if _, err := out.Write(chunk); err != nil {
panic(err)
}
}
If out is a net.Conn as in your other question, then use net.Buffers to write the [][]byte.
b := net.Buffers(split(buf[:n], 100))
_, err := b.WriteTo(out)
if err != nil {
panic(err)
}
Related
I'm trying to download and decrypt HLS streams by using io.ReadFull to process the data in chunks to conserve memory:
Irrelevant parts of code has been left out for simplicity.
func main() {
f, _ := os.Create(out.ts)
for _, v := range mediaPlaylist {
resp, _ := http.Get(v.URI)
for {
r, err := decryptHLS(key, iv, resp.Body)
if err != nil && err == io.EOF {
break
else if err != nil && err != io.ErrUnexpectedEOF {
panic(err)
}
io.Copy(f, r)
}
}
}
func decryptHLS(key []byte, iv []byte, r io.Reader) (io.Reader, error) {
block, _ := aes.NewCipher(key)
buf := make([]byte, 8192)
mode := cipher.NewCBCDecrypter(block, iv)
n, err := io.ReadFull(r, buf)
if err != nil && err != io.ErrUnexpectedEOF {
return nil, err
}
mode.CryptBlocks(buf, buf)
return bytes.NewReader(buf[:n]), err
}
At first this seems to work as file size is correct and no errors during download,
but the video is corrupted. Not completely as the file is still recognized as a video, but image and sound is distorted.
If I change the code to use ioutil.ReadAll instead, the final video files will no longer be corrupted:
func main() {
f, _ := os.Create(out.ts)
for _, v := range mediaPlaylist {
resp, _ := http.Get(v.URI)
segment, _ := ioutil.ReadAll(resp.Body)
r, _ := decryptHLS(key, iv, &segment)
io.Copy(f, r)
}
}
func decryptHLS(key []byte, iv []byte, s *[]byte) io.Reader {
block, _ := aes.NewCipher(key)
mode := cipher.NewCBCDecrypter(block, iv)
mode.CryptBlocks(*s, *s)
return bytes.NewReader(*s)
}
Any ideas why it works correctly when reading the entire segment into memory, and not when using io.ReadFull and processing it in chunks?
Internally, CBCDecrypter makes a copy of your iv, so subsequent blocks start with the initial IV rather than the one that's been mutated by previous decryptions.
Create the decrypter once, and you should be able to keep re-using it to decrypt block by block (assuming the block size is a multiple of the block size expected by this crypto algorithm).
I'm trying to write some convenience wrapper funcs that base64 encodes and decodes byte slices. (Can't understand why this is not conveniently provided in the stdlib.)
However this code (in playground):
func b64encode(b []byte) []byte {
encodedData := &bytes.Buffer{}
encoder := base64.NewEncoder(base64.URLEncoding, encodedData)
defer encoder.Close()
encoder.Write(b)
return encodedData.Bytes()
}
func b64decode(b []byte) ([]byte, error) {
dec := base64.NewDecoder(base64.URLEncoding, bytes.NewReader(b))
buf := &bytes.Buffer{}
_, err := io.Copy(buf, dec)
if err != nil {
return nil, err
}
return buf.Bytes(), nil
}
func main() {
b := []byte("hello")
e := b64encode(b)
d, err := b64decode(e)
if err != nil {
log.Fatalf("could not decode: %s", err)
}
fmt.Println(string(d))
}
generates truncated output when I try to print it:
hel
What's going on?
The defer executes when the function ends. That is AFTER the return statement has been evaluated.
The following works: https://play.golang.org/p/sYn-W6fZh1
func b64encode(b []byte) []byte {
encodedData := &bytes.Buffer{}
encoder := base64.NewEncoder(base64.URLEncoding, encodedData)
encoder.Write(b)
encoder.Close()
return encodedData.Bytes()
}
That being said, if it really is all in memory, you can avoid creating an encoder entirely. Instead, you can do something like:
func b64encode(b []byte) []byte {
ret := make([]byte, base64.URLEncoding.EncodedLen(len(b)))
base64.URLEncoding.Encode(ret, b)
return ret
}
An added benefit of doing it this way it it is more efficient since it only needs to allocate once. It also allows you to no longer ignore errors in the Write and Close methods.
To convert [][]byte to []string, I do this
data, err := ioutil.ReadFile("test.txt")
if err != nil {
return nil, err
}
db := bytes.Split(data, []uint8("\n"))
// Convert [][]byte to []string
s := make([]string, len(db))
for i, val := range db {
s[i] = string(val)
}
fmt.Printf("%v", s)
I am new to golang, I'm not sure is most efficient way to do this.
The most effective way would be to remove this step: db := bytes.Split(data, []uint8("\n")) and instead iterate over data like that:
func main() {
data, _ := ioutil.ReadFile("test.txt")
s := make([]string, 0)
start := 0
for i := range data {
if data[i] == '\n' {
elem := string(data[start : i-1])
s = append(s, elem)
start = i
}
}
fmt.Printf("%v", s)
}
Or if you want to convert [][]byte to []string:
func convert(data [][]byte) []string {
s := make([]string, len(data))
for row := range data {
s[row] = string(data[row])
}
return s
}
If you actually want to convert a file content to a []string, you can use bufio.Scanner which is cleaner (IMO) and more efficient than the code you posted:
func readFile(filename string) ([]string, error) {
file, err := os.Open(filename)
if err != nil {
return nil, err
}
defer file.Close()
scanner := bufio.NewScanner(file)
var data []string
for scanner.Scan() {
line := scanner.Text()
data = append(data, line)
}
if err = scanner.Err(); err != nil {
return nil, err
}
return data, nil
}
Here's a benchmark* comparing the original function (readFile1) and my function (readFile2):
BenchmarkReadFile1-8 300 4632189 ns/op 3035552 B/op 10570 allocs/op
BenchmarkReadFile2-8 1000 1695820 ns/op 2169655 B/op 10587 allocs/op
*the benchmark read a sample file of 1.2 MiB and ~10K lines
The new code runs in 36% of the time and 71% of the memory used by the original function.
I've written a little server which receives a blob of data in the form of an io.Reader, adds a header and streams the result back to the caller.
My implementation isn't particularly efficient as I'm buffering the blob's data in-memory so that I can calculate the blob's length, which needs to form part of the header.
I've seen some examples of io.Pipe() with io.TeeReader but they're more for splitting an io.Reader into two, and writing them away in parallel.
The blobs I'm dealing with are around 100KB, so not huge but if my server gets busy, memory's going to quickly become an issue...
Any ideas?
func addHeader(in io.Reader) (out io.Reader, err error) {
buf := new(bytes.Buffer)
if _, err = io.Copy(buf, in); err != nil {
return
}
header := bytes.NewReader([]byte(fmt.Sprintf("header:%d", buf.Len())))
return io.MultiReader(header, buf), nil
}
I appreciate it's not a good idea to return interfaces from functions but this code isn't destined to become an API, so I'm not too concerned with that bit.
In general, the only way to determine the length of data in an io.Reader is to read until EOF. There are ways to determine the length of the data for specific types.
func addHeader(in io.Reader) (out io.Reader, err error) {
n := 0
switch v := in.(type) {
case *bytes.Buffer:
n = v.Len()
case *bytes.Reader:
n = v.Len()
case *strings.Reader:
n = v.Len()
case io.Seeker:
cur, err := v.Seek(0, 1)
if err != nil {
return nil, err
}
end, err := v.Seek(0, 2)
if err != nil {
return nil, err
}
_, err = v.Seek(cur, 0)
if err != nil {
return nil, err
}
n = int(end - cur)
default:
var buf bytes.Buffer
if _, err := buf.ReadFrom(in); err != nil {
return nil, err
}
n = buf.Len()
in = &buf
}
header := strings.NewReader(fmt.Sprintf("header:%d", n))
return io.MultiReader(header, in), nil
}
This is similar to how the net/http package determines the content length of the request body.
For logging purposes I want to be able to quickly write a slice of any type, whether it be ints, strings, or custom structs, to a file in Go. For instance, in C#, I can do the following in 1 line:
File.WriteAllLines(filePath, myCustomTypeList.Select(x => x.ToString());
How would I go about doing this in Go? The structs implement the Stringer interface.
Edit: I in particular would like the output to be printed to a file and one line per item in the slice
Use the fmt package format values as strings and print to a file:
func printLines(filePath string, values []interface{}) error {
f, err := os.Create(filePath)
if err != nil {
return err
}
defer f.Close()
for _, value := range values {
fmt.Fprintln(f, value) // print values to f, one per line
}
return nil
}
fmt.Fprintln will call Stringer() on your struct type. It will also print int values and string values.
playground example
Use the reflect package to write any slice type:
func printLines(filePath string, values interface{}) error {
f, err := os.Create(filePath)
if err != nil {
return err
}
defer f.Close()
rv := reflect.ValueOf(values)
if rv.Kind() != reflect.Slice {
return errors.New("Not a slice")
}
for i := 0; i < rv.Len(); i++ {
fmt.Fprintln(f, rv.Index(i).Interface())
}
return nil
}
If you have variable values of type myCustomList, then you can call it like this: err := printLines(filePath, values)
playground example