I have a setup where I receive data over the network and serialize it to my struct. It works fine, but now I need to serialize the data to a slice buffer to send it across the network.
I am trying to avoid having to allocate more than needed so I have already set up a buffer which I like to write to for all my serializing. But am not sure how to do this.
My setup is like this:
recieveBuffer := make([]byte, 1500)
header := recieveBuffer[0:1]
message := recieveBuffer[1:]
So I am trying to write fields from a struct to message and the total number of bytes for all the fields as a value for header.
This was how I deserialized to the struct:
// Deserialize ...
func (userSession *UserSession) Deserialize(message []byte) {
userSession.UID = int64(binary.LittleEndian.Uint32(message[0:4]))
userSession.UUID = string(message[4:40])
userSession.Username = string(message[40:])
}
I don't really know how to do the reverse of this, however. Is it possible without creating buffers for each field I want to serialize before copying to message?
Given the preallocated buffer buf, you can reverse the process like this:
buf[0] = byte(40+len(userSession.Username))
binary.LittleEndian.PutUint32(buf[1:], uint32(int32(userSession.UID)))
copy(buf[5:41], userSession.UUID)
copy(buf[41:], userSession.Username)
Given two helper functions.
One to encode a primitive to a byte slice:
func EncodeNumber2NetworkOrder(v interface{}) ([]byte, error) {
switch v.(type) {
case int: // int is at least 32 bits
b := make([]byte, 4)
binary.BigEndian.PutUint32(b, uint32(v.(int)))
return b, nil
case int8:
b := []byte{byte(v.(int8))}
return b, nil
// ... truncated
and one to convert primitive, non-byte slices to a byte slice
func EncodeBigEndian(in []float64) []byte {
var out []byte = make([]byte, len(in)*8)
var wg sync.WaitGroup
wg.Add(len(in))
for i := 0; i < len(in); i++ {
go func(out *[]byte, i int, f float64) {
defer wg.Done()
binary.BigEndian.PutUint64((*out)[(i<<3):], math.Float64bits(f))
}(&out, i, in[i])
}
wg.Wait()
return out
}
your binary serialization might look like this for a bogus struct like
type Foo struct {
time int64
data []float64
}
func Encode(f *Foo) []byte {
da := encoder.EncodeBigEndian(f.data)
bytes := make([]byte,0)
bytes = append(bytes, da...)
return bytes
}
Related
I have a function that can take a number of different argument types. I'd like to use a type switch and reduce code duplication as much as possible. As a very basic example, here I want to copy both uint8 and int8 types into a byte buffer. This code happily works
package main
func switchFn(args ...interface{}) {
var buf []byte
for _, arg := range args {
switch val := arg.(type) {
case uint8:
buf = append(buf, byte(val))
case int8:
buf = append(buf, byte(val))
}
}
}
func main() {
switchFn(int8(42), uint8(42)) // etc
}
You'll notice both the case statements do exactly the same thing! If I combine them though...
package main
func switchFn(args ...interface{}) {
var buf []byte
for _, arg := range args {
switch val := arg.(type) {
case uint8, int8:
buf = append(buf, byte(val))
}
}
}
func main() {
switchFn(int8(42), uint8(42)) // etc
}
I run into an issue of cannot convert val (type interface {}) to type byte: need type assertion. But I'm literally switching on the type! Argh!
Am I stuck with the code duplication here, or is there a smarter way to do this? Note that the copying into byte buffer is used to illustrate the example, my function may be doing other things in the case blocks.
The cases can be combined, but val will have type interface{} in the block. That's not useful for your scenario.
Use a function to reduce code duplication.
func switchFn(args ...interface{}) {
var buf []byte
byteFn := func(b byte) {
buf = append(buf, b)
}
for _, arg := range args {
switch val := arg.(type) {
case uint8:
byteFn(val)
case int8:
byteFn(byte(val))
}
}
}
The reflect API will not help because separate code is required for the signed and unsigned values. The reflect API is helpful for combining all signed integers into block of code and all unsigned integers into another block of code.
for _, arg := range args {
switch val := arg.(type) {
case int, int8, int16, int32, int64:
i := reflect.ValueOf(val).Int()
// i is an int64
fmt.Println(i)
case uint, uint8, uint16, uint32, uint64:
u := reflect.ValueOf(val).Uint()
// u is an uint64
fmt.Println(u)
}
}
Here's a way to avoid the code duplication, at the cost of ... well, a different sort of code duplication:
func switchFn(args ...interface{}) {
var buf []byte
for _, arg := range args {
var val byte
switch v := arg.(type) {
case uint8:
val = byte(v)
case int8:
val = byte(v)
default:
panic("wrong type")
}
buf = append(buf, val)
}
}
For this particular function, the original duplication is probably better. Should the buf = append(buf, val) section get larger or more complicated, this would probably be better.
In still other cases—perhaps most real ones—the method gopher suggests is probably best:
f := func(val byte) {
buffer = append(buffer, val)
}
You can now call f from each case.
I am using a third party library to generate PDFs. In order to write the PDF at the end (after all of content has been added using the lib's API), the pdfWriter type has a Write function that expects an io.WriteSeeker.
This is OK if I want to work with files, but I need to work in-memory. Trouble is, I can't find any way to do this - the only native type I found that implements io.WriteSeeker is File.
This is the part that works by using File for the io.Writer in the Write function of the pdfWriter:
fWrite, err := os.Create(outputPath)
if err != nil {
return err
}
defer fWrite.Close()
err = pdfWriter.Write(fWrite)
Is there way to do this without an actual File? Like getting a []byte or something?
Unfortunately there is no ready solution for an in-memory io.WriteSeeker implementation in the standard lib.
But as always, you can always implement your own. It's not that hard.
An io.WriteSeeker is an io.Writer and an io.Seeker, so basically you only need to implement 2 methods:
Write(p []byte) (n int, err error)
Seek(offset int64, whence int) (int64, error)
Read the general contract of these methods in their documentation how they should behave.
Here's a simple implementation which uses an in-memory byte slice ([]byte). It's not optimized for speed, this is just a "demo" implementation.
type mywriter struct {
buf []byte
pos int
}
func (m *mywriter) Write(p []byte) (n int, err error) {
minCap := m.pos + len(p)
if minCap > cap(m.buf) { // Make sure buf has enough capacity:
buf2 := make([]byte, len(m.buf), minCap+len(p)) // add some extra
copy(buf2, m.buf)
m.buf = buf2
}
if minCap > len(m.buf) {
m.buf = m.buf[:minCap]
}
copy(m.buf[m.pos:], p)
m.pos += len(p)
return len(p), nil
}
func (m *mywriter) Seek(offset int64, whence int) (int64, error) {
newPos, offs := 0, int(offset)
switch whence {
case io.SeekStart:
newPos = offs
case io.SeekCurrent:
newPos = m.pos + offs
case io.SeekEnd:
newPos = len(m.buf) + offs
}
if newPos < 0 {
return 0, errors.New("negative result pos")
}
m.pos = newPos
return int64(newPos), nil
}
Yes, and that's it.
Testing it:
my := &mywriter{}
var ws io.WriteSeeker = my
ws.Write([]byte("hello"))
fmt.Println(string(my.buf))
ws.Write([]byte(" world"))
fmt.Println(string(my.buf))
ws.Seek(-2, io.SeekEnd)
ws.Write([]byte("k!"))
fmt.Println(string(my.buf))
ws.Seek(6, io.SeekStart)
ws.Write([]byte("gopher"))
fmt.Println(string(my.buf))
Output (try it on the Go Playground):
hello
hello world
hello work!
hello gopher
Things that can be improved:
Create a mywriter value with an initial empty buf slice, but with a capacity that will most likely cover the size of the result PDF document. E.g. if you estimate the result PDFs are around 1 MB, create a buffer with capacity for 2 MB like this:
my := &mywriter{buf: make([]byte, 0, 2<<20)}
Inside mywriter.Write() when capacity needs to be increased (and existing content copied over), it may be profitable to use bigger increment, e.g. double the current capacity to a certain extent, which reserves space for future appends and minimizes the reallocations.
I been reading about goroutines and the sync package and my question is... Do I always need to lock unlock when reading writting to data on different goroutines?
For example I have a variable on my server
config := make(map[string]string)
Then on different goroutines I want to read from config. Is it safe to read without using sync or it is not?
I guess writting needs to be done using the sync package. but I am not sure about reading
For example I have a simple in-memory cache system
type Cache interface {
Get(key string) interface{}
Put(key string, expires int64, value interface{})
}
// MemoryCache represents a memory type of cache
type MemoryCache struct {
c map[string]*MemoryCacheValue
rw sync.RWMutex
}
// MemoryCacheValue represents a memory cache value
type MemoryCacheValue struct {
value interface{}
expires int64
}
// NewMemoryCache creates a new memory cache
func NewMemoryCache() Cache {
return &MemoryCache{
c: make(map[string]*MemoryCacheValue),
}
}
// Get stores something into the cache
func (m *MemoryCache) Get(key string) interface{} {
if v, ok := m.c[key]; ok {
return v
}
return nil
}
// Put retrieves something from the cache
func (m *MemoryCache) Put(key string, expires int64, value interface{}) {
m.rw.Lock()
m.c[key] = &MemoryCacheValue{
value,
time.Now().Unix() + expires,
}
m.rw.Unlock()
}
I am acting safe here or I still need to lock unlock when I want to only read?
You're diving into the world of race conditions. The basic rule of thumb is that if ANY routine writes to or changes a piece of data that can be or is read by (or also written to) by any number of other coroutines/threads, you need to have some sort of synchronization system in place.
For example, lets say you have that map. It has ["Joe"] = "Smith" and ["Sean"] = "Howard" in it. One goroutine wants to read the value of ["Joe"]. Another routine is updating ["Joe"] to "Cooper". Which value does the first goroutine read? Depends on which goroutine gets to the data first. That's the race condition, the behavior is undefined and unpredictable.
The easiest method to control that access is with a sync.Mutex. In your case, since some routines only need to read and not write, you can instead use a sync.RWMutex (main difference is that a RWMutex allows any number of threads to read, as long as none are trying to write). You would bake this into the map using a structure like this:
type MutexMap struct {
m map[string]string
*sync.RWMutex
}
Then, in routines that need to read from the map, you would do:
func ReadSomething(o MutexMap, key string) string {
o.RLock() // lock for reading, blocks until the Mutex is ready
defer o.RUnlock() // make SURE you do this, else it will be locked permanently
return o.m[key]
}
And to write:
func WriteSomething(o MutexMap, key, value string) {
o.Lock() // lock for writing, blocks until the Mutex is ready
defer o.Unlock() // again, make SURE you do this, else it will be locked permanently
o.m[key] = value
}
Note that both of these could be written as methods of the struct, rather than functions, if desired.
You can also approach this using channels. You make a controller structure that runs in a goroutine, and you make requests to it over channels. Example:
package main
import "fmt"
type MapCtrl struct {
m map[string]string
ReadCh chan chan map[string]string
WriteCh chan map[string]string
QuitCh chan struct{}
}
func NewMapController() *MapCtrl {
return &MapCtrl{
m: make(map[string]string),
ReadCh: make(chan chan map[string]string),
WriteCh: make(chan map[string]string),
QuitCh: make(chan struct{}),
}
}
func (ctrl *MapCtrl) Control() {
for {
select {
case r := <-ctrl.ReadCh:
fmt.Println("Read request received")
retmap := make(map[string]string)
for k, v := range ctrl.m { // copy map, so it doesn't change in place after return
retmap[k] = v
}
r <- retmap
case w := <-ctrl.WriteCh:
fmt.Println("Write request received with", w)
for k, v := range w {
ctrl.m[k] = v
}
case <-ctrl.QuitCh:
fmt.Println("Quit request received")
return
}
}
}
func main() {
ctrl := NewMapController()
defer close(ctrl.QuitCh)
go ctrl.Control()
m := make(map[string]string)
m["Joe"] = "Smith"
m["Sean"] = "Howard"
ctrl.WriteCh <- m
r := make(chan map[string]string, 1)
ctrl.ReadCh <- r
fmt.Println(<-r)
}
Runnable version
How can I construct a slice out of all of elements consumed from a channel (like Python's list does)? I can use this helper function:
func ToSlice(c chan int) []int {
s := make([]int, 0)
for i := range c {
s = append(s, i)
}
return s
}
but due to the lack of generics, I'll have to write that for every type, won't I? Is there a builtin function that implements this? If not, how can I avoid copying and pasting the above code for every single type I'm using?
If there's only a few instances in your code where the conversion is needed, then there's absolutely nothing wrong with copying the 7 lines of code a few times (or even inlining it where it's used, which reduces it to 4 lines of code and is probably the most readable solution).
If you've really got conversions between lots and lots of types of channels and slices and want something generic, then you can do this with reflection at the cost of ugliness and lack of static typing at the callsite of ChanToSlice.
Here's complete example code for how you can use reflect to solve this problem with a demonstration of it working for an int channel.
package main
import (
"fmt"
"reflect"
)
// ChanToSlice reads all data from ch (which must be a chan), returning a
// slice of the data. If ch is a 'T chan' then the return value is of type
// []T inside the returned interface.
// A typical call would be sl := ChanToSlice(ch).([]int)
func ChanToSlice(ch interface{}) interface{} {
chv := reflect.ValueOf(ch)
slv := reflect.MakeSlice(reflect.SliceOf(reflect.TypeOf(ch).Elem()), 0, 0)
for {
v, ok := chv.Recv()
if !ok {
return slv.Interface()
}
slv = reflect.Append(slv, v)
}
}
func main() {
ch := make(chan int)
go func() {
for i := 0; i < 10; i++ {
ch <- i
}
close(ch)
}()
sl := ChanToSlice(ch).([]int)
fmt.Println(sl)
}
You could make ToSlice() just work on interface{}'s, but the amount of code you save here will likely cost you in complexity elsewhere.
func ToSlice(c chan interface{}) []interface{} {
s := make([]interface{}, 0)
for i := range c {
s = append(s, i)
}
return s
}
Full example at http://play.golang.org/p/wxx-Yf5ESN
That being said: As #Volker said in the comments from the slice (haha) of code you showed it seems like it'd be saner to either process the results in a streaming fashion or "buffer them up" at the generator and just send the slice down the channel.
I'm trying to write functions that will allow me to marshal/unmarshal simple structs into byte arrays. I've succeeded in writing Marshal, with help from the kind folks at #go-nuts, but I'm running into trouble writing Unmarshal.
// Unmarshal unpacks the binary data and stores it in the packet using
// reflection.
func Unmarshal(b []byte, t reflect.Type) (pkt interface{}, err error) {
buf := bytes.NewBuffer(b)
p := reflect.New(t)
v := reflect.ValueOf(p)
for i := 0; i < t.NumField(); i++ {
f := v.Field(i)
switch f.Kind() {
case reflect.String:
// length of string
var l int16
var e error
e = binary.Read(buf, binary.BigEndian, &l)
if e != nil {
err = e
return
}
// read length-of-string bytes from the buffer
raw := make([]byte, l)
_, e = buf.Read(raw)
if e != nil {
err = e
return
}
// convert the bytes to a string
f.SetString(bytes.NewBuffer(raw).String())
default:
e := binary.Read(buf, binary.BigEndian, f.Addr())
if e != nil {
err = e
return
}
}
}
pkt = p
return
}
The problem with the code above is that the call to f.Addr() near the end is apparently trying to get the address of an unaddressable value.
If there is an alternative solution, I would appreciate that as well. Either way, any help would be much appreciated.
Thanks!
I think you should use
v := p.Elem() // Get the value that 'p' points to
instead of
v := reflect.ValueOf(p)
Working example with lots of assumptions and a trivial data format:
package main
import (
"fmt"
"reflect"
"strconv"
)
// example marshalled format. lets say that marshalled data will have
// four bytes of a formatted floating point number followed by two more
// printable bytes.
type m42 []byte
// example struct we'd like to unmarshal into.
type packet struct {
S string // exported fields required for reflection
F float64
}
// example usage
func main() {
var p packet
if err := Unmarshal(m42("3.14Pi"), &p); err == nil {
fmt.Println(p)
} else {
fmt.Println(err)
}
}
func Unmarshal(data m42, structPtr interface{}) error {
vp := reflect.ValueOf(structPtr)
ve := vp.Elem() // settable struct Value
vt := ve.Type() // type info for struct
nStructFields := ve.NumField()
for i := 0; i < nStructFields; i++ {
fv := ve.Field(i) // settable field Value
sf := vt.Field(i) // StructField type information
// struct field name indicates which m42 field to unmarshal.
switch sf.Name {
case "S":
fv.SetString(string(data[4:6]))
case "F":
s := string(data[0:4])
if n, err := strconv.ParseFloat(s, 64); err == nil {
fv.SetFloat(n)
} else {
return err
}
}
}
return nil
}
Appropriate alternative solutions would depend heavily on the real data you need to support.
I'm going to bet that the reason f.Addr() has the problem because it actually isn't addressable.
the reflect package Type object has a method that will tell you if the type is addressable called CanAddr(). Assuming the field is addressable if it's not a string is not always true. If the struct is not passed in as a pointer to a struct then it's fields won't be addressable. For more details about what is and isn't addressable see: http://weekly.golang.org/pkg/reflect/#Value.CanAddr which outlines the correct rules.
Essentially for your code to work I think you need to ensure you always call it with a pointer to a struct.