I can't find a way to layout structure to a byte array programming by GO.
I'm programming extension for GoBGP service. I want to set community for the route via GoBGP API. More about BGP and communities.
My situation:
I'm trying to send the 32bit number. First, 16bits are interpreted as AS number, other 16bits- communities. API has an interface for uint32 to set all this data. I want to have a simple structure like:
type routeCommunity struct {
AS uint16
Community uint16
}
I'm struggling to pass this structure data to a uint32 variable. I would like to layout it to a byte array and then I could just make uint32 from it.
In C++ I would have a pointer of structure pointing to a char array and continued to work with that structure and in the end, I would send char array.
How to make it happen in Golang??
You can pass your structure to a uint32 variable using "unsafe" package:
func (rc *routeCommunity) bits() uint32 {
return *(*uint32)(unsafe.Pointer(rc))
}
https://play.golang.org/p/AQeeoYUn7o0
UPD:
Keep in mind the bits position depends on an order of the structure fields.
Low two bytes of the variable contain the first field value and high two bytes contain the second one.
Related
While playing with Go code, I found out that map values are not addressable. For example,
package main
import "fmt"
func main(){
var mymap map[int]string = make(map[int]string)
mymap[1] = "One"
var myptr *string = &mymap[1]
fmt.Println(*myptr)
}
Generates error
mapaddressable.go:7: cannot take the address of mymap[1]
Whereas, the code,
package main
import "fmt"
func main(){
var mymap map[int]string = make(map[int]string)
mymap[1] = "One"
mystring := mymap[1]
var myptr *string = &mystring
fmt.Println(*myptr)
}
works perfectly fine.
Why is this so? Why have the Go developers chosen to make certain values not addressable? Is this a drawback or a feature of the language?
Edit:
Being from a C++ background, I am not used to this not addressable trend that seems to be prevalent in Go. For example, the following code works just fine:
#include<iostream>
#include<map>
#include<string>
using namespace std;
int main(){
map<int,string> mymap;
mymap[1] = "one";
string *myptr = &mymap[1];
cout<<*myptr;
}
It would be nice if somebody could point out why the same addressability cannot be achieved (or intentionally wasn't achieved) in Go.
Well I do not know about the internal Go implementation of maps but most likely it is a kind of hash table. So if you take and save the address of one of its entries and afterwards put another bunch of entries into it, your saved address may be invalid. This is due to internal reorganizations of hash tables when the load factor exceeds a certain threshold and the hash table needs to grow.
Therefore I guess it is not allowed to take the address of one of its entries in order to avoid such errors.
Being from a C++ background.
Why are [Go] map values not addressable?
If all other languages were like C++ there would be no point in having other languages.
C++ is a complex, hard-to-read language.
Remember the Vasa! - Bjarne Stroustrup
Go, by design, is a simple, readable language.
dotGo 2015 - Rob Pike - Simplicity is Complicated
A Go map is a hash map. A deterministic hash function is applied to a map key. The hash value is used to determine the primary map bucket for the entry (key-value pair). A bucket stores one or more map entries. A primary bucket may overflow to secondary buckets. Buckets are implemented as an array. As the number of map entries increases by insertion, the hash function adapts to provide more buckets. The map entries are copied incrementally to a new, larger bucket array. If the number of map entries decreases by deletion, space may be reclaimed.
In summary, a Go map is a dynamic, self-organizing data structure. The memory address of an entry (key-value pair) is not fixed. Therefore, map values are not addressable.
GopherCon 2016 Keith Randall - Inside the Map Implementation
In Go, map value addressability is not necessary.
I am investigating migrating of a highly customized and efficient binary format to one of the available binary formats. The data is stored on some low powered mobile among other places, so performance is important requirement.
Advantage of the current format is that all strings are stored in a pool. This means that we don't repeat the same string hundred of times in file, we read it only once during deserialization and all objects are referencing it by its index. It also means that we keep only one copy in memory. So a lot of advantages :)
I was not able to find a way for capnproto or flatbuffers to support this. Or would I need to build layer on top, and in generated object use integer index to strings explicitly?
Thanks you!
FlatBuffers supports string pooling. Simply serialize a string once, then refer to that string multiple times in other objects. The string will only occur in memory once.
Simplest example, schema:
table MyObject { name: string; id: string; }
code (C++):
FlatBufferBuilder fbb;
auto s = fbb.CreateString("MyPooledString");
// Both string fields point to the same data:
auto o = CreateMyObject(fbb, s, s);
fbb.Finish(o);
You can always do this manually like:
struct MyMessage {
stringTable #0 :List(Text);
# Now encode string fields as integer indexes into the string table.
someString #1 :UInt32;
otherString #2 :UInt32;
}
Cap'n Proto could in theory allow multiple pointers to point at the same object, but currently prohibits this for security reasons: it would be too easy to DoS servers that don't expect it by sending messages that are cyclic or contain lots of overlapping references. See the section on amplification attacks in the docs.
Assuming the following
type User struct {
name string
}
users := make(map[int]User)
users[5] = User{"Steve"}
Why isn't it possible to access the struct instance now stored in the map?
users[5].name = "Mark"
Can anyone shed some light into how to access the map-stored struct, or the logic behind why it's not possible?
Notes
I know that you can achieve this by making a copy of the struct, changing the copy, and copying back into the map -- but that's a costly copy operation.
I also know this can be done by storing struct pointers in my map, but I don't want to do that either.
The fundamental problem is that you can't take the address of an item within a map. You might think the compiler would re-arrange users[5].name = "Mark" into this
(&users[5]).name = "Mark"
But that doesn't compile, giving this error
cannot take the address of users[5]
This gives the maps the freedom to re-order things at will to use memory efficiently.
The only way to change something explicitly in a map is to assign value to it, i.e.
t := users[5]
t.name = "Mark"
users[5] = t
So I think you either have to live with the copy above or live with storing pointers in your map. Storing pointers have the disadvantage of using more memory and more memory allocations, which may outweigh the copying way above - only you and your application can tell that.
A third alternative is to use a slice - your original syntax works perfectly if you change users := make(map[int]User) to users := make([]User, 10)
Maps are typically sparsely filled hash tables which are reallocated when they exceed the threshold. Re-allocation would create issues when someone is holding the pointers to the values
If you are keen on not creating the copy of the object, you can store the pointer to the object itself as the value
When we are referring the map, the value returned is returned "returned by value", if i may borrow the terminology used in function parameters, editing the returned structure does not have any impact on the contents of the map
In a Google protocol buffer, I'm going to use a field to store values that will be integers in [0,255]. From http://code.google.com/apis/protocolbuffers/docs/proto.html#scalar, it looks like the uint32 will be the appropriate value type to use. Despite the field being able to hold up to 32-bit integers, those extra bits will not be wasted in my case due to the variable length encoding. (Correct me if I'm wrong up to here.)
My question is: how should I indicate that the reader of a serialized message can assume that the largest value in that field will be 255? Just a comment in the protocol buffer specification? Is there any other way?
In .proto there is no such specification; you must simply document it (and presumably cast it appropriately at the consuming code).
Aside: if you happen to be using the C# protobuf-net implementation, then you can do this by working outside a .proto definition (protobuf-net allows code-first):
[ProtoMember(3)] // <=== field number
public byte SomeValue {get;set;}
This is then obviously constrained to 0-255, but is encoded on the wire as you expect (like a uint32). It also does a checked conversion when deserializing, to sanity-check the values.
In .proto, the above is closest to:
optional uint32 someValue = 3;
Using Xcode/Cocoa and the ExtAudioFile API, I'm trying to store away AudioBufferList* objects away for later use, but I'm running into trouble with re-accessing the data. These objects are coming from repeated ExtAudioFileRead calls.
I realize that I can't just stuff these AudioBufferList* objects into an NSArray, but I was under the impression that an NSPointerArray would work for this purpose. However, when trying to access audioBufferList->mBuffers[0].mData after storing the audio buffer lists in the NSPointerArray, they just come back zeroed out.
I was memcpying the audioBufferLists to new audioBufferList objects since I'm reusing the same audio buffer list for each ExtAudioFileRead call. I'm not sure if that's sufficient, though, and memcpying the void* audioBufferList->mBuffers[0].mData objects isn't helping either.
What's the easiest way to store these AudioBufferList* objects? Am I on the right track?
AudioBufferLists hold their data in audioBufferList.mBuffers[i].mData.
mData is void* and the actual type of the values is determined by the output format you specified.
Example:
If you defined
kAudioFormatFlagsCanonical,
kAudioFormatLinearPCM,
mBitsPerChannel = 32
and
mFramesPerPacket = 1
as your output format, the mData-array contains values of type AudioSampleType (which is Float32 a typedef)
If you have chosen another format the array might contain SInt16 values or something else.
So you must be aware of your output type when you want to copy the contents of mData around.
If you know the format you could simply create a c-array
dataCopy = calloc(dataSize, sizeof(Float32));
and memcpy audioBufferList.mBuffers[i].mData into that.
If you want to use a cocoa NSMutableArray, you would have to wrap the floats into a NSNumber object.