I am running into a problem mutating a flatbuffer from bytes. According to the flatbuffer documentation (https://github.com/google/flatbuffers/blob/master/docs/source/Tutorial.md), you can
mutate fixed size fields, such as an int32. And as you can see below, the generated golang TestMutate has a
MutateServerId() function. My problem is that after I mutate it, the bytes don't seem to have changed.
Here is my flatbuffer table definition:
namespace foo;
table TestMutate {
serverId:int32;
}
Here is a unit test I wrote:
func TestMutateFlatbuffer2(test *testing.T) {
builder := flatbuffers.NewBuilder(1024)
packageWalletStorageServicesRPC.TestMutateStart(builder)
packageWalletStorageServicesRPC.TestMutateAddServerId(builder, 1)
endMessage := packageWalletStorageServicesRPC.TestMutateEnd(builder)
builder.Finish(endMessage)
bytes := builder.FinishedBytes()
testMutate := packageWalletStorageServicesRPC.GetRootAsTestMutate(bytes, 0)
success := testMutate.MutateServerId(2)
if !success {
panic("Server id not mutated.")
} else {
logger.Logf(logger.INFO, "serverId mutated to:%d", testMutate.ServerId())
}
mutatedBytes := testMutate.Table().Bytes
if string(mutatedBytes) == string(bytes) {
panic("Bytes were not mutated.")
}
}
Here is the output from the test.
=== RUN TestMutateFlatbuffer2
2019/08/01 19:33:56.801926 foo_test.go:389 : [ I ]: serverId mutated to:2
--- FAIL: TestMutateFlatbuffer2 (0.00s)
panic: Bytes were not mutated. [recovered]
panic: Bytes were not mutated.
Notice that it appears I've mutated the underlying structure, but when I get the bytes of the flatbuffer,
they're not changed. Question 1: Am I getting the bytes the correct way? Question 2: If I'm getting them in the right way, why
are they not changed since the call to mutate seemed to succeed?
Your test string(mutatedBytes) == string(bytes) fails because.. you are comparing the mutated buffer against itself. bytes refers to a buffer, that before your mutation contains a 1, and after it contains a 2. mutatedBytes points to the same buffer, and thus also contains a 2. The fact that testMutate.ServerId() returns 2 should tell you that the buffer got mutated succesfully, because there is no other way it could return 2 :) You'd have to make a (deep) copy of bytes before the mutation if you wanted this comparison to show that the buffers are different.
There's at least two solutions to this problem. The second solution is better in my case because it results in less copying of bytes.
By creating an intermediate string (and therefore, a copy of the bytes). Generally with flatbuffers you want to avoid copies, but for my use case, I'm ok with it.
Wrap table definitions like this:
table LotsOfData {
notMutated:[ubyte];
}
table TestMutate {
notMutated:LotsOfData;
serverId:int32;
}
Related
I'm trying to reduce the amount of http requests my discord bot is making.
It's reading from an API.
With the fetched data it updates an internal database and outputs the changes.
Thing is: that database is different for every server the bot is in, and that's where I'm using the go routines. But, some servers need to fetch the same data, here is where I want to reduce the http requests. Right now I'm making requests regardless if I've already fetched a character. I want to create some sort of data that could be shared between the go routines and before making a request search within this data.
I was advised to use mutex. I'm trying. Original question: Working with unbuffered channels in golang
I made a skeleton of the real code I've tried: https://play.golang.org/p/mt229ns1R8m
In this example master := make([][]map[string]interface{}, 0) is simulating the discord servers.
Chars and Chars2 would be the tracked chars for each individual server.
The char "Test" is mutual to both of them, so it should be fetched from the API only once.
It's outputing this:
[[map[Level:15 Name:Test] map[Level:150 Name:Test2]] [map[Level:1500 Name:Test3] map[Level:15 Name:Test]]]
------
A call would be made
A call would be made
A call would be made
A call would be made
Cache: [map[Level:150 Name:Test2] map[Level:15 Name:Test]]Cache: [map[Level:15 Name:Test] map[Level:1500 Name:Test3]]Done
I was expecting the output to be:
[[map[Level:15 Name:Test] map[Level:150 Name:Test2]] [map[Level:1500 Name:Test3] map[Level:15 Name:Test]]]
------
A call would be made
A call would be made
A call would be made
Cache: [map[Level:150 Name:Test2] map[Level:15 Name:Test] map[Level:1500 Name:Test3]]Done
But a new cache is being generated by every go routine. How can I fix this?
Thanks.
There are too many unknowns here for me to really write a proper design, but let's make a few notes:
Try not to use interface{} at all, if at all possible. In this case, it seems that it must be possible, though I'm not sure what the actual types will be.
Try to make your data as simple as possible, but no simpler. In this case, that probably means: have one data structure for "thing that talks to a Discord server" and a separate one for "thing that talks to the local database" (is this a caching database? if so, what are the criteria for invalidating a cache entry?). But if one "character" (whatever that is—apparently a string) can have different properties per Discord server, that means that your index into your local database is not just a character, but rather a pair of values: the string value itself plus a Discord-server-identifier.
This might give you a functional interface like this:
var cacheServer *CacheServer
func InitCacheServer() error {
cacheServer = ... // whatever it takes to initialize the cache server
}
(I've assumed lazy initialization of the cache server. If you can do up-front initialization, you can drop the next test below. Replace ValueType with the type of the result of a cached lookup of a name.)
func (DiscordServer ds) Get(name string) (ValueType, error) {
if cacheserver == nil {
if err := InitCacheServer(); err != nil {
return nil, err
}
}
// Do a cache lookup. Tell the cache server that if there
// is no entry, it should return a NoEntry error and we will
// fill the cache ourselves, so it should hold this slot as
// "will be filled, so wait for it".
slot, v, err := cacheServer.Lookup(name, ds.identity, CacheServer.IntentToFill)
if err == CacheServer.NoEntry {
// We have the slot held. Try to look up the right info
// directly in the Discord server, then cache it.
v, err = ds.UncachedGet(name)
// Tell cache server that this is the value, or that it should
// produce this error instead of NoCache.
cacheServer.FillSlot(slot, v, err)
}
}
You might only want to cache some error types, rather than all; that's another one of those design questions that needs an answer that I cannot provide here. There are other ways to do this that don't necessarily need a slot pointer return value, too; I've just chosen this one for this example.
Note that most of the "hard work" is now in the cache server, which definitely requires some fancy footwork. In particular you will want to lock the overall data structure for a little while, use that to find the correct slot, then hold the slot itself so that other users of the slot must wait, while releasing the overall lock so that other users of other entries need not wait. This introduces locking order constraints: be careful to avoid deadlock. One method that should work is:
type CacheServer struct {
lock sync.Mutex
data map[string]map[string]*Entry
// more fields
}
type Entry {
lock sync.Mutex
cachedValue ValueType
cachedError error
}
(You'll need some more types, like Intent—just two enumerated integers for now—below, and probably more fields in the above; this is just a skeleton.)
func (cs *CacheServer) Lookup(name, srv string, flags Intent) (*Entry, ValueType, error) {
cs.lock.Lock()
defer cs.lock.Unlock()
// first, look up the server - if it does not exist, create one
smap := cs.data[srv]
if smap == nil {
cs.data[server] = make(map[string]*Entry)
}
entry := smap[name]
if entry == nil {
// no cached entry - if this is a pure lookup, just error,
// but if not, make a locked entry
if flags == CacheServer.IntentToFill {
// make a new entry and return with it locked
entry = &Entry{}
smap[name] = entry
entry.lock.Lock() // and do not unlock
}
return entry, nil, NoEntry
}
entry.lock.Lock() // wait for someone to fill it, if needed
defer entry.lock.Unlock()
return nil, entry.cachedValue, entry.cachedError
}
You need a routine to fill and release the entry as well, but it's pretty simple. You could, if you choose, make this a method on the Entry type rather than on the CacheServer type, as at least in this particular prototype, there is no need to use the cache server data structures directly. If you start getting fancier with cache invalidation, though, it might be nice to have access to the CacheServer object.
Note: I've designed this so that you can do a cache lookup without an intent-to-fill, if that's useful. If not, there's no reason to bother with the Intent argument.
I am working with go, specifically QT bindings. However, I do not understand the use of leading underscores in the struct below. I am aware of the use of underscores in general but not this specific example.
type CustomLabel struct {
core.QObject
_ func() `constructor:"init"`
_ string `property:"text"`
}
Does it relate to the struct tags?
Those are called blank-fields because the blank identifier is used as the field name.
They cannot be referred to (just like any variable that has the blank identifier as its name) but they take part in the struct's memory layout. Usually and practically they are used as padding, to align subsequent fields to byte-positions (or memory-positions) that match layout of the data coming from (or going to) another system. The gain is that so these struct values (or rather their memory space) can be dumped or read simply and efficiently in one step.
#mkopriva's answer details what the specific use case from the question is for.
A word of warning: these blank fields as "type-annotations" should be used sparingly, as they add unnecessary overhead to all (!) values of such struct. These fields cannot be referred to, but they still require memory. If you add a blank field whose size is 8 bytes (e.g. int64), if you create a million elements, those 8 bytes will count a million times. As such, this is a "flawed" use of blank fields: the intention is to add meta info to the type itself (not to its instances), yet the cost is that all elements will require increased memory.
You might say then to use a type whose size is 0, such as struct{}. It's better, as if used in the right position (e.g. being the first field, for reasoning see Struct has different size if the field order is different and also Why position of `[0]byte` in the struct matters?), they won't change the struct's size. Still, code that use reflection to iterate over the struct's fields will still have to loop over these too, so it makes such code less efficient (typically all marshaling / unmarshaling process). Also, since now we can't use an arbitrary type, we lose the advantage of carrying a type information.
This last statement (about when using struct{} we lose the carried type information) can be circumvented. struct{} is not the only type with 0 size, all arrays with 0 length also have zero size (regardless of the actual element type). So we can retain the type information by using a 0-sized array of the type we'd like to incorporate, such as:
type CustomLabel struct {
_ [0]func() `constructor:"init"`
_ [0]string `property:"text"`
}
Now this CustomLabel type looks much better performance-wise as the type in question: its size is still 0. And it is still possible to access the array's element type using Type.Elem() like in this example:
type CustomLabel struct {
_ [0]func() `constructor:"init"`
_ [0]string `property:"text"`
}
func main() {
f := reflect.ValueOf(CustomLabel{}).Type().Field(0)
fmt.Println(f.Tag)
fmt.Println(f.Type)
fmt.Println(f.Type.Elem())
}
Output (try it on the Go Playground):
constructor:"init"
[0]func()
func()
For an overview of struct tags, read related question: What are the use(s) for tags in Go?
You can think of it as meta info of the type, it's not accessible through an instance of that type but can be accessed using reflect or go/ast. This gives the interested package/program some directives as to what to do with that type. For example based on those tags it could generate code using go:generate.
Considering that one of the tags says constructor:"init" and the field's type is func() it's highly probable that this is used with go:generate to generate an constructor function or initializer method named init for the type CustomLabel.
Here's an example of using reflect to get the "meta" info (although as I've already mentioned, the specific qt example is probably meant to be handled by go:generate).
type CustomLabel struct {
_ func() `constructor:"init"`
_ string `property:"text"`
}
fmt.Println(reflect.ValueOf(CustomLabel{}).Type().Field(0).Tag)
// constructor:"init"
fmt.Println(reflect.ValueOf(CustomLabel{}).Type().Field(0).Type)
// func()
https://play.golang.org/p/47yWG4U0uit
This question already has answers here:
Accessing struct fields inside a map value (without copying)
(2 answers)
Closed 7 years ago.
New to Go. Encountered this error and have had no luck finding the cause or the rationale for it:
If I create a struct, I can obviously assign and re-assign the values no problem:
type Person struct {
name string
age int
}
func main() {
x := Person{"Andy Capp", 98}
x.age = 99
fmt.Printf("age: %d\n", x.age)
}
but if the struct is one value in a map:
type Person struct {
name string
age int
}
type People map[string]Person
func main() {
p := make(People)
p["HM"] = Person{"Hank McNamara", 39}
p["HM"].age = p["HM"].age + 1
fmt.Printf("age: %d\n", p["HM"].age)
}
I get cannot assign to p["HM"].age. That's it, no other info. http://play.golang.org/p/VRlSItd4eP
I found a way around this - creating an incrementAge func on Person, which can be called and the result assigned to the map key, eg p["HM"] = p["HM"].incrementAge().
But, my question is, what is the reason for this "cannot assign" error, and why shouldn't I be allowed to assign the struct value directly?
p["HM"] isn't quite a regular addressable value: hashmaps can grow at runtime, and then their values get moved around in memory, and the old locations become outdated. If values in maps were treated as regular addressable values, those internals of the map implementation would get exposed.
So, instead, p["HM"] is a slightly different thing called a "map index expression" in the spec; if you search the spec for the phrase "index expression" you'll see you can do certain things with them, like read them, assign to them, and use them in increment/decrement expressions (for numeric types). But you can't do everything. They could have chosen to implement more special cases than they did, but I'm guessing they didn't just to keep things simple.
Your approach seems good here--you change it to a regular assignment, one of the specifically-allowed operations. Another approach (maybe good for larger structs you want to avoid copying around?) is to make the map value a regular old pointer that you can modify the underlying object through:
package main
import "fmt"
type Person struct {
name string
age int
}
type People map[string]*Person
func main() {
p := make(People)
p["HM"] = &Person{"Hank McNamara", 39}
p["HM"].age += 1
fmt.Printf("age: %d\n", p["HM"].age)
}
The left side of the assignment must b "addressable".
https://golang.org/ref/spec#Assignments
Each left-hand side operand must be addressable, a map index expression, or (for = assignments only) the blank identifier.
and https://golang.org/ref/spec#Address_operators
The operand must be addressable, that is, either a variable, pointer indirection, or slice indexing operation; or a field selector of an addressable struct operand; or an array indexing operation of an addressable array.
as #twotwotwo's comment, p["HM"] is not addressable.
but, there is no such definition show what is "addressable struct operand" in the spec. I think they should add some description for it.
I want to use Go to make an API for an existing database that uses null values extensively. Go will not scan nulls to empty strings (or equivalent), and so I need to implement a workaround.
The workarounds I have discovered have left me unsatisfied. In fact I went looking for a dynamic language because of this problem, but Go has certain attractions and I would like to stick with it if possible. Here are the workarounds that did not satisfy:
Don't use nulls in the database. Unsuitable because the database is pre-existing and I do not have liberty to interfere with its structure. The database is more important than my app, not the other way around.
In sql queries, use COALESCE, ISNULL, etc to convert nulls to empty strings (or equiv) before the data gets to my app. Unsuitable because there are many fields and many tables. Apart from a couple of obvious ones (primary key, surname), I don't know for sure which fields can be relied upon not to give me a null value, so I would be defensively cluttering my sql queries everywhere.
Use sql.NullString, sql.NullInt64, sql.NullFloat64, etc to convert nulls to empty strings (or equiv) as an intermediate step before settling them into their destination type. This suffers from the same problem as above, only I am cluttering my Go code instead of my sql queries.
Use a combination of *pointers and []byte, to scan each item in to a memory location without committing it to a particular type (other than []byte), and then somehow work with the raw data. But to do something meaningful with the data you have to convert it to something more useful, and then you are back to sql.Nullstring or if x==nil{handle it}, and this again is happening on a case by case basis for any field that I need to work with. So, again, we are looking at cluttered, messy, error-prone code and I'm repeating myself all the time instead of being DRY in my coding.
Look to the Go ORM libraries for help. Well I did that, but to my surprise none of them tackle this issue.
Make my own helper package to convert all null strings to "", null ints to 0, null floats to 0.00, null bools to false, etc, and make it part of the process of scanning in from the sql driver, resulting in regular, normal strings, ints, floats and bools.
Unfortunately if 6 is the solution, I do not have the expertise. I suspect the solution would involve something like "if the intended type of the item to be scanned to is a string, make it an sql.NullString and extract an empty string from it. But if the item to be scanned to is an int, make it a NullInt64 and get a zero from that. But if ...(etc)"
Is there anything I have missed? Thank you.
The use of pointers for the sql-scanning destination variables enables the data to be scanned in, worked with (subject to checking if != nil) and marshalled to json, to be sent out from the API, without having to put hundreds of sql.Nullstring, sql.Nullfloat64 etc everywhere. Nulls are miraculously preserved and sent out through the marshalled json. (See Fathername at the bottom). At the other end, the client can work with the nulls in javascript which is better equipped to handle them.
func queryToJson(db *sql.DB) []byte {
rows, err := db.Query(
"select mothername, fathername, surname from fams" +
"where surname = ?", "Nullfather"
)
defer rows.Close()
type record struct {
Mname, Fname, Surname *string // the key: use pointers
}
records := []record{}
for rows.Next() {
var r record
err := rows.Scan(r.Mname, r.Fname, r.Surname) // no need for "&"
if err != nil {
log.Fatal(err)
}
fmt.Println(r)
records = append(records, r)
}
j, err := json.Marshal(records)
if err != nil {
log.Fatal(err)
}
return j
}
j := queryToJson(db)
fmt.Println(string(j)) // [{"Mothername":"Mary", "Fathername":null, "Surname":"Nullfather"}]
http://golang.org/pkg/sort/
This is from Go example.
// OrderedBy returns a Sorter that sorts using the less functions, in order.
// Call its Sort method to sort the data.
func OrderedBy(less ...lessFunc) *multiSorter {
return &multiSorter{
changes: changes,
less: less,
}
}
What does this do by colon? Is it mapping? Is it closure? Too much new syntax here. What should I read to understand this syntax in Go?
It's a factory function, creating and initialising a struct of type multisorter:
https://sites.google.com/site/gopatterns/object-oriented/constructors
Additionally, Go "constructors" can be written succinctly using initializers within a factory function:
function NewMatrix(rows, cols, int) *matrix {
return &matrix{rows, cols, make([]float, rows*cols)}
}
Also, it is using named parameters when initialising:
http://www.golang-book.com/9
This allocates memory for all the fields, sets each of them to their zero value and returns a pointer. (Circle) More often we want to give each of the fields a value. We can do this in two ways. Like this:
c := Circle{x: 0, y: 0, r: 5}
The `less ...lessFunc` in the func declaration means:
any number of parameters, each of type `lessFunc` can be passed here, and will be stored in the slice `less`
So it creates a `multiSorter` struct, which supports the sort interface, and calling the sort method from that interface (and implemented by multiSorter) will cause the object to use each lessFunc in turn while sorting
Does this make sense? I can expand more if needed...