I know from reading around that Maps are intentionally unordered in Go, but they offer a lot of benefits that I would like to use for this problem I'm working on. My question is how might I order a map FIFO style? Is it even worth trying to make this happen? Specifically I am looking to make it so that I can unmarshal into a set of structures hopefully off of an interface.
I have:
type Package struct {
Account string
Jobs []*Jobs
Libraries map[string]string
}
type Jobs struct {
// Name of the job
JobName string `mapstructure:"name" json:"name" yaml:"name" toml:"name"`
// Type of the job. should be one of the strings outlined in the job struct (below)
Job *Job `mapstructure:"job" json:"job" yaml:"job" toml:"job"`
// Not marshalled
JobResult string
// For multiple values
JobVars []*Variable
}
type Job struct {
// Sets/Resets the primary account to use
Account *Account `mapstructure:"account" json:"account" yaml:"account" toml:"account"`
// Set an arbitrary value
Set *Set `mapstructure:"set" json:"set" yaml:"set" toml:"set"`
// Contract compile and send to the chain functions
Deploy *Deploy `mapstructure:"deploy" json:"deploy" yaml:"deploy" toml:"deploy"`
// Send tokens from one account to another
Send *Send `mapstructure:"send" json:"send" yaml:"send" toml:"send"`
// Utilize eris:db's native name registry to register a name
RegisterName *RegisterName `mapstructure:"register" json:"register" yaml:"register" toml:"register"`
// Sends a transaction which will update the permissions of an account. Must be sent from an account which
// has root permissions on the blockchain (as set by either the genesis.json or in a subsequence transaction)
Permission *Permission `mapstructure:"permission" json:"permission" yaml:"permission" toml:"permission"`
// Sends a bond transaction
Bond *Bond `mapstructure:"bond" json:"bond" yaml:"bond" toml:"bond"`
// Sends an unbond transaction
Unbond *Unbond `mapstructure:"unbond" json:"unbond" yaml:"unbond" toml:"unbond"`
// Sends a rebond transaction
Rebond *Rebond `mapstructure:"rebond" json:"rebond" yaml:"rebond" toml:"rebond"`
// Sends a transaction to a contract. Will utilize eris-abi under the hood to perform all of the heavy lifting
Call *Call `mapstructure:"call" json:"call" yaml:"call" toml:"call"`
// Wrapper for mintdump dump. WIP
DumpState *DumpState `mapstructure:"dump-state" json:"dump-state" yaml:"dump-state" toml:"dump-state"`
// Wrapper for mintdum restore. WIP
RestoreState *RestoreState `mapstructure:"restore-state" json:"restore-state" yaml:"restore-state" toml:"restore-state"`
// Sends a "simulated call" to a contract. Predominantly used for accessor functions ("Getters" within contracts)
QueryContract *QueryContract `mapstructure:"query-contract" json:"query-contract" yaml:"query-contract" toml:"query-contract"`
// Queries information from an account.
QueryAccount *QueryAccount `mapstructure:"query-account" json:"query-account" yaml:"query-account" toml:"query-account"`
// Queries information about a name registered with eris:db's native name registry
QueryName *QueryName `mapstructure:"query-name" json:"query-name" yaml:"query-name" toml:"query-name"`
// Queries information about the validator set
QueryVals *QueryVals `mapstructure:"query-vals" json:"query-vals" yaml:"query-vals" toml:"query-vals"`
// Makes and assertion (useful for testing purposes)
Assert *Assert `mapstructure:"assert" json:"assert" yaml:"assert" toml:"assert"`
}
What I would like to do is to have jobs contain a map of string to Job and eliminate the job field, while maintaining order in which they were placed in from the config file. (Currently using viper). Any and all suggestions for how to achieve this are welcome.
You would need to hold the keys in a separate slice and work with that.
type fifoJob struct {
m map[string]*Job
order []string
result []string
// Not sure where JobVars will go.
}
func (str *fifoJob) Enqueue(key string, val *Job) {
str.m[key] = val
str.order = append(str.order, key)
}
func (str *fifoJob) Dequeue() {
if len(str.order) > 0 {
delete(str.m, str.order[0])
str.order = str.order[1:]
}
}
Anyways if you're using viper you can use something like the fifoJob struct defined above. Also note that I'm making a few assumptions here.
type Package struct {
Account string
Jobs *fifoJob
Libraries map[string]string
}
var config Package
config.Jobs = fifoJob{}
config.Jobs.m = map[string]*Job{}
// Your config file would need to store the order in an array.
// Would've been easy if viper had a getSlice method returning []interface{}
config.Jobs.order = viper.GetStringSlice("package.jobs.order")
for k,v := range viper.GetStringMap("package.jobs.jobmap") {
if job, ok := v.(Job); ok {
config.Jobs.m[k] = &job
}
}
for
PS: You're giving too many irrelevant details in your question. I was asking for a MCVE.
Maps are by nature unordered but you can fill up a slice instead with your keys. Then you can range over your slice and sort it however you like. You can pull out specific elements in your slice with [i].
Check out pages 170, 203, or 204 of some great examples of this:
Programming in Go
Related
I am using zerolog for logging in a go application. I'd like to log a map (json) and find a way:
log.Info().
Str("foo", "bar").
Dict("dict", zerolog.Dict().
Str("bar", "baz").
Int("n", 1),
).Msg("hello world")
// Output: {"level":"info","time":1494567715,"foo":"bar","dict":{"bar":"baz","n":1},"message":"hello world"}
in above example, I need to specify each key-value pair in zerolog.Dict() method. I wonder whether there is an automatic way to log a struct.
For example, I have a struct like:
type Message struct {
AWS_REGION string `json:"region"`
LOG_LEVEL string `json:"level"`
STAGE string `json:"stage"`
REQUEST_ID string `json:"requestId"`
}
I am looking for a way to pass Message instance to
msg := Message{ ... }
zerolog.Dict("message", msg)
You can achieve this by using the Interface method.
Example
However, note that Interface uses reflection for serialisation, so code shortness comes at a cost of performance
I'm trying to reduce the amount of http requests my discord bot is making.
It's reading from an API.
With the fetched data it updates an internal database and outputs the changes.
Thing is: that database is different for every server the bot is in, and that's where I'm using the go routines. But, some servers need to fetch the same data, here is where I want to reduce the http requests. Right now I'm making requests regardless if I've already fetched a character. I want to create some sort of data that could be shared between the go routines and before making a request search within this data.
I was advised to use mutex. I'm trying. Original question: Working with unbuffered channels in golang
I made a skeleton of the real code I've tried: https://play.golang.org/p/mt229ns1R8m
In this example master := make([][]map[string]interface{}, 0) is simulating the discord servers.
Chars and Chars2 would be the tracked chars for each individual server.
The char "Test" is mutual to both of them, so it should be fetched from the API only once.
It's outputing this:
[[map[Level:15 Name:Test] map[Level:150 Name:Test2]] [map[Level:1500 Name:Test3] map[Level:15 Name:Test]]]
------
A call would be made
A call would be made
A call would be made
A call would be made
Cache: [map[Level:150 Name:Test2] map[Level:15 Name:Test]]Cache: [map[Level:15 Name:Test] map[Level:1500 Name:Test3]]Done
I was expecting the output to be:
[[map[Level:15 Name:Test] map[Level:150 Name:Test2]] [map[Level:1500 Name:Test3] map[Level:15 Name:Test]]]
------
A call would be made
A call would be made
A call would be made
Cache: [map[Level:150 Name:Test2] map[Level:15 Name:Test] map[Level:1500 Name:Test3]]Done
But a new cache is being generated by every go routine. How can I fix this?
Thanks.
There are too many unknowns here for me to really write a proper design, but let's make a few notes:
Try not to use interface{} at all, if at all possible. In this case, it seems that it must be possible, though I'm not sure what the actual types will be.
Try to make your data as simple as possible, but no simpler. In this case, that probably means: have one data structure for "thing that talks to a Discord server" and a separate one for "thing that talks to the local database" (is this a caching database? if so, what are the criteria for invalidating a cache entry?). But if one "character" (whatever that is—apparently a string) can have different properties per Discord server, that means that your index into your local database is not just a character, but rather a pair of values: the string value itself plus a Discord-server-identifier.
This might give you a functional interface like this:
var cacheServer *CacheServer
func InitCacheServer() error {
cacheServer = ... // whatever it takes to initialize the cache server
}
(I've assumed lazy initialization of the cache server. If you can do up-front initialization, you can drop the next test below. Replace ValueType with the type of the result of a cached lookup of a name.)
func (DiscordServer ds) Get(name string) (ValueType, error) {
if cacheserver == nil {
if err := InitCacheServer(); err != nil {
return nil, err
}
}
// Do a cache lookup. Tell the cache server that if there
// is no entry, it should return a NoEntry error and we will
// fill the cache ourselves, so it should hold this slot as
// "will be filled, so wait for it".
slot, v, err := cacheServer.Lookup(name, ds.identity, CacheServer.IntentToFill)
if err == CacheServer.NoEntry {
// We have the slot held. Try to look up the right info
// directly in the Discord server, then cache it.
v, err = ds.UncachedGet(name)
// Tell cache server that this is the value, or that it should
// produce this error instead of NoCache.
cacheServer.FillSlot(slot, v, err)
}
}
You might only want to cache some error types, rather than all; that's another one of those design questions that needs an answer that I cannot provide here. There are other ways to do this that don't necessarily need a slot pointer return value, too; I've just chosen this one for this example.
Note that most of the "hard work" is now in the cache server, which definitely requires some fancy footwork. In particular you will want to lock the overall data structure for a little while, use that to find the correct slot, then hold the slot itself so that other users of the slot must wait, while releasing the overall lock so that other users of other entries need not wait. This introduces locking order constraints: be careful to avoid deadlock. One method that should work is:
type CacheServer struct {
lock sync.Mutex
data map[string]map[string]*Entry
// more fields
}
type Entry {
lock sync.Mutex
cachedValue ValueType
cachedError error
}
(You'll need some more types, like Intent—just two enumerated integers for now—below, and probably more fields in the above; this is just a skeleton.)
func (cs *CacheServer) Lookup(name, srv string, flags Intent) (*Entry, ValueType, error) {
cs.lock.Lock()
defer cs.lock.Unlock()
// first, look up the server - if it does not exist, create one
smap := cs.data[srv]
if smap == nil {
cs.data[server] = make(map[string]*Entry)
}
entry := smap[name]
if entry == nil {
// no cached entry - if this is a pure lookup, just error,
// but if not, make a locked entry
if flags == CacheServer.IntentToFill {
// make a new entry and return with it locked
entry = &Entry{}
smap[name] = entry
entry.lock.Lock() // and do not unlock
}
return entry, nil, NoEntry
}
entry.lock.Lock() // wait for someone to fill it, if needed
defer entry.lock.Unlock()
return nil, entry.cachedValue, entry.cachedError
}
You need a routine to fill and release the entry as well, but it's pretty simple. You could, if you choose, make this a method on the Entry type rather than on the CacheServer type, as at least in this particular prototype, there is no need to use the cache server data structures directly. If you start getting fancier with cache invalidation, though, it might be nice to have access to the CacheServer object.
Note: I've designed this so that you can do a cache lookup without an intent-to-fill, if that's useful. If not, there's no reason to bother with the Intent argument.
Consider this example. Let's say I have this object which is ubiquitous throughout my codebase:
type Person struct {
Name string
Age int
[some other fields]
}
Somewhere deep in the codebase, I also have some code that creates a new Person struct. Maybe it's something like the following utility function (note that this is just an example of some function that creates a Person-- the point of my question is not to ask about the copy function specifically):
func copyPerson(origPerson Person) *Person {
copy := Person{
Name: origPerson.Name,
Age: origPerson.Age,
[some other fields]
}
return ©
}
Another developer comes along and adds a new field Gender to the Person struct. However, because the copyPerson function is in a distant piece of code they forget to update copyPerson. Since golang doesn't throw any warning or error if you omit a parameter when creating a struct, the code will compile and appear to work fine; the only difference is that the copyPerson method will now fail to copy over the Gender struct, and the result of copyPerson will have Gender replaced with a nil value (e.g. the empty string).
What is the best way to prevent this from happening? Is there a way to ask golang to enforce no missing parameters in a specific struct initialization? Is there a linter that can detect this type of potential error?
The way I would solve this is to just use NewPerson(params) and not export the person. This makes it so the only way to get a person instance is to go through your New method.
package person
// Struct is not exported
type person struct {
Name string
Age int
Gender bool
}
// We are forced to call the constructor to get an instance of person
func New(name string, age int, gender bool) person {
return person{name, age, gender}
}
This forces everyone to get an instance from the same place. When you add a field, you can add it to the function definition and then you get compile time errors anywhere they are constructing a new instance, so you can easily find them and fix them.
First of all, your copyPerson() function does not live up to its name. It copies some fields of a Person, but not (necessarily) all. It should've been named copySomeFieldsOfPerson().
To copy a complete struct value, just assign the struct value. If you have a function receiving a non-pointer Person, that is already a copy, so just return its address:
func copyPerson(p Person) *Person {
return &p
}
That's all, this will copy all present and future fields of Person.
Now there may be cases where fields are pointers or header-like values (like a slice) which should be "detached" from the original field (more precisely from the pointed object), in which case you do need to make manual adjustments, e.g.
type Person struct {
Name string
Age int
Data []byte
}
func copyPerson(p Person) *Person {
p2 := p
p2.Data = append(p2.Data, p.Data...)
return &p2
}
Or an alternative solution which does not make another copy of p but still detaches Person.Data:
func copyPerson(p Person) *Person {
var data []byte
p.Data = append(data, p.Data...)
return &p
}
Of course, if someone adds a field which also needs manual handling, this won't help you out.
You could also use unkeyed literal, like this:
func copyPerson(p Person) *Person {
return &Person{
p.Name,
p.Age,
}
}
This will result in a compile-time error if someone adds a new field to Person, because an unkeyed composite struct literal must list all fields. Again, this will not help you out if someone changes the fields where the new fields are assignable to the old ones (e.g. someone swaps 2 fields next to each other having the same type), also unkeyed literals are discouraged.
Best would be for the package owner to provide a copy constructor, next to the Person type definition. So if someone changes Person, he / she should be responsible keeping CopyPerson() still operational. And as others mentioned, you should already have unit tests which should fail if CopyPerson() does not live up to its name.
The best viable option?
If you can't place the CopyPerson() next to the Person type and have its author maintain it, go ahead with the struct value copying and manual handling of pointer and header-like fields.
And you can create a person2 type which is a "snapshot" of the Person type. Use a blank global variable to receive compile-time alert if the original Person type changes, in which case copyPerson()'s containing source file will refuse to compile, so you'll know it needs adjusting.
This is how it can be done:
type person2 struct {
Name string
Age int
}
var _ = Person(person2{})
The blank variable declaration will not compile if fields of Person and person2 do not match.
A variation of the above compile-time check could be to use typed-nil pointers:
var _ = (*Person)((*person2)(nil))
I'm not aware of a language rule that enforces that.
But you can write custom checkers for Go vet if you'd like. Here's a recent post talking about that.
That said, I would reconsider the design here. If the Person struct is so important in your code base, centralize its creation and copying so that "distant places" don't just create and move Persons around. Refactor your code so that only a single constructor is used to build Persons (maybe something like person.New returning a person.Person), and then you'll be able to centrally control how its fields are initialized.
The idiomatic way would be to not do this at all, and instead make the zero value useful. The example of a copy function doesn't really make sense because it's totally unnecessary - you could just say:
copy := new(Person)
*copy = *origPerson
and not need a dedicated function nor have to keep a listing of fields up to date. If you want a constructor for new instances like NewPerson, just write one and use it as a matter of course. Linters are great for some things but nothing beats well-understood best practices and peer code review.
The best solution I have been able to come up with (and it's not very good) is to define a new struct tempPerson identical to the Person struct and put it nearby to any code which initializes a new Person struct, and to change the code that initializes a Person so that it instead initializes it as a tempPerson but then casts it to a Person. Like this:
type tempPerson struct {
Name string
Age int
[some other fields]
}
func copyPerson(origPerson Person) *Person {
tempCopy := tempPerson{
Name: orig.Name,
Age: orig.Age,
[some other fields]
}
copy := (Person)(tempCopy)
return ©
}
This way if another field Gender is added to Person but not to tempPerson the code will fail at compile-time. Presumably the developer would then see the error, edit tempPerson to match their change to Person, and in doing so notice the nearby code which uses tempPerson and recognize that they should edit that code to also handle the Gender field as well.
I don't love this solution because it involves copying and pasting the struct definition everywhere that we initialize a Person struct and would like to have this safety. Is there any better way?
Approach 1 Add something like copy constructor:
type Person struct {
Name string
Age int
}
func CopyPerson(name string, age int)(*Person, error){
// check params passed if needed
return &Person{Name: name, Age: age}, nil
}
p := CopyPerson(p1.Name, p1.age) // force all fields to be passed
Approach 2: (not sure if this is possible)
Can this be covered in tests say using reflection?
If we compare the number of fields initialised(initialise all the field with values different than the default values) in the original struct and the fields in copy returned by the copy function.
Here is how i would do it:
func copyPerson(origPerson Person) *Person {
newPerson := origPerson
//proof that 'newPerson' points to a new person object
newPerson.name = "new name"
return &newPerson
}
Go Playground
I have two structs, whose types are as follows:
type UserStruct struct {
UserID string `bson:"user_id" json:"user_id"`
Address string `bson:"address" json:"address"`
Email string `bson:"email" json:"email"`
CreatedAt time.Time `bson:"created_at" json:"created_at"`
PhoneNumber string `bson:"phone_number" json:"phone_number"`
PanCard string `bson:"pancard" json:"pancard"`
Details map[string]string `json:"details"`
}
type SecretsStruct struct {
UserID string `r:"user_id" json:"user_id"`
Secrets []string `r:"secrets" json:secrets`
Address string `r:"address" json:"address"`
Email string `r:"email"json:"email"`
CreatedAt time.Time `r:"created_at"json:"created_at"`
PhoneNumber string `r:"phone_number" json:"phone_number"`
PanCard string `r:"pancard" json:"pancard"`
}
I already have an instance of UserStruct. I want to copy the fields common to both structs from UserStruct to a new instance of SecretStruct, without using reflection.
Go is a statically typed language (and is not Python). If you want to copy fields between the structs, you must either cause code to be supplied at compile time which knows how to do this, or use the reflect library to perform the operation at runtime.
Note that I said "cause code to be supplied at compile time" because you don't have to explicitly write that code. You could use code generation to produce the copy code from the struct definitions, or from a higher-level definition (e.g. XML) which generates both the struct definition and the copying code.
However, good Go programmers prefer clear code over clever solutions. If this is a single localized requirement, writing a code generator to avoid "boilerplate" code is almost certainly overkill; its implementation will take longer than the code to copy the structs, and the associated complexity will introduce a risk of more bugs. Similarly, reflect-based solutions are complicated, not clear, and only recommended in cases where you require a generic or extensible solution, and where this cannot be fulfilled at compile time.
I recommend simply write the copying code, and add appropriate comments to the struct definitions and copy methods to ensure future maintainers are aware of their obligation to maintain the copy methods.
Example
// Define your types - bodies elided for brevity
// NOTE TO MAINTAINERS: if editing the fields in these structs, ensure
// the methods defined in source file <filename>.go are updated to
// ensure common fields are copied between structs on instantiation.
type UserStruct struct { ... }
type SecretStruct struct { ... }
// NewSecretStructFromUserStruct populates and returns a SecretStruct
// from the elements common to the two types. This method must be
// updated if the set of fields common to both structs is changed in
// future.
func NewSecretStructFromUserStruct(us *UserStruct) *SecretStruct {
// You should take care to deep copy where necessary,
// e.g. for any maps shared between the structs (not
// currently the case).
ss := new(SecretStruct)
ss.UserID = us.UserID
ss.Address = us.Address
ss.Email = us.Email
ss.CreatedAt = us.CreatedAt
ss.PhoneNumber = us.PhoneNumber
ss.PanCard = us.PanCard
return ss
}
// You may also consider this function to be better suited as
// a receiver method on UserStruct.
I have a situation where I need to interact with a bunch of API's that all have similar purposes but different data structures returned. I need to operate the same way on data from all the API's but I then also need to maybe send that data back to the API at a later time so it can perform further operations on it.
type API1Response struct {
Time time.Time
Price int64
Status int
ID string
}
type API2Response struct {
Price float64
Status string
ID int64
}
// and more API responsed that have irreconcilable differences in their structures
both APIs have similar methods, like Buy and Sell and CancelOrder so I thought I
could use interfaces to make my calls to the methods more usable but then I will have to
make the methods return interface{} instead of returning their specific API response...
now imagine the situation where I want to Buy and I get API1Response. So I call the
method defined in my interface but then I want to CancelOrder which needs me to send
API1Response back to the API. How can I convert the interface back to API1Response properly?
the only way I could think to do it would be to make a long switch case that tries to convert to
every possible data type but IDK if that is idiomatic or error prone in the long run...
if t, ok := emptyInterface.(API1Response); ok {
//do stuff
} else if t, ok := emptyInterface.(API2Response); ok {
// do stuff
}
How would you go about solving this problem idiomatically in go?