I am using zerolog for logging in a go application. I'd like to log a map (json) and find a way:
log.Info().
Str("foo", "bar").
Dict("dict", zerolog.Dict().
Str("bar", "baz").
Int("n", 1),
).Msg("hello world")
// Output: {"level":"info","time":1494567715,"foo":"bar","dict":{"bar":"baz","n":1},"message":"hello world"}
in above example, I need to specify each key-value pair in zerolog.Dict() method. I wonder whether there is an automatic way to log a struct.
For example, I have a struct like:
type Message struct {
AWS_REGION string `json:"region"`
LOG_LEVEL string `json:"level"`
STAGE string `json:"stage"`
REQUEST_ID string `json:"requestId"`
}
I am looking for a way to pass Message instance to
msg := Message{ ... }
zerolog.Dict("message", msg)
You can achieve this by using the Interface method.
Example
However, note that Interface uses reflection for serialisation, so code shortness comes at a cost of performance
Related
I need to print all form requests to log, for later debugging. I need to have something like this, where formInput are different kind of structs. (All defined in my code.)
type SomeFormInput {
Name string
Age int
}
func handleForm(formInput interface{}) {
...
logger.Logf("%+v", formInput);
}
However, some of the forms contain "secret" data that needs to be sanitized before logging - specifically, passwords.
What is the best and easiest way to do this, without losing the generality of handleForm?
I still need it to accept interface{}.
I can use tags if needed.
type SomeFormInput struct {
Name string
Password string `hide:"true"` // something like this
}
There is no completely general solution to this. There are only specific solutions with different trade-offs.
Perhaps the most general solution would be to write a function that inspects each form type with reflection, and omits any sensitive fields in the output.
logger.Logf("%s", stripPassword(formInput));
This has the side effect of no longer using the %+v verb, which often won't matter, but if your type implements the fmt.Formatter interface, it will change behavior. But then again, your goal is to change behavior, so...
Which leads to a second option:
You could implement a custom Format() method on your type, to cause %+v (or %v or %s) to output whatever you desire (omitting private fields).
Along the same lines, but easier, if you're logging (or willing to log) JSON output, just impliment your own json.Marshaler for each type. Example:
func (i *SomeFormInput) MarshalJSON() ([]byte, error) {
intermediate := struct{
*SomeFormInput
Password struct{} `json:"-"` // Occlude Password field in the embedded struct
MarshalJSON struct{} `json:"-"` // Occlude MarshalJSON method on embedded struct, to avoid infinite loop
}{
SomeFormInput: i,
}
return json.Marshal(intermediate)
}
A similar option is to use the json:"-" tag for fields that should not be output. Example:
type SomeFormInput struct {
Name string
Password string `json:"-"`
}
And as a final option, you could filter the log output, prior to sending it to the logger. If you're using JSON, you could parse the data using the tokenizing json.Decoder, to omit only the keys you want. For arbitrary text, perhaps a regular expression could be used.
For most applications I work on, where I practically always use JSON logging, I would probably prefer option #4, followed by #5.
Consider this example. Let's say I have this object which is ubiquitous throughout my codebase:
type Person struct {
Name string
Age int
[some other fields]
}
Somewhere deep in the codebase, I also have some code that creates a new Person struct. Maybe it's something like the following utility function (note that this is just an example of some function that creates a Person-- the point of my question is not to ask about the copy function specifically):
func copyPerson(origPerson Person) *Person {
copy := Person{
Name: origPerson.Name,
Age: origPerson.Age,
[some other fields]
}
return ©
}
Another developer comes along and adds a new field Gender to the Person struct. However, because the copyPerson function is in a distant piece of code they forget to update copyPerson. Since golang doesn't throw any warning or error if you omit a parameter when creating a struct, the code will compile and appear to work fine; the only difference is that the copyPerson method will now fail to copy over the Gender struct, and the result of copyPerson will have Gender replaced with a nil value (e.g. the empty string).
What is the best way to prevent this from happening? Is there a way to ask golang to enforce no missing parameters in a specific struct initialization? Is there a linter that can detect this type of potential error?
The way I would solve this is to just use NewPerson(params) and not export the person. This makes it so the only way to get a person instance is to go through your New method.
package person
// Struct is not exported
type person struct {
Name string
Age int
Gender bool
}
// We are forced to call the constructor to get an instance of person
func New(name string, age int, gender bool) person {
return person{name, age, gender}
}
This forces everyone to get an instance from the same place. When you add a field, you can add it to the function definition and then you get compile time errors anywhere they are constructing a new instance, so you can easily find them and fix them.
First of all, your copyPerson() function does not live up to its name. It copies some fields of a Person, but not (necessarily) all. It should've been named copySomeFieldsOfPerson().
To copy a complete struct value, just assign the struct value. If you have a function receiving a non-pointer Person, that is already a copy, so just return its address:
func copyPerson(p Person) *Person {
return &p
}
That's all, this will copy all present and future fields of Person.
Now there may be cases where fields are pointers or header-like values (like a slice) which should be "detached" from the original field (more precisely from the pointed object), in which case you do need to make manual adjustments, e.g.
type Person struct {
Name string
Age int
Data []byte
}
func copyPerson(p Person) *Person {
p2 := p
p2.Data = append(p2.Data, p.Data...)
return &p2
}
Or an alternative solution which does not make another copy of p but still detaches Person.Data:
func copyPerson(p Person) *Person {
var data []byte
p.Data = append(data, p.Data...)
return &p
}
Of course, if someone adds a field which also needs manual handling, this won't help you out.
You could also use unkeyed literal, like this:
func copyPerson(p Person) *Person {
return &Person{
p.Name,
p.Age,
}
}
This will result in a compile-time error if someone adds a new field to Person, because an unkeyed composite struct literal must list all fields. Again, this will not help you out if someone changes the fields where the new fields are assignable to the old ones (e.g. someone swaps 2 fields next to each other having the same type), also unkeyed literals are discouraged.
Best would be for the package owner to provide a copy constructor, next to the Person type definition. So if someone changes Person, he / she should be responsible keeping CopyPerson() still operational. And as others mentioned, you should already have unit tests which should fail if CopyPerson() does not live up to its name.
The best viable option?
If you can't place the CopyPerson() next to the Person type and have its author maintain it, go ahead with the struct value copying and manual handling of pointer and header-like fields.
And you can create a person2 type which is a "snapshot" of the Person type. Use a blank global variable to receive compile-time alert if the original Person type changes, in which case copyPerson()'s containing source file will refuse to compile, so you'll know it needs adjusting.
This is how it can be done:
type person2 struct {
Name string
Age int
}
var _ = Person(person2{})
The blank variable declaration will not compile if fields of Person and person2 do not match.
A variation of the above compile-time check could be to use typed-nil pointers:
var _ = (*Person)((*person2)(nil))
I'm not aware of a language rule that enforces that.
But you can write custom checkers for Go vet if you'd like. Here's a recent post talking about that.
That said, I would reconsider the design here. If the Person struct is so important in your code base, centralize its creation and copying so that "distant places" don't just create and move Persons around. Refactor your code so that only a single constructor is used to build Persons (maybe something like person.New returning a person.Person), and then you'll be able to centrally control how its fields are initialized.
The idiomatic way would be to not do this at all, and instead make the zero value useful. The example of a copy function doesn't really make sense because it's totally unnecessary - you could just say:
copy := new(Person)
*copy = *origPerson
and not need a dedicated function nor have to keep a listing of fields up to date. If you want a constructor for new instances like NewPerson, just write one and use it as a matter of course. Linters are great for some things but nothing beats well-understood best practices and peer code review.
The best solution I have been able to come up with (and it's not very good) is to define a new struct tempPerson identical to the Person struct and put it nearby to any code which initializes a new Person struct, and to change the code that initializes a Person so that it instead initializes it as a tempPerson but then casts it to a Person. Like this:
type tempPerson struct {
Name string
Age int
[some other fields]
}
func copyPerson(origPerson Person) *Person {
tempCopy := tempPerson{
Name: orig.Name,
Age: orig.Age,
[some other fields]
}
copy := (Person)(tempCopy)
return ©
}
This way if another field Gender is added to Person but not to tempPerson the code will fail at compile-time. Presumably the developer would then see the error, edit tempPerson to match their change to Person, and in doing so notice the nearby code which uses tempPerson and recognize that they should edit that code to also handle the Gender field as well.
I don't love this solution because it involves copying and pasting the struct definition everywhere that we initialize a Person struct and would like to have this safety. Is there any better way?
Approach 1 Add something like copy constructor:
type Person struct {
Name string
Age int
}
func CopyPerson(name string, age int)(*Person, error){
// check params passed if needed
return &Person{Name: name, Age: age}, nil
}
p := CopyPerson(p1.Name, p1.age) // force all fields to be passed
Approach 2: (not sure if this is possible)
Can this be covered in tests say using reflection?
If we compare the number of fields initialised(initialise all the field with values different than the default values) in the original struct and the fields in copy returned by the copy function.
Here is how i would do it:
func copyPerson(origPerson Person) *Person {
newPerson := origPerson
//proof that 'newPerson' points to a new person object
newPerson.name = "new name"
return &newPerson
}
Go Playground
I know from reading around that Maps are intentionally unordered in Go, but they offer a lot of benefits that I would like to use for this problem I'm working on. My question is how might I order a map FIFO style? Is it even worth trying to make this happen? Specifically I am looking to make it so that I can unmarshal into a set of structures hopefully off of an interface.
I have:
type Package struct {
Account string
Jobs []*Jobs
Libraries map[string]string
}
type Jobs struct {
// Name of the job
JobName string `mapstructure:"name" json:"name" yaml:"name" toml:"name"`
// Type of the job. should be one of the strings outlined in the job struct (below)
Job *Job `mapstructure:"job" json:"job" yaml:"job" toml:"job"`
// Not marshalled
JobResult string
// For multiple values
JobVars []*Variable
}
type Job struct {
// Sets/Resets the primary account to use
Account *Account `mapstructure:"account" json:"account" yaml:"account" toml:"account"`
// Set an arbitrary value
Set *Set `mapstructure:"set" json:"set" yaml:"set" toml:"set"`
// Contract compile and send to the chain functions
Deploy *Deploy `mapstructure:"deploy" json:"deploy" yaml:"deploy" toml:"deploy"`
// Send tokens from one account to another
Send *Send `mapstructure:"send" json:"send" yaml:"send" toml:"send"`
// Utilize eris:db's native name registry to register a name
RegisterName *RegisterName `mapstructure:"register" json:"register" yaml:"register" toml:"register"`
// Sends a transaction which will update the permissions of an account. Must be sent from an account which
// has root permissions on the blockchain (as set by either the genesis.json or in a subsequence transaction)
Permission *Permission `mapstructure:"permission" json:"permission" yaml:"permission" toml:"permission"`
// Sends a bond transaction
Bond *Bond `mapstructure:"bond" json:"bond" yaml:"bond" toml:"bond"`
// Sends an unbond transaction
Unbond *Unbond `mapstructure:"unbond" json:"unbond" yaml:"unbond" toml:"unbond"`
// Sends a rebond transaction
Rebond *Rebond `mapstructure:"rebond" json:"rebond" yaml:"rebond" toml:"rebond"`
// Sends a transaction to a contract. Will utilize eris-abi under the hood to perform all of the heavy lifting
Call *Call `mapstructure:"call" json:"call" yaml:"call" toml:"call"`
// Wrapper for mintdump dump. WIP
DumpState *DumpState `mapstructure:"dump-state" json:"dump-state" yaml:"dump-state" toml:"dump-state"`
// Wrapper for mintdum restore. WIP
RestoreState *RestoreState `mapstructure:"restore-state" json:"restore-state" yaml:"restore-state" toml:"restore-state"`
// Sends a "simulated call" to a contract. Predominantly used for accessor functions ("Getters" within contracts)
QueryContract *QueryContract `mapstructure:"query-contract" json:"query-contract" yaml:"query-contract" toml:"query-contract"`
// Queries information from an account.
QueryAccount *QueryAccount `mapstructure:"query-account" json:"query-account" yaml:"query-account" toml:"query-account"`
// Queries information about a name registered with eris:db's native name registry
QueryName *QueryName `mapstructure:"query-name" json:"query-name" yaml:"query-name" toml:"query-name"`
// Queries information about the validator set
QueryVals *QueryVals `mapstructure:"query-vals" json:"query-vals" yaml:"query-vals" toml:"query-vals"`
// Makes and assertion (useful for testing purposes)
Assert *Assert `mapstructure:"assert" json:"assert" yaml:"assert" toml:"assert"`
}
What I would like to do is to have jobs contain a map of string to Job and eliminate the job field, while maintaining order in which they were placed in from the config file. (Currently using viper). Any and all suggestions for how to achieve this are welcome.
You would need to hold the keys in a separate slice and work with that.
type fifoJob struct {
m map[string]*Job
order []string
result []string
// Not sure where JobVars will go.
}
func (str *fifoJob) Enqueue(key string, val *Job) {
str.m[key] = val
str.order = append(str.order, key)
}
func (str *fifoJob) Dequeue() {
if len(str.order) > 0 {
delete(str.m, str.order[0])
str.order = str.order[1:]
}
}
Anyways if you're using viper you can use something like the fifoJob struct defined above. Also note that I'm making a few assumptions here.
type Package struct {
Account string
Jobs *fifoJob
Libraries map[string]string
}
var config Package
config.Jobs = fifoJob{}
config.Jobs.m = map[string]*Job{}
// Your config file would need to store the order in an array.
// Would've been easy if viper had a getSlice method returning []interface{}
config.Jobs.order = viper.GetStringSlice("package.jobs.order")
for k,v := range viper.GetStringMap("package.jobs.jobmap") {
if job, ok := v.(Job); ok {
config.Jobs.m[k] = &job
}
}
for
PS: You're giving too many irrelevant details in your question. I was asking for a MCVE.
Maps are by nature unordered but you can fill up a slice instead with your keys. Then you can range over your slice and sort it however you like. You can pull out specific elements in your slice with [i].
Check out pages 170, 203, or 204 of some great examples of this:
Programming in Go
I'm developing an application where data is stored in mongodb. There are several collections and of course all of them have some common fields (like Id, creation date, etc) and methods (for example Insert). In my vision, I need to create base model struct with needed fields and methods, and then embed this struct into my models. Unfortunately, this doesn't work because method defined for base model doesn't see child fields.
I don't know how to explain further. Here is code in playground:
https://play.golang.org/p/_x-B78g4TV
It uses json instead of mgo, but idea is still the same.
I want the output to be:
Saving to 'my_model_collection'
{"_id":42, "foo": "Some value for foo", "bar": "Here we set some value for bar"}
Not:
Saving to 'my_model_collection'
{"_id":42}
Writing that insert method for each my model seems to be against DRY, so what is correct/idiomatic way to achieve this in Go?
This is not possible, for details see my answer: Can embedded struct method have knowledge of parent/child?
You may do 2 things:
1. Abandon method and make it a helper / utility function
The idea is to make Insert() detached from BaseModel and make it a simple function, and you pass the document to it which you want to save.
I personally prefer this option, as it requires less hassle and maintenance. It could look like this:
func Insert(doc interface{}) {
j, _ := json.Marshal(doc)
fmt.Println(string(j))
}
You also had a "typo" in the tags:
type MyModel struct {
*BaseModel
Foo string `json:"foo"`
Bar string `json:"bar"`
}
Using it:
Insert(m)
Output (try it on the Go Playground):
{"_id":42,"foo":"Some value for foo","bar":"Here we set some value for bar"}
2. Pass the (pointer to) the wrapper to the BaseModel
In this approach, you have to pass a pointer to the embedder struct so the BaseModel.Insert() method will have a pointer to it, and may use that to save / marshal. This is basically manually maintaining a "reference" to the struct that embeds us and is being saved/marshalled.
This is how it could look like:
type BaseModel struct {
Id int `json:"_id"`
collectionName string
wrapper interface{}
}
And then in the Insert() method save the wrapper:
func (m *BaseModel) Insert() {
fmt.Printf("Saving to '%v'\n", m.collectionName)
j, _ := json.Marshal(m.wrapper)
fmt.Println(string(j))
}
Creation is slightly more complex:
func NewMyModel() *MyModel {
mm := &MyModel{
Foo: "Some value for foo",
}
mm.BaseModel = NewBaseModel("my_model_collection", mm)
return mm
}
But output is as you wish:
Saving to 'my_model_collection'
{"_id":42,"foo":"Some value for foo","bar":"Here we set some value for bar"}
Try it on the Go Playground.
In Golang, you can't override a parent method, because that's not how polymorphism works. The Insert method will apply on the BaseModel member, and not on MyModel.
Also, you're trying to use mgo in an improper way. If you want to insert documents in collections, then you already have an Insert method for a Collection struct which works on interface{} types (same as json.Marshal).
Of course, you can have a BaseModel that will contain fields shared by all of your models. In fact, GORM uses a similar approach and provides a Model struct to be included in every child model.
Well known problem ;o) Member variables (like collectionName) which name starts with lower letter are not visible from other packages (like json). Therefore change struct to:
type BaseModel struct {
Id int `json:"_id"`
CollectionName string `json:"collectionName"`
}
and world will be better place to live in.
Whats the correct way in go to distinguish between when a value in a struct was never set, or is just empty, for example, given the following:
type Organisation struct {
Category string
Code string
Name string
}
I need to know (for example) if the category was never set, or was saved as blank by the user, should I be doing this:
type Organisation struct {
Category *string
Code *string
Name *string
}
I also need to ensure I correctly persist either null or an empty string to the database
I'm still learning GO so it is entirely possible my question needs more info.
The zero value for a string is an empty string, and you can't distinguish between the two.
If you are using the database/sql package, and need to distinguish between NULL and empty strings, consider using the sql.NullString type. It is a simple struct that keeps track of the NULL state:
type NullString struct {
String string
Valid bool // Valid is true if String is not NULL
}
You can scan into this type and use it as a query parameter, and the package will handle the NULL state for you.
Google's protocol buffers (https://code.google.com/p/goprotobuf/) use pointers to describe optional fields.
The generated objects provide GetFoo methods which take the pain away from testing for nil (a.GetFoo() returns an empty string if a.Foo is nil, otherwise it returns *a.Foo).
It introduces a nuisance when you want to write literal structs (in tests, for example), because &"something" is not valid syntax to generate a pointer to a string, so you need a helper function (see, for example, the source code of the protocol buffer library for proto.String).
// String is a helper routine that allocates a new string value
// to store v and returns a pointer to it.
func String(v string) *string {
return &v
}
Overall, using pointers to represent optional fields is not without drawbacks, but it's certainly a viable design choice.
The standard database/sql package provides a NullString struct (members are just String string and Valid bool). To take care of some of the repetitive work of persistence, you could look at an object-relational manager like gorp.
I looked into whether there was some way to distinguish two kinds of empty string just out of curiosity, and couldn't find one. With []bytes, []byte{} == []byte(nil) currently returns false, but I'm not sure if the spec guarantees that to always remain true. In any case, it seems like the most practical thing to do is to go with the flow and use NullString.