Can I copy the value of pointer of interface with tags in golang with unknown type
I have a function called "CensorPayload" it will receive a pointer and Unmarshal and censor some field that have tag called "censored" but I don't want to update value in argument called "target" that mean I have to copy the value of target to new interface{}. I try many way such as reflect.New but the tag does not pass through Unmarshal or Do function.
type HelloWorld struct {
Hello string `json:"hello"`
Password string `json:"password" censored:"*****"`
}
a := HelloWorld{}
CensorPayload(&a, `{"hello": "world", "password": "12345"}`)
func CensorPayload(target interface{}, payload string) (string, error) {
err := json.Unmarshal([]byte(payload), target)
if err != nil {
return "", err
}
err = Do(target)
if err != nil {
return "", err
}
censoredPayload, err := json.Marshal(target)
if err != nil {
return "", err
}
return string(censoredPayload), nil
}
There are quite a few ways to improve and accomplish this. I'll start with some improvements, and work my way up to possible implementations of what you're looking for.
Improvements
First up, I'd avoid using target interface{} as an argument. There is absolutely nothing there to guarantee that the caller will pass in a pointer. At the very least, I'd consider using a trick that protobuf uses to ensure arguments of the interface proto.Message are pointer values. This is done by adding an empty method with pointer receiver to the types. In this case, that would be:
type JSONPtr interface {
JSONPtr()
}
// implemented:
func (*HelloWorld) JSONPtr() {} // empty
func CensorPayload(target JSONPtr, payload string) (string, error) {
// pretty much the same as you have here
}
Another option would be to use generics, depending on how many types like this you actually need to handle, this could be a valid use-case for them:
type CensorTypes interface {
*HelloWorld | *AnotherType | *YetAnother
}
func CensorPayload[T CensorTypes](target T, payload string) (string, error) {
// same as what you have now
}
Reducing duplication with embedding
If you have a number of types with a Password field, all of which need to be capable of being censored, you could just split out the Password field into its own type and embed it where needed:
type Password struct {
Password string `json:"password"`
}
type HelloWorld struct {
// regular fields here
Password // embed the password field
}
In case you need to use this password censoring a lot, I'd definitely take this approach. Towards the end, I'll include an example where I use embedded types to accomplish what you're trying to do with minimal code duplication...
Possible implementations
All this does, however, is ensure that a pointer is passed, which the interface approach already does anyway. You want to be able to censor the password, and have a specific call that does this in your code, so why not implement a method on the relevant types, and add that to an interface. Adding generics, we can get a compile-time check to ensure that the relevant method(s) were implemented correctly (ie with a pointer receiver):
type Censored interface {
*HelloWorld | *AnotherType // and so on
CensorPass()
}
func (h *HelloWorld) CensorPass() { // pointer receiver
h.Password = "******"
}
// in case `Password` is an embedded type, no need to implement it on `HelloWorld`
func (p *Password) CensorPass() {
p.Password = "******"
}
func CenssorPayload[T Censored](target T, payload string) (string, error) {
// same as before until:
// Do() is replaced with:
target.CensorPass()
// the rest is the same
}
Now, if your HelloWorld type implements CensorPass with a value receiver (func (h HelloWorld) CensorPass), or the caller fails to pass in a pointer value as the target argument, you'll get an error.
Option 2: custom marshallers
Another option here is to create a wrapper type that embeds all of the types you need to handle, and implement a custom marshaller interface on said type:
type Wrapper struct {
*HelloWorld
*AnotherType
*YestAnother
}
func (w Wrapper) MarshalJSON() ([]byte, error) {
if w.HelloWorld != nil {
w.HelloWorld.Password = "*****"
return json.Marshal(w.HelloWorld)
}
if w.AnotherType != nil {
w.AnotherType.Pass = "*****"
w.AnotherType.AnotherSensitiveField = "*******"
return json.Marshal(w.AnotherType)
}
func w.YetAnother != nil {
w.YetAnother.User = "<censored>"
return json.Marshal(w.YetAnother)
}
}
Populating this Wrapper type can all be handled by a single function and a type-switch. I'm using any (short for interface{} below, but I'd recommend you use an interface or generics for the reasons given above if you want to call this method from other packages, if you're calling it from CensorPayload alone, and you've already changed it to use interfaces/generics, the code below is fine:
func wrap(target any) (*Wrapper, error) {
switch tt := target.(type) {
case *HelloWorld:
return &Wrapper{
HelloWorld: tt,
}, nil
case *AnotherType:
return &Wrapper{
AnotherType: tt,
}, nil
case *YetAnother:
return &Wrapper{
YetAnother: tt,
}, nil
}
// if we reach this point, the type passed to this function is unknown/can't be censored
return nil, errors.New("type not supported by wrapper")
}
With this in place, you can simply change your CensorPayload function to something like this:
// CensorPayload using interface to ensure pointer argument
func CensorPayload(target JSONPtr, payload string) (string, error) {
if err := json.Unmarshal(target, []byte(payload)); err != nil {
return "", err
}
wrapped, err := wrap(target)
if err != nil {
// type not handled by wrapper, either return an error
return "", err // target was not supported
// or in case that means this data should not be censored, just return the marhsalled raw data:
raw, err := json.Marshal(target)
if err != nil {
return "", err
}
return string(raw), nil
}
raw, err := json.Marshal(wrapped) // this will call Wrapper.MarshalJSON and censor what needs to be censored
if err != nil {
return "", err
}
return string(raw), nil
}
What I'd go for
I'd probably use the Censored type in the CensorPayload function. It has the benefit of not needing the wrapper at all, shifts the responsibility of knowing what fields/values to censor on to the data-types, and ensures that both the CensorPass method is implemented correctly (pointer receiver), and the actual value being passed is a pointer. To reduce the total LoC to a minimum, I'd leave it to the caller to cast the []byte return value to a string, too. Same goes for the payload argument. Raw JSON data is represented as a byte slice, so the payload argument, IMO, should reflect this:
func CensorPayload[T Censored](target T, payload []byte) ([]byte, error) {
if err := json.Unmarshal(target, payload); err != nil {
return nil, err
}
target.CensorPass() // type knows what fields to censor
return json.Marshal(target) // return the data censored & marshalled
}
As mentioned above, I'd combine this approach with commonly censored fields being embedded types:
type Password struct {
Password `json:"password"`
}
type AnotherSensitiveField struct {
AnotherSensitiveField string `json:"example_db_ip"`
}
// Password implements CensorPass, is embedded so we don't need to do anything here
type HelloWorld struct {
// fields
Password
}
// this one has 2 fields that implement CensorPass
// we need to ensure both are called
type AnotherType struct {
// fields
Password
AnotherSensitiveField
}
// this is a one-off, so we need to implement CensorPass on the type
type YetAnother struct {
// all fields - no password
User string `json:"user"` // this one needs to be censored
}
func (a *AnotherType) CensorPass() {
a.Password.CensorPass()
a.AnotherSensitiveField.CensorPass()
}
func (y *YetAnother) CensorPass() {
y.User = "<censored>"
}
func (p *Password) CensorPass() {
p.Password = "******"
}
func (a *AnotherSensitiveField) CensorPass() {
a.AnotherSensitiveField = "******"
}
Any type that embeds one type like Password will automatically have a CensorPass method. The types are embedded, and no other embedded types have a name collision, so the top level type can resolve HelloWorld.CensorPass to HelloWorld.Password.CensorPass(). This doesn't work for AnotherType, as it has 2 embedded types that provide this method. To get around this, we simply implement the method at the top level, and pass the call on to the embedded types. The third example is one where we want to censor a specific field that isn't used anywhere else (yet). Creating a separate type to embed isn't required, so we can just implement the method directly on the YetAnother type. Should we have data where user credentials are present, and we wish to censor multiple fields in different places, we can easily create a type for this:
type Credentials struct {
User string `json:"user"`
Pass string `json:"pass"`
}
type FullName struct {
Name string `json:"name"`
First string `json:"first_name"`
}
func (c *Credentials) CensorPass() {
c.User = "<redacted>"
c.Pass = "******"
}
func (f *FullName) CensorPass() {
if len(f.First) > 0 {
f.First = string([]rune(f.First)[0])
}
if len(f.Last) == 0 {
return
}
last := make([]string, 0, 2)
for _, v := range strings.Split(f.Last) {
if len(v) > 0 {
last = append(last, string([]rune(v)[0]))
}
}
last = append(last, "") // for the final .
f.Last = strings.Join(last, ". ") // initials only
}
And all we have to do is embed this type wherever we need it. Once that's done, our main function still looks like this:
func CensorPayload[T Censored](target T, payload []byte) ([]byte, error) {
if err := json.Unmarshal(target, payload); err != nil {
return nil, err
}
target.CensorPass() // type knows what fields to censor
return json.Marshal(target) // return the data censored & marshalled
}
But now, we can add new types to this Censored constraint very easily:
type Login struct {
Credentials // username pass
FullName
}
type Person struct {
FullName
Status string `json:"status"`
}
func (n *NewType) CensorPass() {
n.Credentials.CensorPass()
n.FullName.CensorPass()
}
With these couple of lines of code, we now have 2 more types that can be passed in to the CensorPayload function (by updating the constraint, of course). Given a Login and Person payload like:
// login
{
"user": "myLogin",
"pass": "weakpass",
"name": "Pothering Smythe",
"first_name": "Emanual",
}
// person
{
"name": "Doe",
"first_name": "John",
"status": "missing",
}
We should get the output:
{
"user": "<redacted>",
"pass": "******",
"name": "P. S. "
"first_name": "E",
}
{
"name": "D. ",
"first_name": "J",
"status": "missing",
}
Note:
I've written all of the code above without testing it out first. There could by typo's and mistakes here and there, but it should contain enough useful information and examples to get you started. If I have some spare time, I'll probably give this a whirl and update the snippets where needed
I have a struct like this
type Data struct {
Foo string `json:"foo" binding:"required"`
}
And I use ShouldBind to bind query or json body to the struct.
data := Data{}
err := ctx.ShouldBind(&data)
I was wondering what is the best practice to trim white space for the string field?
transform {"foo": " bar "} to struct {"foo": "bar"}
I have tried using custom string type, and add custom UnmarshalJSON function, but it won't work for ctx.shouldBind if it is query.
type Data struct {
Foo TrimSpaceString `json:"foo" binding:"required"`
}
type TrimSpaceString string
func (t *TrimSpaceString) UnmarshalJSON(data []byte) error {
data = bytes.Trim(data, "\"")
data = bytes.Trim(data, " ")
*t = TrimSpaceString(strings.TrimSpace(string(data)))
return nil
}
I also tried to use conform and add tag for struct. But I have to add conform.Strings(data) after bind it and it is not convinence.
type Data struct {
Foo TrimSpaceString `json:"foo" binding:"required" conform:"trim"`
}
err := ctx.ShouldBind(&data)
conform.Strings(&data)
Should I custom a Binding and trim string inside Binding?
I got this working by modifying UnmarshalJSON function a bit. I don't have anything to explain as I'm also new, but will try, what's different is that we are converting byte data into a string by json.Unmarshal.Then we trim the space on string and appends it to the field or the original string
type trim string
func (t *trim) UnmarshalJSON(data []byte) error {
var s string
if err := json.Unmarshal(data, &s); err != nil {
return err
}
*t = trim(strings.TrimSpace(s))
fmt.Printf("these are the strings: '%s'\n", *t)
return nil
}
Here is the playground if you want to try it yourself
https://go.dev/play/p/BuxvMQibVsn
Is there any way to generate a OpenAPI specification document for XML using golang structure? My structures are annotated with encoding/xml annotations like this:
type Error struct {
Text string `xml:",chardata"`
Type string `xml:"Type,attr"`
Code string `xml:"Code,attr"`
ShortText string `xml:"ShortText,attr"`
}
I would like to generate a OpenAPI model automatically, which respect these annotations
You can use the reflect package from the standard library to inspect the struct and its tags and you can use the gopkg.in/yaml.v3 package to generate the Open API format. You just need to write the logic that translates your type into another, one that matches the structure of the Open API document.
Here's an example of how you could approach this, note that it is incomplete and should therefore not be used as is:
// declare a structure that matches the OpenAPI's yaml document
type DataType struct {
Type string `yaml:"type,omitempty"`
Props map[string]*DataType `yaml:"properties,omitempty"`
XML *XML `yaml:"xml,omitempty"`
}
type XML struct {
Name string `yaml:"name,omitempty"`
Attr bool `yaml:"attribute,omitempty"`
}
// write a marshal func that converts a given type to the structure declared above
// - this example converts only plain structs
func marshalOpenAPI(v interface{}) (interface{}, error) {
rt := reflect.TypeOf(v)
if rt.Kind() == reflect.Struct {
obj := DataType{Type: "object"}
for i := 0; i < rt.NumField(); i++ {
f := rt.Field(i)
tag := strings.Split(f.Tag.Get("xml"), ",")
if len(tag) == 0 || len(tag[0]) == 0 { // no name?
continue
}
if obj.Props == nil {
obj.Props = make(map[string]*DataType)
}
name := tag[0]
obj.Props[name] = &DataType{
Type: goTypeToOpenAPIType(f.Type),
}
if len(tag) > 1 && tag[1] == "attr" {
obj.Props[name].XML = &XML{Attr: true}
}
}
return obj, nil
}
return nil, fmt.Errorf("unsupported type")
}
// have all your types implement the yaml.Marshaler interface by
// delegating to the marshal func implemented above
func (e Error) MarshalYAML() (interface{}, error) {
return marshalOpenAPI(e)
}
// marshal the types
yaml.Marshal(map[string]map[string]interface{}{
"schemas": {
"error": Error{},
// ...
},
})
Try it on playground.
I want to unmarshal several types from JSON and use the interface to represent the actual struct that it is different. But when I send the struct as interface{} it converts it to a map. The animal.json is:
"{"type":"cat","policies":[{"name":"kitty","parameter":{"duration":600,"percent":90}}]}"
package main
import (
"reflect"
log "github.com/sirupsen/logrus"
"github.com/spf13/viper"
)
func main() {
var err error
animal := New()
viper.SetConfigType("json")
viper.SetConfigName("animal")
viper.AddConfigPath("~/Desktop/")
viper.AddConfigPath(".")
if err := viper.ReadInConfig(); err != nil {
return
}
if err = viper.Unmarshal(&animal); err != nil {
return
}
for _, policy := range animal.Policies {
log.Info(policy.Name)
log.Info(policy.Parameter)
//INFO[0000] map[duration:600 percent:90]
log.Info(reflect.TypeOf(policy.Parameter))
//INFO[0000] map[string]interface {}, Why is it not an interface{} and how do I get it?
switch t := policy.Parameter.(type) {
//why does the switch not work?
case *CatParameter:
log.Info("cat", t)
case *DogParameter:
log.Info("dog", t)
}
}
}
func New() *Animal {
var animal Animal
animal.Type = "cat"
return &animal
}
type Animal struct {
Type string `json:"type" form:"type"`
Policies []Policy `json:"policies" form:"policies"`
}
type CatParameter struct {
Duration int `json:"duration"`
Percent int `json:"percent"`
}
type DogParameter struct {
Percent int `json:"percent"`
Duration int `json:"duration"`
Operation string `json:"operation"`
}
type Policy struct {
Name string `json:"name"`
Parameter interface{} `json:"parameter"`
}
It's json unmarshal feature
If you use an interface{} as a decoder, the default json object for interface{} is map[string]interface{}
You can see it here:
https://godoc.org/encoding/json#Unmarshal
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
So in t := policy.Parameter.(type), the t is map[string]interface{}
For solving your problem, you can try to define another field to distinguish CatParameter or DogParameter
Maybe:
type Policy struct {
Name string `json:"name"`
Parameter Parameter `json:"parameter"`
}
type Parameter struct {
Name string `json:"name"` // cat or dog
Percent int `json:"percent,omitempty"`
Duration int `json:"duration,omitempty"`
Operation string `json:"operation,omitempty"`
}
Code below generates this error json: cannot unmarshal number into Go struct field TMP.a of type string
package main
import (
"encoding/json"
"fmt"
)
var b = []byte(`{"a": "str", "A": 123}`)
type TMP struct {
// A interface{} `json:"a"`
A string `json:"a"`
// A int `json:"A"`
}
func main() {
var tmp TMP
err := json.Unmarshal(b, &tmp)
if err != nil {
fmt.Println(err)
}
}
I've read through https://golang.org/pkg/encoding/json/#Marshal and cannot find anything that states this shouldn't work. What am I missing?
It's been asked that I clarify, and rightly so. What I'm really wondering is why when I use JSON with only 2 keys that differ in uppercase/lowercase why Unmarshal is not keeping the case I've provided in the struct json tag.
First of all change the names of your fields. Since they have same name only uppercase A and lowercase a. So when go try to marshal the fields it is unable to recognize between the fields.
package main
import (
"encoding/json"
"fmt"
)
var b = []byte(`{"a": "str", "B": 123}`)
type TMP struct {
// A interface{} `json:"a"`
A string `json:"A"`
B int `json:"B"`
}
func main() {
var tmp TMP
err := json.Unmarshal(b, &tmp)
if err != nil {
fmt.Println(err)
}
fmt.Println(tmp)
}
As the error says
json: cannot unmarshal string into Go struct field TMP.A of type int
even if you try to pass both fields like below it will give same error
var b = []byte(`{"a": "str", "A": 123}`)
type TMP struct {
// A interface{} `json:"a"`
a string `json:"a"`
A int `json:"A"`
}
The real problem is that you're trying to unmarshal int to a string field. Error will occur even if you remove the "a": "str", and will work fine if you double quote the 123
What is a little funny, however, is that the case is ignored with only one field. Changing your struct to:
type TMP struct {
// A interface{}json:"a"
A stringjson:"a"
B intjson:"A"
}
also works.