I am trying to write a method that will return a function that can satisfy the json.Marshaler interface. My reasoning is to provide different representations of the struct. Perhaps I am approaching this completely wrong.
func (api *Api) SiteList(c *gin.Context) {
var sites []db.Site
if err := api.db.Find(&sites).Error; err != nil {
}
var payload []json.Marshaler
for _, site := range sites {
payload = append(payload, site.ToApi())
}
c.JSON(http.StatusOK, payload)
}
The result I get from this function is the correct number of items in the list, but the same value for each:
[
{
"key": "NZ7LCA9HQN3",
"name": "autumn-waterfall-1573"
},
{
"key": "NZ7LCA9HQN3",
"name": "autumn-waterfall-1573"
},
{
"key": "NZ7LCA9HQN3",
"name": "autumn-waterfall-1573"
},
{
"key": "NZ7LCA9HQN3",
"name": "autumn-waterfall-1573"
},
{
"key": "NZ7LCA9HQN3",
"name": "autumn-waterfall-1573"
}
]
Finally, here is the ToApi implementation:
type EncoderFunc func() ([]byte, error)
func (fn EncoderFunc) MarshalJSON() ([]byte, error) {
return fn()
}
func (site *Site) ToApi() json.Marshaler {
return EncoderFunc(func() ([]byte, error) {
var payload public.Site
payload.Name = site.Name
payload.Key = site.Key
data, err := json.Marshal(payload)
if err != nil {
return nil, err
}
return data, nil
})
}
This seems like a classic closure gotcha. There is a related FAQ section.
Basically, site in your for-loop has the same address every time. All your functions are closed over this address. So when you evaluate it after the for-loop, it will repeatedly call MarshalJSON on the value on the same (last) address. You can correct that by creating a new value on every iteration:
for _, site := range sites {
site := site // Perfectly legal and idiomatic.
payload = append(payload, site.ToApi())
}
Playground: https://play.golang.org/p/eFujC1hEyD
Another related piece of documentation from Effective Go:
The bug is that in a Go for loop, the loop variable is reused for each iteration, so the req variable is shared across all goroutines. That's not what we want. (...) Another solution is just to create a new variable with the same name, as in this example:
for req := range queue {
req := req // Create new instance of req for the goroutine.
sem <- 1
go func() {
process(req)
<-sem
}()
}
This section is about goroutines, but apparently this also applies to all closures.
Related
We want to unmarshal (in golang) a GRPC message and transform it into a map[string]interface{} to further process it. After using this code:
err := ptypes.UnmarshalAny(resource, config)
configMarshal, err := json.Marshal(config)
var configInterface map[string]interface{}
err = json.Unmarshal(configMarshal, &configInterface)
we get the following structure:
{
"name": "envoy.filters.network.tcp_proxy",
"ConfigType": {
"TypedConfig": {
"type_url": "type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy",
"value": "ChBCbGFja0hvbGVDbHVzdGVyEhBCbGFja0hvbGVDbHVzdGVy"
}
}
}
Where the TypedConfig field remains encoded. How can we decode the TypedConfig field? We know the type_url and we know the value, but to unmarshal the field, it needs to be of the pbany.Any type. But because the TypedConfig structure is a map[string] interface {}, our program either fails to compile, or it crashes, complaining that it is expecting a pbany.Any type, but instead it is getting a map[string] interface {}.
We have the following questions:
Is there a way to turn the structure under TypedConfig into a pbany.Any type that can be subsequently unmarshalled?
Is there a way to recursively unmarshal the entire GRPC message?
Edit (provide more information about the code, schemas/packages used)
We are looking at the code of xds_proxy.go here: https://github.com/istio/istio/blob/master/pkg/istio-agent/xds_proxy.go
This code uses a *discovery.DiscoveryResponse structure in this function:
func forwardToEnvoy(con *ProxyConnection, resp *discovery.DiscoveryResponse) {
The protobuf schema for discovery.DiscoveryResponse (and every other structure used in the code) is in the https://github.com/envoyproxy/go-control-plane/ repository in this file: https://github.com/envoyproxy/go-control-plane/blob/main/envoy/service/discovery/v3/discovery.pb.go
We added code to the forwardToEnvoy function to see the entire unmarshalled contents of the *discovery.DiscoveryResponse structure:
var config proto.Message
switch resp.TypeUrl {
case "type.googleapis.com/envoy.config.route.v3.RouteConfiguration":
config = &route.RouteConfiguration{}
case "type.googleapis.com/envoy.config.listener.v3.Listener":
config = &listener.Listener{}
// Six more cases here, truncated to save space
}
for _, resource := range resp.Resources {
err := ptypes.UnmarshalAny(resource, config)
if err != nil {
proxyLog.Infof("UnmarshalAny err %v", err)
return false
}
configMarshal, err := json.Marshal(config)
if err != nil {
proxyLog.Infof("Marshal err %v", err)
return false
}
var configInterface map[string]interface{}
err = json.Unmarshal(configMarshal, &configInterface)
if err != nil {
proxyLog.Infof("Unmarshal err %v", err)
return false
}
}
And this works well, except that now we have these TypedConfig fields that are still encoded:
{
"name": "virtualOutbound",
"address": {
"Address": {
"SocketAddress": {
"address": "0.0.0.0",
"PortSpecifier": {
"PortValue": 15001
}
}
}
},
"filter_chains": [
{
"filter_chain_match": {
"destination_port": {
"value": 15001
}
},
"filters": [
{
"name": "istio.stats",
"ConfigType": {
"TypedConfig": {
"type_url": "type.googleapis.com/udpa.type.v1.TypedStruct",
"value": "CkF0eXBlLmdvb2dsZWFwaXMuY29tL2Vudm95LmV4dGVuc2lvbnMuZmlsdG"
}
}
},
One way to visualize the contents of the TypedConfig fields is to use this code:
for index1, filterChain := range listenerConfig.FilterChains {
for index2, filter := range filterChain.Filters {
proxyLog.Infof("Listener %d: Handling filter chain %d, filter %d", i, index1, index2)
switch filter.ConfigType.(type) {
case *listener.Filter_TypedConfig:
proxyLog.Infof("Found TypedConfig")
typedConfig := filter.GetTypedConfig()
proxyLog.Infof("typedConfig.TypeUrl = %s", typedConfig.TypeUrl)
switch typedConfig.TypeUrl {
case "type.googleapis.com/envoy.extensions.filters.network.tcp_proxy.v3.TcpProxy":
tcpProxyConfig := &tcp_proxy.TcpProxy{}
err := typedConfig.UnmarshalTo(tcpProxyConfig)
if err != nil {
proxyLog.Errorf("Failed to unmarshal TCP proxy configuration")
} else {
proxyLog.Infof("TcpProxy Config for filter chain %d filter %d: %s", index1, index2, prettyPrint(tcpProxyConfig))
}
But then the code becomes very complex, as we have a large number of structures, and these structures can occur in different order in the messages.
So we wanted to get a generic way of unmarshalling these TypedConfig message by using pbAny, and hence our questions.
I want to validate my input map of array with govalidator.ValidateMap.
Please can someone suggest for Sample mapTemplate for map of array.
Please find below the code snippet.
Thanks in Advance
package main
import (
"fmt"
"github.com/asaskevich/govalidator"
)
func main() {
var mapTemplate = map[string]interface{}{
"name": "required,alpha",
"categories": []interface{}{",alpha"}, //error: map validator has to be either map[string]interface{} or string; got []interface {}
}
var inputMap = map[string]interface{}{
"name": "Prabhu",
"categories": []interface{}{"category1", "category2"},
}
result, err := govalidator.ValidateMap(inputMap, mapTemplate)
if err != nil {
fmt.Println("error: " + err.Error())
}
fmt.Printf("result : %v\n", result)
for _, v := range inputMap["categories"].([]interface{}) {
fmt.Printf("category : %v\n", v)
}
}
It seems validation of slices has not yet been implemented. There is no check in the What to contribute list for slices/arrays.
You can however use the function ValidateArray to iterate over a slice and validate its members.
govalidator.ValidateArray(inputMap["categories"], func(val interface{}, i int) bool {
valStr, ok := val.(string)
if !ok {
return false
}
return govalidator.IsAlpha(valStr)
})
I have a project function which returns a slice containing the field values by name of each struct or map in an input slice. I am having trouble with case where the input slice contains pointers to structs. I have setup a recursive function to operate on the value, but need to know how to convert from kind reflect.Ptr to the underlying reflect.Struct. How is this done? Any other design recommendations are appreciated. I am still a bit new to Go.
Here is the code:
func project(in []interface{}, property string) []interface{} {
var result []interface{}
var appendValue func(list []interface{}, el interface{})
appendValue = func(list []interface{}, el interface{}) {
v := reflect.ValueOf(el)
kind := v.Kind()
if kind == reflect.Ptr {
// How do I get the struct behind this ptr?
// appendValue(list, el)
} else if kind == reflect.Struct {
result = append(result, v.FieldByName(property).Interface())
} else if kind == reflect.Map {
result = append(result, el.(map[string]interface{})[property])
} else {
panic("Value must be a struct or map")
}
}
for _, el := range in {
appendValue(result, el)
}
return result
}
... and the test cases:
func Test_project(t *testing.T) {
cases := map[string]struct {
input []interface{}
property string
expected []interface{}
}{
"simple-map": {
[]interface{}{
map[string]interface{}{
"a": "a1",
},
},
"a",
[]interface{}{"a1"},
},
"simple-struct": {
[]interface{}{
simpleStruct{
A: "a1",
},
},
"A",
[]interface{}{"a1"},
},
// THIS ONE FAILS
"simple-struct-ptr": {
[]interface{}{
&simpleStruct{
A: "a1",
},
},
"A",
[]interface{}{"a1"},
},
}
for k, v := range cases {
t.Run(k, func(t *testing.T) {
got := project(v.input, v.property)
if !reflect.DeepEqual(got, v.expected) {
t.Fatalf("Expected %+v, got %+v", v.expected, got)
}
})
}
}
Use Elem() to go from a reflect.Ptr to the value it points to.
This question already has an answer here:
Return custom error message from struct tag validation
(1 answer)
Closed 1 year ago.
For example I've got the following struct
type Address struct {
City string `json:"city" binding:"required"`
AddressLine string `json:"address_line" binding:"required"`
}
and I've got the following function to handle request from users
func AddressCreate(c *gin.Context) {
var address Address
if err := c.BindJSON(&address); err == nil {
// if everything is good save to database
// and return success message
db.Create(&address)
c.JSON(http.StatusOK, gin.H {"status":"success"})
} else {
c.JSON(http.StatusBadRequest, err)
}
}
Expected behavior is to return JSON, formatted this way
[
{
"city":"required"
}
{
"address_line":"required"
}
]
But I'm getting an error formatted like this
"Address.City": {
"FieldNamespace": "Address.City",
"NameNamespace": "City",
"Field": "City",
"Name": "City",
"Tag": "required",
"ActualTag": "required",
"Kind": 24,
"Type": {},
"Param": "",
"Value": ""
},
"Address.AddressLine": {
"FieldNamespace": "AddressLine",
"NameNamespace": "AddressLine",
"Field": "AddressLine",
"Name": "AddressLine",
"Tag": "required",
"ActualTag": "required",
"Kind": 24,
"Type": {},
"Param": "",
"Value": ""
}
What I tried:
I created function which casts error to ValidationErrors and iterates through all FieldError's in it
func ListOfErrors(e error) []map[string]string {
ve := e.(validator.ValidationErrors)
InvalidFields := make([]map[string]string, 0)
for _, e := range ve {
errors := map[string]string{}
// field := reflect.TypeOf(e.NameNamespace)
errors[e.Name] = e.Tag
InvalidFields = append(InvalidFields, errors)
}
return InvalidFields
}
The output look's much better
[
{
"City":"required"
},
{
"AddressLine":"required"
}
]
But I cannot solve the problem with the name of the fields. I cannot swap Name into name which I noted in structs tag json:"city". So my question is did I choose correct way to solve the problem if the answer is yes how to get structs JSON tag for field?
If you want it to be same as defined in your json tag, then you should use reflection to pull that tag from your data type.
I don't have your libraries, so can't compile and check it. But I believe what you are after should go along those lines:
func ListOfErrors(address *Address, e error) []map[string]string {
ve := e.(validator.ValidationErrors)
InvalidFields := make([]map[string]string, 0)
for _, e := range ve {
errors := map[string]string{}
// field := reflect.TypeOf(e.NameNamespace)
field, _ := reflect.TypeOf(address).Elem().FieldByName(e.Name)
jsonTag := string(field.Tag.Get("json"))
errors[jsonTag] = e.Tag
InvalidFields = append(InvalidFields, errors)
}
return InvalidFields
}
Note that it is a bit contrived as type of address parameter is essentially known. So, not strictly required as a function parameter. But you can change address *Address to address interface{} and use it for other types too.
Disclaimer: I skipped error checking for brevity, but you certainly should check for errors in your code (e.g. no such field error or no json tag on that field).
You can use ToSnake to snake case the names:
import (
"unicode"
)
// ToSnake convert the given string to snake case following the Golang format:
// acronyms are converted to lower-case and preceded by an underscore.
func ToSnake(in string) string {
runes := []rune(in)
length := len(runes)
var out []rune
for i := 0; i < length; i++ {
if i > 0 && unicode.IsUpper(runes[i]) && ((i+1 < length && unicode.IsLower(runes[i+1])) || unicode.IsLower(runes[i-1])) {
out = append(out, '_')
}
out = append(out, unicode.ToLower(runes[i]))
}
return string(out)
}
func ListOfErrors(e error) []map[string]string {
ve := e.(validator.ValidationErrors)
invalidFields := make([]map[string]string, 0)
for _, e := range ve {
errors := map[string]string{}
errors[ToSnake(e.Name)] = e.Tag
invalidFields = append(InvalidFields, errors)
}
return invalidFields
}
Can someone explain why the two aren't equivalent? The latter does build, but doesn't work as expected. I thought slices would be changed automatically, as contain a pointer to the array.
// Working spec
func TestProcessRecords(t *testing.T) {
var messageSent []*sqs.SendMessageInput
w := &SQSWriter{
queueURL: aws.String("aQueueURL"),
service: &mock.SQS{
SendMessageStub: func(input *sqs.SendMessageInput) (*sqs.SendMessageOutput, error) {
messageSent = append(messageSent, input)
return nil, nil
},
},
}
inputEvent := readFirehoseEventFromFile(t, "../../../../testdata/firehose_event.json")
processRecords(inputEvent.Records, w)
assert.Equal(t, 2, len(inputEvent.Records))
assert.Equal(t, 1, len(messageSent))
}
Attempted refactoring, as the mockedWriter will be used across specs
// Not Working spec
func mockWriter(messageSent []*sqs.SendMessageInput) *SQSWriter{
return &SQSWriter{
queueURL: aws.String("aQueueURL"),
service: &mock.SQS{
SendMessageStub: func(input *sqs.SendMessageInput) (*sqs.SendMessageOutput, error) {
messageSent = append(messageSent, input)
return nil, nil
},
},
}
}
func TestProcessRecords(t *testing.T) {
messageSent := []*sqs.SendMessageInput{}
inputEvent := readFirehoseEventFromFile(t, "../../../../testdata/firehose_event.json")
processRecords(inputEvent.Records, mockWriter(messageSent))
assert.Equal(t, 2, len(inputEvent.Records))
assert.Equal(t, 1, len(messageSent))
}
I should mention that I'm coming from a background in JS/Ruby/Python, and it is taking a bit of time to get a firmer grasp of go fundamentals.
Thanks in advance
This was the answer, all credits to #mkopriva.
func mockWriter(messageSent *[]*sqs.SendMessageInput) *SQSWriter{
return &SQSWriter{
queueURL: aws.String("aQueueURL"),
service: &mock.SQS{
SendMessageStub: func(input *sqs.SendMessageInput) (*sqs.SendMessageOutput, error) {
*messageSent = *append(messageSent, input)
return nil, nil
},
},
}
}
func TestProcessRecords(t *testing.T) {
messageSent := []*sqs.SendMessageInput{}
inputEvent := readFirehoseEventFromFile(t, "../../../../testdata/firehose_event.json")
processRecords(inputEvent.Records, mockWriter(&messageSent))
assert.Equal(t, 2, len(inputEvent.Records))
assert.Equal(t, 1, len(messageSent))
}