Golang Gin Binding Request body XML to Slice - go

I am having an issue with the default binding in Gin. I have an incoming request where the body is multiple Entity objects like so:
<Entity>
<Name>Name One here</Name>
...
</Entity>
<Entity>
<Name>Name Two here</Name>
...
</Entity>
My goal is to map it to a corresponding slice. So the struct for the desired object is like so:
type Entity struct {
XMLName xml.Name `bson:"-" json:"-" xml:"Entity"`
Name string `bson:"name,omitempty" json:",omitempty" xml:",omitempty"`
...
}
The problem I'm experiencing is that only one of the supplied objects is ever mapped into the slice, regardless of how many are passed in the request body. Note that the JSON version of the request parses correctly.
[
{
Name: "Name One",
...
},
{
Name: "Name Two",
...
}
]
I have a struct to model request structure
type ApplicationRequest struct {
XMLName xml.Name `bson:"-" xml:"Entities"`
Entities []Entity `binding:"required" xml:"Entity"`
ParameterOne bool
...
}
So now within the controller function, I handle the binding like this:
func RequestHandler() gin.HandlerFunc {
return func(c *gin.Context) {
var request ApplicationRequest
if err := c.Bind(&request.Entities); err != nil {
responseFunction(http.StatusBadRequest, ..., Message: err.Error()})
return
}
// At this point, the request.Entities slice has ONE element, never more than one
}
}
Note I'm using the gin context.Bind(...) function because it handles the parsing of JSON or XML implicitly, and works for all other scenarios that I need.
Hopefully this provides enough context, any help would be greatly appreciated! Thanks!

it is not a gin problem:
unmarshal-xml-array-in-golang-only-getting-the-first-element
follow is two way to deal it:
add a root node just like #zangw
change the bind method by 'for'
github.com\gin-gonic\gin#v1.8.1\binding\xml.go line 28 func decodeXML
from
func decodeXML(r io.Reader, obj any) error {
decoder := xml.NewDecoder(r)
if err := decoder.Decode(obj); err != nil {
return err
}
return validate(obj)
}
to
func decodeXML(r io.Reader, obj any) error {
decoder := xml.NewDecoder(r)
for {
if err := decoder.Decode(obj); err != nil {
if err == io.EOF{
break
}
return err
}
}
return validate(obj)
}

Related

Can I copy the value of pointer of interface with tags in golang with unknown type

Can I copy the value of pointer of interface with tags in golang with unknown type
I have a function called "CensorPayload" it will receive a pointer and Unmarshal and censor some field that have tag called "censored" but I don't want to update value in argument called "target" that mean I have to copy the value of target to new interface{}. I try many way such as reflect.New but the tag does not pass through Unmarshal or Do function.
type HelloWorld struct {
Hello string `json:"hello"`
Password string `json:"password" censored:"*****"`
}
a := HelloWorld{}
CensorPayload(&a, `{"hello": "world", "password": "12345"}`)
func CensorPayload(target interface{}, payload string) (string, error) {
err := json.Unmarshal([]byte(payload), target)
if err != nil {
return "", err
}
err = Do(target)
if err != nil {
return "", err
}
censoredPayload, err := json.Marshal(target)
if err != nil {
return "", err
}
return string(censoredPayload), nil
}
There are quite a few ways to improve and accomplish this. I'll start with some improvements, and work my way up to possible implementations of what you're looking for.
Improvements
First up, I'd avoid using target interface{} as an argument. There is absolutely nothing there to guarantee that the caller will pass in a pointer. At the very least, I'd consider using a trick that protobuf uses to ensure arguments of the interface proto.Message are pointer values. This is done by adding an empty method with pointer receiver to the types. In this case, that would be:
type JSONPtr interface {
JSONPtr()
}
// implemented:
func (*HelloWorld) JSONPtr() {} // empty
func CensorPayload(target JSONPtr, payload string) (string, error) {
// pretty much the same as you have here
}
Another option would be to use generics, depending on how many types like this you actually need to handle, this could be a valid use-case for them:
type CensorTypes interface {
*HelloWorld | *AnotherType | *YetAnother
}
func CensorPayload[T CensorTypes](target T, payload string) (string, error) {
// same as what you have now
}
Reducing duplication with embedding
If you have a number of types with a Password field, all of which need to be capable of being censored, you could just split out the Password field into its own type and embed it where needed:
type Password struct {
Password string `json:"password"`
}
type HelloWorld struct {
// regular fields here
Password // embed the password field
}
In case you need to use this password censoring a lot, I'd definitely take this approach. Towards the end, I'll include an example where I use embedded types to accomplish what you're trying to do with minimal code duplication...
Possible implementations
All this does, however, is ensure that a pointer is passed, which the interface approach already does anyway. You want to be able to censor the password, and have a specific call that does this in your code, so why not implement a method on the relevant types, and add that to an interface. Adding generics, we can get a compile-time check to ensure that the relevant method(s) were implemented correctly (ie with a pointer receiver):
type Censored interface {
*HelloWorld | *AnotherType // and so on
CensorPass()
}
func (h *HelloWorld) CensorPass() { // pointer receiver
h.Password = "******"
}
// in case `Password` is an embedded type, no need to implement it on `HelloWorld`
func (p *Password) CensorPass() {
p.Password = "******"
}
func CenssorPayload[T Censored](target T, payload string) (string, error) {
// same as before until:
// Do() is replaced with:
target.CensorPass()
// the rest is the same
}
Now, if your HelloWorld type implements CensorPass with a value receiver (func (h HelloWorld) CensorPass), or the caller fails to pass in a pointer value as the target argument, you'll get an error.
Option 2: custom marshallers
Another option here is to create a wrapper type that embeds all of the types you need to handle, and implement a custom marshaller interface on said type:
type Wrapper struct {
*HelloWorld
*AnotherType
*YestAnother
}
func (w Wrapper) MarshalJSON() ([]byte, error) {
if w.HelloWorld != nil {
w.HelloWorld.Password = "*****"
return json.Marshal(w.HelloWorld)
}
if w.AnotherType != nil {
w.AnotherType.Pass = "*****"
w.AnotherType.AnotherSensitiveField = "*******"
return json.Marshal(w.AnotherType)
}
func w.YetAnother != nil {
w.YetAnother.User = "<censored>"
return json.Marshal(w.YetAnother)
}
}
Populating this Wrapper type can all be handled by a single function and a type-switch. I'm using any (short for interface{} below, but I'd recommend you use an interface or generics for the reasons given above if you want to call this method from other packages, if you're calling it from CensorPayload alone, and you've already changed it to use interfaces/generics, the code below is fine:
func wrap(target any) (*Wrapper, error) {
switch tt := target.(type) {
case *HelloWorld:
return &Wrapper{
HelloWorld: tt,
}, nil
case *AnotherType:
return &Wrapper{
AnotherType: tt,
}, nil
case *YetAnother:
return &Wrapper{
YetAnother: tt,
}, nil
}
// if we reach this point, the type passed to this function is unknown/can't be censored
return nil, errors.New("type not supported by wrapper")
}
With this in place, you can simply change your CensorPayload function to something like this:
// CensorPayload using interface to ensure pointer argument
func CensorPayload(target JSONPtr, payload string) (string, error) {
if err := json.Unmarshal(target, []byte(payload)); err != nil {
return "", err
}
wrapped, err := wrap(target)
if err != nil {
// type not handled by wrapper, either return an error
return "", err // target was not supported
// or in case that means this data should not be censored, just return the marhsalled raw data:
raw, err := json.Marshal(target)
if err != nil {
return "", err
}
return string(raw), nil
}
raw, err := json.Marshal(wrapped) // this will call Wrapper.MarshalJSON and censor what needs to be censored
if err != nil {
return "", err
}
return string(raw), nil
}
What I'd go for
I'd probably use the Censored type in the CensorPayload function. It has the benefit of not needing the wrapper at all, shifts the responsibility of knowing what fields/values to censor on to the data-types, and ensures that both the CensorPass method is implemented correctly (pointer receiver), and the actual value being passed is a pointer. To reduce the total LoC to a minimum, I'd leave it to the caller to cast the []byte return value to a string, too. Same goes for the payload argument. Raw JSON data is represented as a byte slice, so the payload argument, IMO, should reflect this:
func CensorPayload[T Censored](target T, payload []byte) ([]byte, error) {
if err := json.Unmarshal(target, payload); err != nil {
return nil, err
}
target.CensorPass() // type knows what fields to censor
return json.Marshal(target) // return the data censored & marshalled
}
As mentioned above, I'd combine this approach with commonly censored fields being embedded types:
type Password struct {
Password `json:"password"`
}
type AnotherSensitiveField struct {
AnotherSensitiveField string `json:"example_db_ip"`
}
// Password implements CensorPass, is embedded so we don't need to do anything here
type HelloWorld struct {
// fields
Password
}
// this one has 2 fields that implement CensorPass
// we need to ensure both are called
type AnotherType struct {
// fields
Password
AnotherSensitiveField
}
// this is a one-off, so we need to implement CensorPass on the type
type YetAnother struct {
// all fields - no password
User string `json:"user"` // this one needs to be censored
}
func (a *AnotherType) CensorPass() {
a.Password.CensorPass()
a.AnotherSensitiveField.CensorPass()
}
func (y *YetAnother) CensorPass() {
y.User = "<censored>"
}
func (p *Password) CensorPass() {
p.Password = "******"
}
func (a *AnotherSensitiveField) CensorPass() {
a.AnotherSensitiveField = "******"
}
Any type that embeds one type like Password will automatically have a CensorPass method. The types are embedded, and no other embedded types have a name collision, so the top level type can resolve HelloWorld.CensorPass to HelloWorld.Password.CensorPass(). This doesn't work for AnotherType, as it has 2 embedded types that provide this method. To get around this, we simply implement the method at the top level, and pass the call on to the embedded types. The third example is one where we want to censor a specific field that isn't used anywhere else (yet). Creating a separate type to embed isn't required, so we can just implement the method directly on the YetAnother type. Should we have data where user credentials are present, and we wish to censor multiple fields in different places, we can easily create a type for this:
type Credentials struct {
User string `json:"user"`
Pass string `json:"pass"`
}
type FullName struct {
Name string `json:"name"`
First string `json:"first_name"`
}
func (c *Credentials) CensorPass() {
c.User = "<redacted>"
c.Pass = "******"
}
func (f *FullName) CensorPass() {
if len(f.First) > 0 {
f.First = string([]rune(f.First)[0])
}
if len(f.Last) == 0 {
return
}
last := make([]string, 0, 2)
for _, v := range strings.Split(f.Last) {
if len(v) > 0 {
last = append(last, string([]rune(v)[0]))
}
}
last = append(last, "") // for the final .
f.Last = strings.Join(last, ". ") // initials only
}
And all we have to do is embed this type wherever we need it. Once that's done, our main function still looks like this:
func CensorPayload[T Censored](target T, payload []byte) ([]byte, error) {
if err := json.Unmarshal(target, payload); err != nil {
return nil, err
}
target.CensorPass() // type knows what fields to censor
return json.Marshal(target) // return the data censored & marshalled
}
But now, we can add new types to this Censored constraint very easily:
type Login struct {
Credentials // username pass
FullName
}
type Person struct {
FullName
Status string `json:"status"`
}
func (n *NewType) CensorPass() {
n.Credentials.CensorPass()
n.FullName.CensorPass()
}
With these couple of lines of code, we now have 2 more types that can be passed in to the CensorPayload function (by updating the constraint, of course). Given a Login and Person payload like:
// login
{
"user": "myLogin",
"pass": "weakpass",
"name": "Pothering Smythe",
"first_name": "Emanual",
}
// person
{
"name": "Doe",
"first_name": "John",
"status": "missing",
}
We should get the output:
{
"user": "<redacted>",
"pass": "******",
"name": "P. S. "
"first_name": "E",
}
{
"name": "D. ",
"first_name": "J",
"status": "missing",
}
Note:
I've written all of the code above without testing it out first. There could by typo's and mistakes here and there, but it should contain enough useful information and examples to get you started. If I have some spare time, I'll probably give this a whirl and update the snippets where needed

Marshal/Unmarshal google.protobuf.Any proto message

I'm currently using gRPC for my API communication. Now I need the value of my request can accept any data type, be it a struct object or just a primitive data type like single integer, string, or boolean. I tried using google.protobuf.Any as below but it doesn't work when I want to marshal and unmarshal it.
message MyRequest {
google.protobuf.Any item = 1; // could be 9, "a string", true, false, {"key": value}, etc.
}
proto-compiled result:
type MyRequest struct {
Item *anypb.Any `protobuf:"bytes,1,opt,name=item,proto3" json:"item,omitempty"`
}
I will marshal the incoming request through gRPC and save it as a JSON string in the database:
bytes, err = json.Marshal(m.request.Item)
if err != nil {
return err
}
itemStr := string(bytes)
But when I want to unmarshal the string back into *anypb.Any by:
func getItem(itemStr string) (structpb.Value, error) {
var item structpb.Value
err := json.Unmarshal([]byte(itemStr), &item)
if err != nil {
return item, err
}
// also I've no idea on how to convert `structpb.Value` to `anypb.Any`
return item, nil
}
It returns an error which seems to be because the struct *anypb.Any doesn't contain any of the fields in my encoded item string. Is there a correct way to solve this problem?
Consider using anypb
In the documentation it shows an example of how to unmarshal it
m := new(foopb.MyMessage)
if err := any.UnmarshalTo(m); err != nil {
... // handle error
}
... // make use of m

Go-Gin binding data with one-to-many relationship

I'm new to Golang and Gin framework, I have created two models
type Product struct {
gorm.Model
Name string
Media []Media
}
type Media struct {
gorm.Model
URI string
ProductID uint
}
and I send a POST request to save a new product, the body was:
{
"name": "Product1",
"media": [
"https://server.com/image1",
"https://server.com/image2",
"https://server.com/image3",
"https://server.com/video1",
"https://server.com/video2"
]
}
And I save a new product using this code
product := Product{}
if err := context.ShouldBindJSON(product); err != nil { // <-- here the error
context.String(http.StatusBadRequest, fmt.Sprintf("err: %s", err.Error()))
return
}
tx := DB.Create(&product)
if tx.Error != nil {
context.String(http.StatusBadRequest, fmt.Sprintf("err: %s", tx.Error))
return
}
the return error message is
err: json: cannot unmarshal string into Go struct field Product.Media of type models.Media
I know that ShouldBindJSON can't convert media-string to media-object, but what is the best practice to do this?
Your payload doesn't match the model. In the JSON body, media is an array of strings, whereas in the model it's a struct with two fields and the embedded gorm model.
If you can't change anything of your current setup, implement UnmarshalJSON on Media and set the URI field from the raw bytes. In the same method you may also initialize ProductID to something (if needed).
func (m *Media) UnmarshalJSON(b []byte) error {
m.URI = string(b)
return nil
}
Then the binding will work as expected:
product := Product{}
// pass a pointer to product
if err := context.ShouldBindJSON(&product); err != nil {
// handle err ...
return
}
fmt.Println(product) // {Product1 [{"https://server.com/image1" 0} ... }

Bind request method POST

I have a problem with binding my request, because there are a lot of parameters, so I used struct containing param.
package api
import (
"github.com/labstack/echo/v4"
"net/http"
"trains-api/domain/models"
"trains-api/domain/services"
)
type reqCreate struct {
RequestNotifi models.ResquestCreateNotifi
}
func CreateNotification (c echo.Context) error {
req := reqCreate{}
if err := c.Bind(req); err != nil {
return c.JSON(http.StatusNotFound, err)
}
}
package models
type RequestCreateNotifi struct {
Name_param1 string `db:"Name_param1"`
Name_param2 string `db:"Name_param2"`
....
Name_param_n string `db:"Name_paramN"`
}
error at if err := c.Bind(req); err != nil
r = {interface {} | string } "reflect: Elem of invalid type"
You need to set the JSON equivalent of each field in the model like so:
package models
type RequestCreateNotifi struct {
Name_param1 string `json:"name_param1" db:"Name_param1"`
Name_param2 string `json:"name_param2" db:"Name_param2"`
....
Name_param_n string `json:"name_param_n" db:"Name_param n"`
}
This json field specifies how the field is represented in the request so it can bind it to the correct value.
You need to add the pointer
req := reqCreate{}
if err := c.Bind(&req); err != nil {
return c.JSON(http.StatusNotFound, err)
}
Unfortunately you can't bind automatically query parameter using Post methode for security reasons according to issue#1670, the way to do it is using echo.QueryParamsBinder
type Query struct {
Param1 string `query:"param1"`
Param2 string `query:"param2"`
}
...
query := new(Query)
err := echo.QueryParamsBinder(ctx).String("param1", &query.Param1).String("param2", &query.Param2).BindError()
...

Strange behaviour when Unmarshalling into struct in Go

I'm developing a tool that can be implemented to simplify the process of creating simple CRUD operations/endpoints. Since my endpoints don't know what kind of struct they'll be receiving, I've created an interface that users can implement, and return an empty object to be filled.
type ItemFactory interface {
GenerateEmptyItem() interface{}
}
And the users would implement something like:
type Test struct {
TestString string `json:"testString"`
TestInt int `json:"testInt"`
TestBool bool `json:"testBool"`
}
func (t Test) GenerateEmptyItem() interface{} {
return Test{}
}
When the Test object gets created, its type is "Test", even though the func returned an interface{}. However, as soon as I try to unmarshal some json of the same format into it, it strips it of its type, and becomes of type "map[string]interface {}".
item := h.ItemFactory.GenerateEmptyItem()
//Prints "Test"
fmt.Printf("%T\n", item)
fmt.Println(reflect.TypeOf(item))
err := ConvertRequestBodyIntoObject(r, &item)
if err != nil {...}
//Prints "map[string]interface {}"
fmt.Printf("%T\n", item)
Func that unmarshalls item:
func ConvertRequestBodyIntoObject(request *http.Request, object interface{}) error {
body, err := ioutil.ReadAll(request.Body)
if err != nil {
return err
}
// Unmarshal body into request object
err = json.Unmarshal(body, object)
if err != nil {
return err
}
return nil
}
Any suggestions as to why this happens, or how I can work around it?
Thanks
Your question lacks an example showing this behavior so I'm just guessing this is what is happening.
func Generate() interface{} {
return Test{}
}
func GeneratePointer() interface{} {
return &Test{}
}
func main() {
vi := Generate()
json.Unmarshal([]byte(`{}`), &vi)
fmt.Printf("Generate Type: %T\n", vi)
vp := GeneratePointer()
json.Unmarshal([]byte(`{}`), vp)
fmt.Printf("GenerateP Type: %T\n", vp)
}
Which outputs:
Generate Type: map[string]interface {}
GenerateP Type: *main.Test
I suggest you return a pointer in GenerateEmptyItem() instead of the actual struct value which is demonstrated in the GenerateP() example.
Playground Example

Resources