When Unmarshalling data from an endpoint, I almost always have a struct with the data that I am want to get from my request that I unmarshal into. Well there are times where I want to see everything that is returning back to me but I am not sure what some of that data is so I don't know how to define it in my struct. Is there a way to have a struct that can just grab all data being unmarshalled without me having to specify it?
This is what I thought would work but that is not the case
resp, err := httpClient.Get("/api/stuff")
data, _ := ioutil.ReadAll(resp.Body)
var myStruct struct{}
json.Unmarshal(data, myStruct)
If you do not know the composition of a JSON object in advance, you can unmarshal into a map[string]interface{}.
var myMap map[string]interface{}
json.Unmarshal(data, &myMap)
See an example here.
If you don't know how to define your struct, then maybe you should use a map. It's a pretty good fit for unmarshalling JSON. Here is an example of how to do it without knowing exactly what data you are receiving:
func parseJSON(r *http.Request) (map[string]interface{}, error) {
var postData interface{}
decoder := json.NewDecoder(r.Body)
err := decoder.Decode(&postData)
if err != nil {
return nil, err
}
return postData.(map[string]interface{}), nil
}
Now you at least have a string name of each piece of data, which should give your application some idea of how to handle it.
Related
I am trying to use generics to convert a JSON object into a GRPC response that has an enum, i.e.:
type GRPCResponse {
str string
enu EnumType
}
type EnumType int32
const (
Type1 EnumType = 0
Type2 EnumType = 1
)
The function for unmarshalling looks like this:
func assertHTTPResponseOK[T any](t *testing.T, endpoint string) T {
body, err := GetResponse(endpoint)
var v T
err := json.Unmarshal(body, &v)
require.Nil(t, err)
return v
}
And the code calling it looks like this:
assertHTTPResponseOK[*GRPCResponse](t, "some-endpoint")
The JSON object in question looks like this:
{"str":"hello", "enu": "Type2"}
and I am getting an error along the lines of:
json: cannot unmarshal string into Go struct field GRPCResponse.enu of type EnumType
From similar questions, I have seen that the usual advice is to use jsonpb.Unmarshal or protojson.Unmarshal instead of the typical json.Unmarshal.
In changing the Unmarshal function, I also had to change T to be protoreflect.ProtoMessage. However, this prevents me from passing a pointer to v to Unmarshal, because it is a pointer to the interface, not the interface. Of course, I also cannot pass in a nil pointer (not take the address of v).
So my questions are:
Is there a way to have the pointer of this generic object satisfy the interface protoreflect.ProtoMessage?
Is there a better Unmarshalling function that would better fit my problem?
I ended up passing in the object I am unmarshalling into.
obj := new(GRPCResponse)
assertHTTPResponseOK[*GRPCResponse](t, ctx, "some-endpoint", obj)
func assertHTTPResponseOK[T protoreflect.ProtoMessage](t *testing.T, ctx context.Context, endpoint string, object T) {
body, err := GetResponse(endpoint)
require.Nil(t, err)
err = protojson.Unmarshal(body, object)
require.Nil(t, err)
}
Here's a generics-friendly proto unmarshaller that avoids passing the second type, at the cost of a reflective invoke to peek the type inside the pointer and call it's constructor.
var msg T // Constrained to proto.Message
// Peek the type inside T (as T= *SomeProtoMsgType)
msgType := reflect.TypeOf(msg).Elem()
// Make a new one, and throw it back into T
msg = reflect.New(msgType).Interface().(T)
errUnmarshal := proto.Unmarshal(body, msg)
When working with DynamoDB in Golang, if a call to query has more results, it will set LastEvaluatedKey on the QueryOutput, which you can then pass in to your next call to query as ExclusiveStartKey to pick up where you left off.
This works great when the values stay in Golang. However, I am writing a paginated API endpoint, so I would like to serialize this key so I can hand it back to the client as a pagination token. Something like this, where something is the magic package that does what I want:
type GetDomainObjectsResponse struct {
Items []MyDomainObject `json:"items"`
NextToken string `json:"next_token"`
}
func GetDomainObjects(w http.ResponseWriter, req *http.Request) {
// ... parse query params, set up dynamoIn ...
dynamoIn.ExclusiveStartKey = something.Decode(params.NextToken)
dynamoOut, _ := db.Query(dynamoIn)
response := GetDomainObjectsResponse{}
dynamodbattribute.UnmarshalListOfMaps(dynamoOut.Items, &response.Items)
response.NextToken := something.Encode(dynamoOut.LastEvaluatedKey)
// ... marshal and write the response ...
}
(please forgive any typos in the above, it's a toy version of the code I whipped up quickly to isolate the issue)
Because I'll need to support several endpoints with different search patterns, I would love a way to generate pagination tokens that doesn't depend on the specific search key.
The trouble is, I haven't found a clean and generic way to serialize the LastEvaluatedKey. You can marshal it directly to JSON (and then e.g. base64 encode it to get a token), but doing so is not reversible. LastEvaluatedKey is a map[string]types.AttributeValue, and types.AttributeValue is an interface, so while the json encoder can read it, it can't write it.
For example, the following code panics with panic: json: cannot unmarshal object into Go value of type types.AttributeValue.
lastEvaluatedKey := map[string]types.AttributeValue{
"year": &types.AttributeValueMemberN{Value: "1993"},
"title": &types.AttributeValueMemberS{Value: "Benny & Joon"},
}
bytes, err := json.Marshal(lastEvaluatedKey)
if err != nil {
panic(err)
}
decoded := map[string]types.AttributeValue{}
err = json.Unmarshal(bytes, &decoded)
if err != nil {
panic(err)
}
What I would love would be a way to use the DynamoDB-flavored JSON directly, like what you get when you run aws dynamodb query on the CLI. Unfortunately the golang SDK doesn't support this.
I suppose I could write my own serializer / deserializer for the AttributeValue types, but that's more effort than this project deserves.
Has anyone found a generic way to do this?
OK, I figured something out.
type GetDomainObjectsResponse struct {
Items []MyDomainObject `json:"items"`
NextToken string `json:"next_token"`
}
func GetDomainObjects(w http.ResponseWriter, req *http.Request) {
// ... parse query params, set up dynamoIn ...
eskMap := map[string]string{}
json.Unmarshal(params.NextToken, &eskMap)
esk, _ = dynamodbattribute.MarshalMap(eskMap)
dynamoIn.ExclusiveStartKey = esk
dynamoOut, _ := db.Query(dynamoIn)
response := GetDomainObjectsResponse{}
dynamodbattribute.UnmarshalListOfMaps(dynamoOut.Items, &response.Items)
lek := map[string]string{}
dynamodbattribute.UnmarshalMap(dynamoOut.LastEvaluatedKey, &lek)
response.NextToken := json.Marshal(lek)
// ... marshal and write the response ...
}
(again this is my real solution hastily transferred back to the toy problem, so please forgive any typos)
As #buraksurdar pointed out, attributevalue.Unmarshal takes an inteface{}. Turns out in addition to a concrete type, you can pass in a map[string]string, and it just works.
I believe this will NOT work if the AttributeValue is not flat, so this isn't a general solution [citation needed]. But my understanding is the LastEvaluatedKey returned from a call to Query will always be flat, so it works for this usecase.
Inspired by Dan, here is a solution to serialize and deserialize to/from base64
package dynamodb_helpers
import (
"encoding/base64"
"encoding/json"
"github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue"
"github.com/aws/aws-sdk-go-v2/service/dynamodb/types"
)
func Serialize(input map[string]types.AttributeValue) (*string, error) {
var inputMap map[string]interface{}
err := attributevalue.UnmarshalMap(input, &inputMap)
if err != nil {
return nil, err
}
bytesJSON, err := json.Marshal(inputMap)
if err != nil {
return nil, err
}
output := base64.StdEncoding.EncodeToString(bytesJSON)
return &output, nil
}
func Deserialize(input string) (map[string]types.AttributeValue, error) {
bytesJSON, err := base64.StdEncoding.DecodeString(input)
if err != nil {
return nil, err
}
outputJSON := map[string]interface{}{}
err = json.Unmarshal(bytesJSON, &outputJSON)
if err != nil {
return nil, err
}
return attributevalue.MarshalMap(outputJSON)
}
I have a use case where I have the code as below. I have a request coming in to hit the backend where I need to append data to a map. My question is how do I convert the below type to a []byte to unmarshal?
Any ideas would be appreciated.
type Example struct {
Category string `json:"category"`
Name string `json:"name"`
}
Incoming Postman request json looks like this:
[{"Category":"TestCategory", "Name":"Sample1"}]
but after doing
jsonString Type: []Example
if err := gc.ShouldBindJSON(&jsonString) it looks like [{TestCategory Sample1}] ; how do I convert this to a []byte?
for _, req := range blob{
var jsonString Example
if err := json.Unmarshal([]byte(jsonString), &blob); err != nil { //this does not work
logger.Fatal(err)
}
//I am checking if a key-value is present and appending it to the map
dict := make(map[string][]Example)
dict[req.Category] = append(dict[req.Category], req)
fmt.Println(dict)
if value, ok := dict["TestCategory"]; ok {
fmt.Printf("Found %d\n", value)
} else {
fmt.Println("not found")
}
}
//I was able to test the above logic by declaring the jsonString as a const and it works
There are two directions in which you can move the data:
from JSON to a Go data structure
// This is your payload coming from the request.
jsonStr := `[{"Category":"TestCategory", "Name":"Sample1"}]`
// This is the Go struct that will hold the unmarshalled data.
var examples []Example
err := json.Unmarshal([]byte(jsonStr), &examples)
if err != nil {
log.Fatal(err)
}
fmt.Println("Examples:", examples) // prints "Examples: [{TestCategory Sample1}]"
from a Go data structure to JSON (either string or []byte)
exampleBytes, err := json.Marshal(examples)
if err != nil {
log.Fatal(err)
}
fmt.Println("Example bytes:", string(exampleBytes)) // prints "Example bytes: [{"category":"TestCategory","name":"Sample1"}]"
You should check out "Go by Example" if you haven't already: https://gobyexample.com/json
Looking at your code:
You are looping on blob but instead of using the req you are trying to unmarshal onto the entire blob each time. I'm not sure what you are trying to achieve there but nothing good can come out of changing a struct you're looping over from within the loop.
The request JSON you are listing is an array of JSON objects. You are trying to unmarshal that into a single Example struct. That won't work, you need an array of those.
I'm just picking up go, so apologies if my terminology isn't precise. My end goal is to add the name of a cachebusted CSS file to the layout template of my Go application. The CSS file built on the fly when the application starts up so can't be hardcoded. In my template file I have this:
//More html here
<link href="{{.CSSFile}}" rel="stylesheet">
//more html here
I have a Render method on a View type like shown below. It takes data interface{} as an argument and then runs ExecuteTemplate. It is called by every controller in one way or another that sends the data argument and exposes information. I know how to add it as data from the controller that then calls the Render method, but I obviously don't want to add the CSS file in every single controller action, so it makes the most sense to add it in the Render function one time and have it added to the data that gets passed to ExecuteTemplate. My issue is how do I add this information to the data already being passed to Render and then pass that whole of information to ExecuteTemplate. What I have below works for the CSS file, but it obviously doesn't send along the data that was passed to the original Render argument.
type View struct {
Template *template.Template
Layout string
}
func (v *View) Render(w http.ResponseWriter, data interface{}) error {
d := Data{}
d.AddCSSFile()
w.Header().Set("Content-Type", "text/html")
err := v.Template.ExecuteTemplate(w, v.Layout, d)
if err != nil {
log.Println(err)
fmt.Fprintln(w, "<h1>Something went wrong. Please contact us at support")
}
return nil
}
type Data struct {
Alerts []Alert
Yield interface{}
CSSFile interface{}
}
func (d *Data) AddCSSFile() {
ss, _ := filepath.Glob("./assets/site-*.css")
fp := strings.Join(ss, "")
_, d.CSSFile = filepath.Split(fp)
}
I've created a gist which, not entirely complete, is a little more fleshed out of what I'm trying to do:
https://gist.github.com/codelitt/549a68149add0482c6dc2514a46aa580
Not sure I understand exactly what you're asking but if what you want is to combine the data interface{} argument with the d := Data{} value inside Render, then you could do something like this...
// ...
func (v *View) Render(w http.ResponseWriter, data interface{}) error {
p := Page{Data:data}
p.AddCSSFile()
w.Header().Set("Content-Type", "text/html")
err := v.Template.ExecuteTemplate(w, v.Layout, p)
if err != nil {
log.Println(err)
fmt.Fprintln(w, "<h1>Something went wrong. Please contact us at support")
}
return nil
}
type Page struct {
Alerts []Alert
Yield interface{}
CSSFile interface{}
Data interface{}
}
func (p *Page) AddCSSFile() {
// ...
}
Edit: Or you could also just initialize an anonymous struct value and pass it to ExecuteTemplate without having to change you existing Data type.
// ...
err := v.Template.ExecuteTemplate(w, v.Layout, struct{
Data
Args interface{}
}{Data:d, Args:data})
// ...
Edit2: So if I understand your comment correctly the data interface{} argument passed to the Render method could in some or all instances be or contain a value of a type that matches one of the Data fields' types, in which case you would like to set that value to that field so as to pass it together to the ExecuteTemplate method. At least one solution to that, as you've already found out, is to use type assertion. Below is a slightly modified version of your example from the comment in the context of your original example from the question.
func (v *View) Render(w http.ResponseWriter, data interface{}) error {
d := Data{}
d.AddCSSFile()
if alerts, ok := data.([]Alert); ok {
d.Alerts = alerts
}
w.Header().Set("Content-Type", "text/html")
err := v.Template.ExecuteTemplate(w, v.Layout, d)
if err != nil {
log.Println(err)
fmt.Fprintln(w, "<h1>Something went wrong. Please contact us at support")
}
return nil
}
I'm trying to dynamically create a slice of structs that I can marshal my data into, but doing it at runtime based on some unknown struct type. I'm doing this because I want to have one common bit of code that can unmarshal anything that implements a specific interface.
e.g. (pseudo code)
func unmarshalMyData(myInterface MyInterface, jsonData []byte) {
targetSlice := myInterface.EmptySlice()
json.Unmarshal(jsonData, &targetSlice)
}
I've tried several different alternatives. The one that seems the most promising is using the reflect package to construct the slice. Unfortunately, after unmarshalling, the dynamically created slice has a type of []interface{}. If I print out the type of the dynamically created slice before unmarshalling it prints []*main.myObj. Can anyone explain why? See playground link: https://play.golang.org/p/vvf1leuQeY
Is there some other way to dynamically create a slice of structs that unmarshals correctly?
I'm aware of json.RawMessage, but there are a few reasons I can't use it... there is no "type" field in my json. Also, the code where I am unmarshalling has no compile time knowledge of the struct that it is unmarshalling. It only knows that the struct implements a specific interface.
Some code that baffles me as to why dynamically created slice doesn't maintain its type after unmarshalling:
package main
import (
"encoding/json"
"fmt"
"reflect"
)
func main() {
input := `[{"myField":"one"},{"myField":"two"}]`
myObjSlice := []*myObj{}
fmt.Printf("Type of myObjSlice before unmarshalling %T\n", myObjSlice)
err := json.Unmarshal([]byte(input), &myObjSlice)
if err != nil {
panic(err)
}
fmt.Printf("Type of myObjSlice after unmarshalling %T\n", myObjSlice)
myConstructedObjSlice := reflect.MakeSlice(reflect.SliceOf(reflect.TypeOf(&myObj{})), 0, 0).Interface()
fmt.Printf("Type of myConstructedObjSlice before unmarshalling %T\n", myConstructedObjSlice)
err = json.Unmarshal([]byte(input), &myConstructedObjSlice)
if err != nil {
panic(err)
}
fmt.Printf("Type of myConstructedObjSlice after unmarshalling %T\n", myConstructedObjSlice)
}
type myObj struct {
myField string
}
The output:
Type of myObjSlice before unmarshalling []*main.myObj
Type of myObjSlice after unmarshalling []*main.myObj
Type of myConstructedObjSlice before unmarshalling []*main.myObj
Type of myConstructedObjSlice after unmarshalling []interface {}
You're building a []*myObj using reflect, but you're then passing an *interface{} to json.Unmarshal. The json package sees the target of the pointer as the type interface{}, and therefore uses its default types to unmarshal into. You need to create a pointer to the slice type you create, so the call to the Interface() method returns the exactly type you want to unmarshal into, which is *[]*myObj.
sliceType := reflect.SliceOf(reflect.TypeOf(&myObj{}))
slicePtr := reflect.New(sliceType)
err = json.Unmarshal([]byte(input), slicePtr.Interface())