I'm new to Go and I try to build a Json-builder functionality to practice. My aim is to create a recursive library to build json.
This is the warning I get for the "second" field.
Unexpected newline in composite literal
and here's my attempt. I don't see a mistake here:
package main
import (
"fmt"
)
type JsonNode struct{
fields map[string]interface{}
}
func BuildJson (fields) JsonNode {
jn := &JsonNode{}
for key,value := range fields {
jn.fields[key] = value
}
return jn
}
func main () {
json := BuildJson(
map[string]any{
"first": 1,
"second": BuildJson(map[string]any{"child": "test"}) // Error for this line.
}
)
fmt.Printf(json)
}
You have multiple errors in your code. This version works, I suggest you use some IDE that report errors prior to compilation (they sometimes fix it for you).
package main
import (
"fmt"
)
type JsonNode struct {
fields map[string]interface{}
}
func BuildJson(fields map[string]any) JsonNode {
jn := &JsonNode{}
jn.fields = make(map[string]interface{})
for key, value := range fields {
jn.fields[key] = value
}
return *jn
}
func main() {
json := BuildJson(
map[string]any{
"first": 1,
"second": BuildJson(map[string]any{"child": "test"}), // Error for this line.
},
)
fmt.Println(json)
}
playground
Related
While writing two methods (one for a slice and one for a string map) I realized the implementation for both methods is identical and the only thing that changes is the prototype of the functions.
I'm trying to avoid repeating it and originally I thought of the following (see the FIXME part):
package main
import (
"fmt"
"strings"
)
type SomethingList []*Something
type SomethingMap map[string]*Something
type Something struct {
ID string
Type int
}
func (sl SomethingList) GetIDsString() string {
return getIDsString(sl)
}
func (sm SomethingMap) GetIDsString() string {
return getIDsString(sm)
}
func getIDsString(elements interface{}) string {
var ids []string
// FIXME: Yes, I know I can't iterate over an interface
for element = range elements {
ids = append(ids, element.ID)
}
return strings.Join(ids, ",")
}
func main() {
s1 := Something{ID: "ABC", Type: 1}
s2 := Something{ID: "DEF", Type: 1}
sl := SomethingList{&s1, &s2}
sm := SomethingMap{s1.ID: &s1, s2.ID: &s2}
fmt.Println(sl.GetIDsString())
fmt.Println(sm.GetIDsString())
}
The important part is the function getIDsString which basically takes the ID field of the struct and concatenates it's content across all the members of the slice or map.
I realize now after reading a bit about how interfaces work (yes, I'm quite a newbie in Go as is probably obvious already :-)) that this is not going to work, as Go is statically typed and I can't simply change the types on runtime. Also the interfaces don't allow me to iterate. I've been able to get close using a loop that iterates using the Len() method on the reflect.ValueOf and Index() to access each element. But Index() doesn't work on the string map.
What would be the most idiomatic way of solving this without duplicating much code?
Thanks!
In general repetition of small part of the code in golang is quite common. But in case you have a large amount of duplicate code, you can have that logic in one structure, and ad hoc transform the second structure to the first one to invoke that logic:
package main
import (
"fmt"
"strings"
)
type (
SomethingList []*Something
SomethingMap map[string]*Something
Something struct {
ID string
Type int
}
)
func (sl SomethingList) GetIDsString() string {
ids := make([]string, len(sl))
for i := range sl {
ids[i] = sl[i].ID
}
return strings.Join(ids, ",")
}
func (sm SomethingMap) GetIDsString() string {
l := make(SomethingList, len(sm))
i := 0
for key := range sm {
l[i] = sm[key]
i++
}
return l.GetIDsString()
}
func main() {
s1 := Something{ID: "ABC", Type: 1}
s2 := Something{ID: "DEF", Type: 1}
sl := SomethingList{&s1, &s2}
sm := SomethingMap{s1.ID: &s1, s2.ID: &s2}
fmt.Println(sl.GetIDsString())
fmt.Println(sm.GetIDsString())
}
Alternatively you could decouple the creation of IDsString from the structure itself in a following way.
package main
import (
"fmt"
"strings"
)
type (
SomethingList []*Something
SomethingMap map[string]*Something
Something struct {
ID string
Type int
}
somethingIterator interface {
ForEach(func(value Something))
}
)
func (sl SomethingList) ForEach(f func(value Something)) {
for i := range sl {
f(*sl[i])
}
}
func (sm SomethingMap) ForEach(f func(value Something)) {
for key := range sm {
f(*sm[key])
}
}
func GetIDsString(iterator somethingIterator) string {
var ids []string
iterator.ForEach(func(value Something) {
// Some sophisticated logic is here.
ids = append(ids, value.ID)
})
return strings.Join(ids, ",")
}
func main() {
s1 := Something{ID: "ABC", Type: 1}
s2 := Something{ID: "DEF", Type: 1}
sl := SomethingList{&s1, &s2}
sm := SomethingMap{s1.ID: &s1, s2.ID: &s2}
fmt.Println(GetIDsString(sl))
fmt.Println(GetIDsString(sm))
}
Second approach allows to avoid extra intermediate structure creation which could be beneficial for big list/map.
I need to create a map from a Json array , I started with below in GO , kind of stuck , any pointers ?
package main
import (
"encoding/json"
"fmt"
"strconv"
)
func main() {
jsonStr := `{
"employee": [
{
"id": 14325,
"grpname": "senior"
},
{
"id": 234,
"grpname": "junior"
}
]
}`
type Group struct {
Employee []struct {
GroupName string
GroupId int
}
}
var group []Group
var groupMap []map[string]interface{}
err := json.Unmarshal([]byte(jsonStr), &groupMap)
if err != nil {
panic(err)
}
for _, groupData := range groupMap {
// convert map to array of Group struct
var g Group
g.GroupName = fmt.Sprintf("%s", groupData["grpname"])
g.GroupId, _ = strconv.Atoi(fmt.Sprintf("%v", groupData["id"]))
group = append(group, g)
}
fmt.Println(group)
}
Error:
./prog.go:45:4: g.GroupName undefined (type Group has no field or method GroupName)
./prog.go:46:4: g.GroupId undefined (type Group has no field or method GroupId)
Output expected:
{"senior": 14325,"junior": 234}
Tried few things like below , but getting error : There was an error:%!(EXTRA string=json: cannot unmarshal array into Go struct field GetEmpResponse.employee of type map[string][]model.Employee)
type GetEmpResponse struct {
Employee map[string][]Employee json:"employee"
}
Tried to simplify the Json for my testing , please refer to play.golang.org
empResponse.Employee is an array, so you have to access its elements by index, e.g.
empResponse.Employee[0].ID
In Go, Is it possible to get the tags from a struct field while I'm unmarshaling JSON content to it? Here's my failed attempt at doing so:
package main
import (
"log"
"encoding/json"
)
type Person struct {
ProfileName AltField `json:"profile_name"`
}
type AltField struct {
Val string
}
func (af *AltField) UnmarshalJSON(b []byte) error {
log.Println("Show tags")
//log.Println(af.Tag) // I want to see `json:"profile_name"`
if e := json.Unmarshal(b,&af.Val); e != nil {
return e
}
return nil
}
func main() {
p := Person{}
_ = json.Unmarshal([]byte(`{"profile_name":"Af"}`),&p)
}
I commented out the line log.Println(af.Tag) because it causes compilation errors. If I can get a handle on the tags from the Person struct, that will allow me to develop some other conditional logic.
Is this possible?
Use reflection to get the value of struct field tag. The reflect package provides functions to work with tags including to get the value of tag
package main
import (
"log"
"encoding/json"
"reflect"
)
type Person struct {
ProfileName AltField `json:"profile_name"`
}
type AltField struct {
Val string `json:"val"`
}
func (af *AltField) UnmarshalJSON(b []byte) error {
field, ok := reflect.TypeOf(*af).FieldByName("Val")
if !ok {
panic("Field not found")
}
log.Println(string(field.Tag))
if e := json.Unmarshal(b,&af.Val); e != nil {
return e
}
return nil
}
func main() {
p := Person{}
_ = json.Unmarshal([]byte(`{"profile_name":"Af"}`),&p)
}
You can only get the value of those field tags which has them. The struct field reflect object should be created for fetching the tags of its fields.
Working Code on Playground
I'm trying to set-up an AWS-lambda using aws-sdk-go that is triggered whenever a new user is added to a certain dynamodb table.
Everything is working just fine but I can't find a way to unmarshal a map map[string]DynamoDBAttributeValue like:
{
"name": {
"S" : "John"
},
"residence_address": {
"M": {
"address": {
"S": "some place"
}
}
}
}
To a given struct, for instance, a User struct. Here is shown an example of unsmarhaling a map[string]*dynamodb.AttributeValue into a given interface, but I can't find a way to do the same thing with map[string]DynamoDBAttributeValue even though these types seem to fit the same purposes.
map[string]DynamoDBAttributeValue is returned by a events.DynamoDBEvents from package github.com/aws/aws-lambda-go/events. This is my code:
package handler
import (
"context"
"github.com/aws/aws-lambda-go/events"
"github.com/aws/aws-sdk-go/service/dynamodb/dynamodbattribute"
"github.com/aws/aws-sdk-go/service/dynamodb"
)
func HandleDynamoDBRequest(ctx context.Context, e events.DynamoDBEvent) {
for _, record := range e.Records {
if record.EventName == "INSERT" {
// User Struct
var dynamoUser model.DynamoDBUser
// Of course this can't be done for incompatible types
_ := dynamodbattribute.UnmarshalMap(record.Change.NewImage, &dynamoUser)
}
}
}
Of course, I can marshal record.Change.NewImage to JSON and unmarshal it back to a given struct, but then, I would have to manually initialize dynamoUser attributes starting from the latter ones.
Or I could even write a function that parses map[string]DynamoDBAttributeValue to map[string]*dynamodb.AttributeValue like:
func getAttributeValueMapFromDynamoDBStreamRecord(e events.DynamoDBStreamRecord) map[string]*dynamodb.AttributeValue {
image := e.NewImage
m := make(map[string]*dynamodb.AttributeValue)
for k, v := range image {
if v.DataType() == events.DataTypeString {
s := v.String()
m[k] = &dynamodb.AttributeValue{
S : &s,
}
}
if v.DataType() == events.DataTypeBoolean {
b := v.Boolean()
m[k] = &dynamodb.AttributeValue{
BOOL : &b,
}
}
// . . .
if v.DataType() == events.DataTypeMap {
// ?
}
}
return m
}
And then simply use dynamodbattribute.UnmarshalMap, but on events.DataTypeMap it would be quite a tricky process.
Is there a way through which I can unmarshal a DynamoDB record coming from a events.DynamoDBEvent into a struct with a similar method shown for map[string]*dynamodb.AttributeValue?
I tried the function you provided, and I met some problems with events.DataTypeList, so I managed to write the following function that does the trick:
// UnmarshalStreamImage converts events.DynamoDBAttributeValue to struct
func UnmarshalStreamImage(attribute map[string]events.DynamoDBAttributeValue, out interface{}) error {
dbAttrMap := make(map[string]*dynamodb.AttributeValue)
for k, v := range attribute {
var dbAttr dynamodb.AttributeValue
bytes, marshalErr := v.MarshalJSON(); if marshalErr != nil {
return marshalErr
}
json.Unmarshal(bytes, &dbAttr)
dbAttrMap[k] = &dbAttr
}
return dynamodbattribute.UnmarshalMap(dbAttrMap, out)
}
I was frustrated that the type of NewImage from the record wasn't map[string]*dynamodb.AttributeValue so I could use the dynamodbattribute package.
The JSON representation of events.DynamoDBAttributeValue seems to be the same as the JSON represenation of dynamodb.AttributeValue.
So I tried creating my own DynamoDBEvent type and changed the type of OldImage and NewImage, so it would be marshalled into map[string]*dynamodb.AttributeValue instead of map[string]events.DynamoDBAttributeValue
It is a little bit ugly but it works for me.
package main
import (
"github.com/aws/aws-lambda-go/events"
"github.com/aws/aws-lambda-go/lambda"
"github.com/aws/aws-sdk-go/service/dynamodb"
"github.com/aws/aws-sdk-go/service/dynamodb/dynamodbattribute"
"fmt"
)
func main() {
lambda.Start(lambdaHandler)
}
// changed type of event from: events.DynamoDBEvent to DynamoDBEvent (see below)
func lambdaHandler(event DynamoDBEvent) error {
for _, record := range event.Records {
change := record.Change
newImage := change.NewImage // now of type: map[string]*dynamodb.AttributeValue
var item IdOnly
err := dynamodbattribute.UnmarshalMap(newImage, &item)
if err != nil {
return err
}
fmt.Println(item.Id)
}
return nil
}
type IdOnly struct {
Id string `json:"id"`
}
type DynamoDBEvent struct {
Records []DynamoDBEventRecord `json:"Records"`
}
type DynamoDBEventRecord struct {
AWSRegion string `json:"awsRegion"`
Change DynamoDBStreamRecord `json:"dynamodb"`
EventID string `json:"eventID"`
EventName string `json:"eventName"`
EventSource string `json:"eventSource"`
EventVersion string `json:"eventVersion"`
EventSourceArn string `json:"eventSourceARN"`
UserIdentity *events.DynamoDBUserIdentity `json:"userIdentity,omitempty"`
}
type DynamoDBStreamRecord struct {
ApproximateCreationDateTime events.SecondsEpochTime `json:"ApproximateCreationDateTime,omitempty"`
// changed to map[string]*dynamodb.AttributeValue
Keys map[string]*dynamodb.AttributeValue `json:"Keys,omitempty"`
// changed to map[string]*dynamodb.AttributeValue
NewImage map[string]*dynamodb.AttributeValue `json:"NewImage,omitempty"`
// changed to map[string]*dynamodb.AttributeValue
OldImage map[string]*dynamodb.AttributeValue `json:"OldImage,omitempty"`
SequenceNumber string `json:"SequenceNumber"`
SizeBytes int64 `json:"SizeBytes"`
StreamViewType string `json:"StreamViewType"`
}
I have found the same problem and the solution is to perform a simple conversion of types. This is possible because in the end the type received by lambda events events.DynamoDBAttributeValue and the type used by the SDK V2 of AWS DynamoDB types.AttributeValue are the same. Next I show you the conversion code.
package aws_lambda
import (
"github.com/aws/aws-lambda-go/events"
"github.com/aws/aws-sdk-go-v2/feature/dynamodb/attributevalue"
"github.com/aws/aws-sdk-go-v2/service/dynamodb/types"
)
func UnmarshalDynamoEventsMap(
record map[string]events.DynamoDBAttributeValue, out interface{}) error {
asTypesMap := DynamoDbEventsMapToTypesMap(record)
err := attributevalue.UnmarshalMap(asTypesMap, out)
if err != nil {
return err
}
return nil
}
func DynamoDbEventsMapToTypesMap(
record map[string]events.DynamoDBAttributeValue) map[string]types.AttributeValue {
resultMap := make(map[string]types.AttributeValue)
for key, rec := range record {
resultMap[key] = DynamoDbEventsToTypes(rec)
}
return resultMap
}
// DynamoDbEventsToTypes relates the dynamo event received by AWS Lambda with the data type that is
// used in the Amazon SDK V2 to deal with DynamoDB data.
// This function is necessary because Amazon does not provide any kind of solution to make this
// relationship between types of data.
func DynamoDbEventsToTypes(record events.DynamoDBAttributeValue) types.AttributeValue {
var val types.AttributeValue
switch record.DataType() {
case events.DataTypeBinary:
val = &types.AttributeValueMemberB{
Value: record.Binary(),
}
case events.DataTypeBinarySet:
val = &types.AttributeValueMemberBS{
Value: record.BinarySet(),
}
case events.DataTypeBoolean:
val = &types.AttributeValueMemberBOOL{
Value: record.Boolean(),
}
case events.DataTypeList:
var items []types.AttributeValue
for _, value := range record.List() {
items = append(items, DynamoDbEventsToTypes(value))
}
val = &types.AttributeValueMemberL{
Value: items,
}
case events.DataTypeMap:
items := make(map[string]types.AttributeValue)
for k, v := range record.Map() {
items[k] = DynamoDbEventsToTypes(v)
}
val = &types.AttributeValueMemberM{
Value: items,
}
case events.DataTypeNull:
val = nil
case events.DataTypeNumber:
val = &types.AttributeValueMemberN{
Value: record.Number(),
}
case events.DataTypeNumberSet:
val = &types.AttributeValueMemberNS{
Value: record.NumberSet(),
}
case events.DataTypeString:
val = &types.AttributeValueMemberS{
Value: record.String(),
}
case events.DataTypeStringSet:
val = &types.AttributeValueMemberSS{
Value: record.StringSet(),
}
}
return val
}
There is a package that allows conversion from events.DynamoDBAttributeValue to dynamodb.AttributeValue
https://pkg.go.dev/github.com/aereal/go-dynamodb-attribute-conversions/v2
From there one can unmarshal AttributeValue into struct
func Unmarshal(attribute map[string]events.DynamoDBAttributeValue, out interface{}) error {
av := ddbconversions.AttributeValueMapFrom(attribute)
return attributevalue.UnmarshalMap(av, out)
}
I try to convert interface{} to struct person...
package main
import (
"encoding/json"
"fmt"
)
func FromJson(jsonSrc string) interface{} {
var obj interface{}
json.Unmarshal([]byte(jsonSrc), &obj)
return obj
}
func main() {
type person struct {
Name string
Age int
}
json := `{"Name": "James", "Age": 22}`
actualInterface := FromJson(json)
fmt.Println("actualInterface")
fmt.Println(actualInterface)
var actual person
actual = actualInterface // error fires here -------------------------------
// -------------- type assertion always gives me 'not ok'
// actual, ok := actualInterface.(person)
// if ok {
// fmt.Println("actual")
// fmt.Println(actual)
// } else {
// fmt.Println("not ok")
// fmt.Println(actual)
// }
}
... But got error:
cannot use type interface {} as type person in assignment: need type assertion
To solve this error I tried to use type assertion actual, ok := actualInterface.(person) but always got not ok.
Playground link
The usual way to handle this is to pass a pointer to the output value to your decoding helper function. This avoids type assertions in your application code.
package main
import (
"encoding/json"
"fmt"
)
func FromJson(jsonSrc string, v interface{}) error {
return json.Unmarshal([]byte(jsonSrc), v)
}
func main() {
type person struct {
Name string
Age int
}
json := `{"Name": "James", "Age": 22}`
var p person
err := FromJson(json, &p)
fmt.Println(err)
fmt.Println(p)
}
Your problem is that you're creating an empty interface to begin with, and telling json.Unmarshal to unmarshal into it. While you've defined a person type, json.Unmarshal has no way of knowing that that's what you intend the type of the JSON to be. To fix this, move the definition of person to the top level (that is, move it out of the body of main), and changeFromJson` to this:
func FromJson(jsonSrc string) interface{} {
var obj person{}
json.Unmarshal([]byte(jsonSrc), &obj)
return obj
}
Now, when you return obj, the interface{} that's returned has person as its underlying type. You can run this code on the Go Playground.
By the way, your code is a bit un-idiomatic. I left the original Playground link unmodified except for my corrections so that it wouldn't be needlessly confusing. If you're curious, here's a version that's cleaned up to be more idiomatic (including comments on why I made the changes I did).