Parsing JSON using struct - go

I'm trying to parse JSON using Go. Can anyone tell me why my code is not working as expected?
package main
import (
"encoding/json"
"fmt"
)
type Message struct {
Name string
Body string
Time int64
}
type Person struct {
M Message
}
func get_content() {
body := []byte(`{"person":{"Name":"Alice","Body":"Hello","Time":1294706395881547000}}`)
var data Person
err := json.Unmarshal(body, &data)
if err != nil {
panic(err.Error())
}
fmt.Printf("%v",data.M.Name)
}
func main() {
get_content()
}
I'm expecting it to print the Name.
Go playground Code

There are two problems in the code.
The first one is what #umar-hayat mentioned above -> you are unmarshalling into the data object and you should be aiming at the data.M field.
The second problem is that your JSON's structure doesn't match your struct's structure. Your Person has a single field called M. If we want to represent this as JSON it would look like this:
{
"M": {
"Name": "Joe",
"Body": "Hi",
"time": 2600
}
}
Instead, you have a field called person in your JSON which cannot be matched to any field in your struct. The fact that it's similar to the name of the struct's type doesn't help in any way, I'm afraid.
So, you can either change your JSON and your target:
body := []byte(`{"Name":"Alice","Body":"Hello","Time":1294706395881547000}`)
var data Person
err := json.Unmarshal(body, &data.M)
Or just your JSON:
body := []byte(`{"M":{"Name":"Alice","Body":"Hello","Time":1294706395881547000}}`)
var data Person
err := json.Unmarshal(body, &data)
But it's essential that the names of the fields in your JSON match the names of the fields in your struct. Or, as mentioned by Konstantinos, you can use tags in order to specify particular names with which your struct's fields will be represented in the JSON.
You might find this helpful: https://gobyexample.com/json

Here is how to Unmarshel JSON to the struct. you can check it on Go Playground here:
package main
import (
"encoding/json"
"fmt"
)
type Message struct {
Name string
Body string
Time int64
}
type Person struct {
M Message
}
func get_content() {
body := []byte(`{"Name":"Alice","Body":"Hello","Time":1294706395881547000}`)
var data Person
err := json.Unmarshal(body, &data.M)
if err != nil {
panic(err.Error())
}
fmt.Printf(data.M.Name)
}
func main() {
get_content()
}
Replace data with data.M in below line.
err := json.Unmarshal(body, &data)

As long as you intent to map Json keys on structs whose fields have different names you should add tags:
type Message struct {
Name string `json:"Name"`
Body string `json:"Body"`
Time int64 `json:"Time"`
}
type Person struct {
M Message `json:"person"`
}
You can find more information here
In addition this answer explains in an nutshell the purpose of tags in go.

Related

How to get and store the document ID using a struct

I've defined a data structure like so:
type Person struct {
Name string `firestore:"name,omitempty"`
}
When I query all the documents in a collection I'd like to be able to attach the ID to the documents for later reference, but not necessarily have ID as an attribute stored in Firestore (unless its the only way). In javascript or python this is straightforward as the data structures are dynamic and I can just query the ID post get() and add it as a dynamic key/value. myObj.id = doc.id
How would I do this with Go?
package main
import (
"fmt"
"cloud.google.com/go/firestore"
"context"
"google.golang.org/api/iterator"
"log"
)
type Person struct {
Name string `firestore:"name,omitempty"`
}
func main() {
ctx := context.Background()
c, err := firestore.NewClient(ctx, "my-project")
if err != nil {
log.Fatalf("error: %v", err)
}
var people []Person
iter := c.Collection("people").Documents(ctx)
for {
doc, err := iter.Next()
if err == iterator.Done {
break
}
if err != nil {
log.Fatalf("error: %v", err)
}
var p Person
err = doc.DataTo(p)
if err != nil {
log.Fatalf("error: %v", err)
}
// id := doc.Ref.ID
people = append(people, p)
}
fmt.Println(people)
}
output, no ID
>> [{John Smith}]
I believe that the firestore struct tags works the same way as the tags in the encoding/json package. So a tag with a value of "-" would mean ignore the field.
So
type Person struct {
ID int `firestore:"-"`
Name string `firestore:"name,omitempty"`
}
should do the trick.
You can set ID yourself, but the firestore pkg will ignore it when reading/writing data.
If you want to store the firestore document ID on the Person type, your struct must have a declared field for it.
Golang firestore docs don't mention this explicitly, but since a firestore doc ID is not part of the document fields, the func (*DocumentSnapshot) DataTo does not populate the ID. Instead, you may get the document ID from the DocumentRef type and add it to Person yourself.
The doc also states that:
Note that this client supports struct tags beginning with "firestore:" that work like the tags of the encoding/json package, letting you rename fields, ignore them, or omit their values when empty
Therefore, if you want to omit the ID when marshaling back for firestore, your could use the tag firestore:"-"
The Person would look like this:
type Person struct {
ID string `firestore:"-"`
Name string `firestore:"name,omitempty"`
}
inside the loop:
var p Person
err := docSnap.DataTo(&p)
if err != nil {
// handle it
}
p.ID = doc.Ref.ID

How to convert string to struct array

I have a json array which is converted into a string. Now I want to map the string to a struct array so that I can modify the string json. Below is my code base
type ProcessdetailsEntity struct {
Source []int64 `json:"source"`
Node string `json:"node"`
Targets []int64 `json:"targets"`
Issave bool `json:"isSave"`
Instate []int64 `json:"inState"`
OutState []int64 `json:"outState"`
}
func main() {
var stringJson = "[{\"source\":[-1],\"node\":\"1_1628008588902\",\"targets\":[],\"isSave\":true,\"inState\":[1],\"outState\":[2]},{\"source\":[\"1_1628008588902\",\"5_1628008613446\"],\"node\":\"2_1628008595757\",\"targets\":[],\"isSave\":true,\"inState\":[2,5],\"outState\":[3,6]}]"
in := []byte(stringJson)
detailsEntity := []ProcessdetailsEntity{}
err := json.Unmarshal(in, &detailsEntity)
if err != nil {
log.Print(err)
}
}
Now when I run this code base I got the error:
json: cannot unmarshal string into Go struct field ProcessdetailsEntity.source of type int64
How to properly map the string to struct so that I can modify the inState and outState value of the json ?
The error you get is already pretty much on the nose:
cannot unmarshal string into Go struct field ProcessdetailsEntity.source of type int64
That tells you that (at least one) of your source fields appears to have the wrong type: a string instead of something that can be represented by a int64.
So let's check your source fields in your stringJson:
"source":[-1]
"source":["1_1628008588902","5_1628008613446"]
As you can see the second source is an array of string. Hence the error.
To solve this you need to make sure that the source is an array of int. Unfortunately, 1_1628008588902 and 5_1628008613446 are not valid integers in Go.
I slightly modified your JSON and fixed your code an then it works:
package main
import (
"encoding/json"
"log"
)
type ProcessdetailsEntity struct {
Source []int64 `json:"source"`
Node string `json:"node"`
Targets []int64 `json:"targets"`
Issave bool `json:"isSave"`
Instate []int64 `json:"inState"`
OutState []int64 `json:"outState"`
}
func main() {
var stringJson = `[
{
"source":[-1],
"node":"1_1628008588902",
"targets":[],
"isSave":true,
"inState":[1],
"outState":[2]
},
{
"source":[11628008588902,51628008613446],
"node":"2_1628008595757",
"targets":[],
"isSave":true,
"inState":[2,5],
"outState":[3,6]
}
]`
in := []byte(stringJson)
detailsEntity := []ProcessdetailsEntity{}
err := json.Unmarshal(in, &detailsEntity)
if err != nil {
log.Print(err)
}
}
See: https://play.golang.org/p/kcrkfRliWJ5

Reading and Unmarshalling API results in Golang

In the below program I'm extracting some data from an API.
It outputs a rather complex data.
When I ioutil.ReadAll(resp.Body), the result is of type []uint8.
If I try to read the results, its just a random array of integers.
However, I'm able to read it if I convert it to string using string(diskinfo)
But I want to use this in a Struct and having trouble unmarshalling.
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
"net/url"
"reflect"
)
type ApiResults struct {
results []struct {
statement_id int `json.statement_id`
series []struct {
name string `json.name`
tags struct {
host string `json.host`
}
columns []string `json.columns`
values []interface{} `json.values`
}
}
}
func main() {
my_url := "my_url"
my_qry := fmt.Sprintf("my_query")
resp, err := http.Get(my_url + url.QueryEscape(my_qry))
if err != nil {
fmt.Printf("ERROR: %v\n", err)
} else {
fmt.Println(reflect.TypeOf(resp))
diskinfo, _ := ioutil.ReadAll(resp.Body)
fmt.Println(reflect.TypeOf((diskinfo)))
fmt.Println(diskinfo)
fmt.Println(string(diskinfo))
diskinfo_string := string(diskinfo)
data := ApiResults{}
json.Unmarshal([]byte(diskinfo_string), &data)
//fmt.Printf("Values = %v\n", data.results.series.values)
//fmt.Printf("Server = %v\n", data.results.series.tags.host)
}
}
If I view the data as a string, I get this (formatted):
{"results":[
{"statement_id":0,
"series":[
{"name":"disk",
"tags":{"host":"myServer1"},
"columns":["time","disk_size"],
"values":[["2021-07-07T07:53:32.291490387Z",1044]]},
{"name":"disk",
"tags":{"host":"myServer2"},
"columns":["time","disk_size"],
"values":[["2021-07-07T07:53:32.291490387Z",1046]]}
]}
]}
I think my Apireturn struct is also structured incorrectly because the API results have info for multiple hosts.
But first, I doubt if the data has to be sent in a different format to the struct. Once I do this, I can probably try to figure out how to read from the Struct next.
The ioutil.ReadAll already provides you the data in the type byte[]. Therefore you can just call json.Unmarshal passing it as a parameter.
import (
"encoding/json"
"io/ioutil"
"net/http"
)
func toStruct(res *http.Response) (*ApiResults, error) {
body, err := ioutil.ReadAll(res.Body)
if err != nil {
return nil, err
}
defer res.Body.Close()
data := ApiResults{}
if err := json.Unmarshal(body, &data); err != nil {
return nil, err
}
return data, nil
}
There also seems to be an issue with your struct. The correct way to use struct tags is as follows. Plus, fields need to be exported for the json tag (used by json.Umarshal) to work – starting with uppercase will do it.
type ApiResults struct {
Results []struct {
StatementId int `json:"statement_id"`
Series []struct {
Name string `json:"name"`
Tags struct {
Host string `json:"host"`
} `json:"tags"`
Columns []string `json:"columns"`
Values []interface{} `json:"values"`
} `json:"series"`
} `json:"results"`
}

Merge structs with some overlapping fields

I saw several questions asking on how to merge unique structs and how to merge identical structs.
But how would I merge structs that have some overlap? and which fields get taken & when?
e.g.:
type structOne struct {
id string `json:id`
date string `json:date`
desc string `json:desc`
}
and
type structTwo struct {
id string `json:id`
date string `json:date`
name string `json:name`
}
how would I merge it such that I get
{
id string `json:id`
date string `json:date`
desc string `json:desc`
name string `json:name`
}
also, what happens if in this case the two id's are the same (assuming a join over id's) but the names are different?
In javascript, doing something like Object.assign(structOne, structTwo).
Go is a strongly typed language, unlike javascript you can't merge two struct into one combined struct because all type are determined at compile-time. You have two solution here :
Using embedded struct:
One great solution is to use embedded struct because you don't have to merge anything anymore.
package main
import (
"bytes"
"encoding/json"
"fmt"
"log"
)
// Shared field
type common struct {
ID string `json:id`
Date string `json:date`
}
type merged struct {
// Common field is embedded
common
Name string `json:name`
Desc string `json:desc`
}
func main() {
buf := bytes.Buffer{}
buf.WriteString("{ \"id\": \"1\", \"date\": \"27/07/2020\", \"desc\": \"the decription...\" }")
merged := &merged{}
err := json.Unmarshal(buf.Bytes(), merged)
if err != nil {
log.Fatal(err)
}
// Look how you can easily access field from
// embedded struct
fmt.Println("ID:", merged.ID)
fmt.Println("Date:", merged.Date)
fmt.Println("Name:", merged.Name)
fmt.Println("Desc:", merged.Desc)
// Output:
// ID: 1
// Date: 27/07/2020
// Name:
// Desc: the decription...
}
If you want to read more about struct embedding:
golangbyexample.com
travix.io
Using Maps
Another solution is to use maps but you will loose the benefits of struct and methods. This example is not the simplest but there is some great example in the other responses.
In this example I'm using Mergo. Mergo is library that can merge structs and map. Here it is used for creating maps object in the Map methods but you can totally write your own methods.
package main
import (
"fmt"
"log"
"github.com/imdario/mergo"
)
type tOne struct {
ID string
Date string
Desc string
}
// Map build a map object from the struct tOne
func (t1 tOne) Map() map[string]interface{} {
m := make(map[string]interface{}, 3)
if err := mergo.Map(&m, t1); err != nil {
log.Fatal(err)
}
return m
}
type tTwo struct {
ID string
Date string
Name string
}
// Map build a map object from the struct tTwo
func (t2 tTwo) Map() map[string]interface{} {
m := make(map[string]interface{}, 3)
if err := mergo.Map(&m, t2); err != nil {
log.Fatal(err)
}
return m
}
func main() {
dst := tOne{
ID: "destination",
Date: "26/07/2020",
Desc: "destination object",
}.Map()
src := tTwo{
ID: "src",
Date: "26/07/1010",
Name: "source name",
}.Map()
if err := mergo.Merge(&dst, src); err != nil {
log.Fatal(err)
}
fmt.Printf("Destination:\n%+v", dst)
// Output:
// Destination:
// map[date:26/07/2020 desc:destination object iD:destination name:object name
}
Go structs and JavaScript objects are very different. Go structs do not have dynamic fields.
If you want dynamic key/value sets that you can easily iterate over and merge, and are very JSON friendly, why not a map[string]interface{}?
$ go run t.go
map[a:1 b:4]
map[a:1 b:4 c:3]
$ cat t.go
package main
import(
"fmt"
)
type MyObj map[string]interface{}
func (mo MyObj)Merge(omo MyObj){
for k, v := range omo {
mo[k] = v
}
}
func main() {
a := MyObj{"a": 1, "b": 4}
b := MyObj{"b": 2, "c": 3}
b.Merge(a)
fmt.Printf("%+v\n%+v\n", a, b)
}
you can use github.com/fatih/structs to convert your struct to map. Then iterate over that map and choose which fields need to copy over. I have a snippet of code which illustrates this solution.
func MergeStruct (a structOne,b structTwo) map[string]interface{}{
a1:=structs.Map(a)
b1:=structs.Map(b)
/* values of structTwo over writes values of structOne */
var myMap=make(map[string]interface{})
for val,key:=range(a1){
myMap[key]=val
}
for val,key:=range(b1){
myMap[key]=val
}
return myMap
}
You can use the reflect package to do this. Try iterating through the two structs and then you can either use another struct type to store the values or maybe use a map.
Check out this question to find out how you can iterate over a struct.
Check out this question to find out how to get the name of the fields.
Remember to use exported field names for the reflect package to work.
Here is an example which works.

how to access deeply nested json keys and values

I'm writing a websocket client in Go. I'm receiving the following JSON from the server:
{"args":[{"time":"2013-05-21 16:57:17"}],"name":"send:time"}
I'm trying to access the time parameter, but just can't grasp how to reach deep into an interface type:
package main;
import "encoding/json"
import "log"
func main() {
msg := `{"args":[{"time":"2013-05-21 16:56:16", "tzs":[{"name":"GMT"}]}],"name":"send:time"}`
u := map[string]interface{}{}
err := json.Unmarshal([]byte(msg), &u)
if err != nil {
panic(err)
}
args := u["args"]
log.Println( args[0]["time"] ) // invalid notation...
}
Which obviously errors, since the notation is not right:
invalid operation: args[0] (index of type interface {})
I just can't find a way to dig into the map to grab deeply nested keys and values.
Once I can get over grabbing dynamic values, I'd like to declare these messages. How would I write a type struct to represent such complex data structs?
You may like to consider the package github.com/bitly/go-simplejson
See the doc: http://godoc.org/github.com/bitly/go-simplejson
Example:
time, err := json.Get("args").GetIndex(0).String("time")
if err != nil {
panic(err)
}
log.Println(time)
The interface{} part of the map[string]interface{} you decode into will match the type of that field. So in this case:
args.([]interface{})[0].(map[string]interface{})["time"].(string)
should return "2013-05-21 16:56:16"
However, if you know the structure of the JSON, you should try defining a struct that matches that structure and unmarshal into that. Ex:
type Time struct {
Time time.Time `json:"time"`
Timezone []TZStruct `json:"tzs"` // obv. you need to define TZStruct as well
Name string `json:"name"`
}
type TimeResponse struct {
Args []Time `json:"args"`
}
var t TimeResponse
json.Unmarshal(msg, &t)
That may not be perfect, but should give you the idea
I'm extremely new to Golang coming from Python, and have always struggled with encode/decoding json. I found gjson at https://github.com/tidwall/gjson, and it helped me immensely:
package main
import "github.com/tidwall/gjson"
func main() {
msg := (`{"args":[{"time":"2013-05-21 16:56:16", "tzs":[{"name":"GMT"}]}],"name":"send:time"}`)
value := gjson.Get(msg, "args.#.time")
println(value.String())
}
-----------------------
["2013-05-21 16:56:16"]
Additionally, I noticed the comment of how to convert into Struct
package main
import (
"encoding/json"
"fmt"
)
type msgFormat struct {
Time string `json:"time"`
Tzs msgFormatTzs `json:"tzs"`
Name string `json:"name"`
}
type msgFormatTzs struct {
TzsName string `json:"name"`
}
func main() {
msg := (`{"args":[{"time":"2013-05-21 16:56:16", "tzs":[{"name":"GMT"}]}],"name":"send:time"}`)
r, err := json.Marshal(msgFormatTzs{msg})
if err != nil {
panic(err)
}
fmt.Printf("%v", r)
}
Try on Go playground

Resources