Reading and Unmarshalling API results in Golang - go

In the below program I'm extracting some data from an API.
It outputs a rather complex data.
When I ioutil.ReadAll(resp.Body), the result is of type []uint8.
If I try to read the results, its just a random array of integers.
However, I'm able to read it if I convert it to string using string(diskinfo)
But I want to use this in a Struct and having trouble unmarshalling.
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
"net/url"
"reflect"
)
type ApiResults struct {
results []struct {
statement_id int `json.statement_id`
series []struct {
name string `json.name`
tags struct {
host string `json.host`
}
columns []string `json.columns`
values []interface{} `json.values`
}
}
}
func main() {
my_url := "my_url"
my_qry := fmt.Sprintf("my_query")
resp, err := http.Get(my_url + url.QueryEscape(my_qry))
if err != nil {
fmt.Printf("ERROR: %v\n", err)
} else {
fmt.Println(reflect.TypeOf(resp))
diskinfo, _ := ioutil.ReadAll(resp.Body)
fmt.Println(reflect.TypeOf((diskinfo)))
fmt.Println(diskinfo)
fmt.Println(string(diskinfo))
diskinfo_string := string(diskinfo)
data := ApiResults{}
json.Unmarshal([]byte(diskinfo_string), &data)
//fmt.Printf("Values = %v\n", data.results.series.values)
//fmt.Printf("Server = %v\n", data.results.series.tags.host)
}
}
If I view the data as a string, I get this (formatted):
{"results":[
{"statement_id":0,
"series":[
{"name":"disk",
"tags":{"host":"myServer1"},
"columns":["time","disk_size"],
"values":[["2021-07-07T07:53:32.291490387Z",1044]]},
{"name":"disk",
"tags":{"host":"myServer2"},
"columns":["time","disk_size"],
"values":[["2021-07-07T07:53:32.291490387Z",1046]]}
]}
]}
I think my Apireturn struct is also structured incorrectly because the API results have info for multiple hosts.
But first, I doubt if the data has to be sent in a different format to the struct. Once I do this, I can probably try to figure out how to read from the Struct next.

The ioutil.ReadAll already provides you the data in the type byte[]. Therefore you can just call json.Unmarshal passing it as a parameter.
import (
"encoding/json"
"io/ioutil"
"net/http"
)
func toStruct(res *http.Response) (*ApiResults, error) {
body, err := ioutil.ReadAll(res.Body)
if err != nil {
return nil, err
}
defer res.Body.Close()
data := ApiResults{}
if err := json.Unmarshal(body, &data); err != nil {
return nil, err
}
return data, nil
}
There also seems to be an issue with your struct. The correct way to use struct tags is as follows. Plus, fields need to be exported for the json tag (used by json.Umarshal) to work – starting with uppercase will do it.
type ApiResults struct {
Results []struct {
StatementId int `json:"statement_id"`
Series []struct {
Name string `json:"name"`
Tags struct {
Host string `json:"host"`
} `json:"tags"`
Columns []string `json:"columns"`
Values []interface{} `json:"values"`
} `json:"series"`
} `json:"results"`
}

Related

How to extract data from map[string]interface{}?

I am sending the data to the API like following:
{"after": {"amount": 811,"id":123,"status":"Hi"}, "key": [70]}
and i am getting following while printing :
map[after:map[amount:811 id:123 status:Hi ] key:[70]]
Is there any way to print individual field like this??
amount::800
id:123
status:Hi
The code:
package main
import (
"encoding/json"
"fmt"
"io"
"io/ioutil"
"log"
"net/http"
"strings"
)
var (
PORT = ":8080"
)
func main() {
fmt.Println("In Main")
http.HandleFunc("/", changedData)
http.ListenAndServe(PORT, nil)
}
type Data struct {
Id int64 `json:"id"`
Amount float64 `json:"amount"`
Status string `json:"status"`
}
type mark map[string]interface{}
func changedData(w http.ResponseWriter, r *http.Request) {
fmt.Println("Coming From API")
reqBody, _ := ioutil.ReadAll(r.Body)
fmt.Println("Data coming from API ", string(reqBody))
digit := json.NewDecoder(strings.NewReader(string(reqBody)))
for digit.More() {
var result mark
err := digit.Decode(&result)
if err != nil {
if err != io.EOF {
log.Fatal(err)
}
break
}
fmt.Println("final_data ", result)
}
}
Decode to a Go type that matches the structure of the JSON document. You declared a type for the "after" field. Wrap that type with a struct to match the document.
func changedData(w http.ResponseWriter, r *http.Request) {
var v struct{ After Data }
err := json.NewDecoder(r.Body).Decode(&v)
if err != nil {
http.Error(w, "bad request", 400)
return
}
fmt.Printf("final_data: %#v", v.After)
}
Playground example.
I think you can define a struct type if you know the JSON file format or if the JSON format is predefined. As far as I know that mostly using interface{} is a way when you don't know the JSON format or there is no predefined format of the JSON. If you define a struct type  and use it while unmarshaling the JSON to struct, you can access the variables by typing like data.Id or data.Status.
Here's an example code:
package main
import (
"encoding/json"
"fmt"
)
type Data struct {
AfterData After `json:"after"`
Key []int `json:"key"`
}
type After struct {
Id int64 `json:"id"`
Amount float64 `json:"amount"`
Status string `json:"status"`
}
func main() {
j := []byte(`{"after": {"amount": 811,"id":123,"status":"Hi"}, "key": [70]}`)
var data *Data
err := json.Unmarshal(j, &data)
if err != nil {
fmt.Println(err.Error())
return
}
fmt.Println(data.AfterData)
fmt.Println(data.AfterData.Id)
fmt.Println(data.AfterData.Amount)
fmt.Println(data.AfterData.Status)
}
Output will be
{123 811 Hi}
123
811
Hi
Go Playground

How to omit empty json fields using json.decoder

I try to understand why both functions return the same output.
As far as I understood, the point of omit empty is to not add that key to the result struct.
I wrote this example, I was expecting the first output not to have the "Empty" key, but for some reason its value still shows as 0.
package main
import (
"encoding/json"
"fmt"
"strings"
)
type agentOmitEmpty struct {
Alias string `json:"Alias,omitempty"`
Skilled bool `json:"Skilled,omitempty"`
FinID int32 `json:"FinId,omitempty"`
Empty int `json:"Empty,omitempty"`
}
type agent struct {
Alias string `json:"Alias"`
Skilled bool `json:"Skilled"`
FinID int32 `json:"FinId"`
Empty int `json:"Empty"`
}
func main() {
jsonString := `{
"Alias":"Robert",
"Skilled":true,
"FinId":12345
}`
fmt.Printf("output with omit emtpy: %v\n", withEmpty(strings.NewReader(jsonString)))
// output with omit emtpy: {Robert true 12345 0}
fmt.Printf("output regular: %v\n", withoutEmpty(strings.NewReader(jsonString)))
// output without omit: {Robert true 12345 0}
}
func withEmpty(r *strings.Reader) agentOmitEmpty {
dec := json.NewDecoder(r)
body := agentOmitEmpty{}
err := dec.Decode(&body)
if err != nil {
panic(err)
}
return body
}
func withoutEmpty(r *strings.Reader) agent {
dec := json.NewDecoder(r)
body := agent{}
err := dec.Decode(&body)
if err != nil {
panic(err)
}
return body
}
You need to define Empty as *int so it will be replaced with nil when there is no value. Then it will not be saved in the database.

Golang JSON Array

I am trying to use WordNik () to get random words for the dictionary for this script: https://github.com/jmagrippis/password
WordNik is outputting:
[{"id":7936915,"word":"Tanganyikan"},{"id":27180,"word":"cartographic"},{"id":48094,"word":"deterministic"},{"id":1485119,"word":"higher-risk"},{"id":120986,"word":"juristic"},{"id":1830806,"word":"magnetorheological"},{"id":320495,"word":"quelled"},{"id":324610,"word":"remoter"},{"id":215158,"word":"telemetric"},{"id":225207,"word":"uninquisitive"}]
Here is my code:
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"log"
"net/http"
"time"
"github.com/jmagrippis/password"
)
type Words struct {
id []int64
word []string
}
type GetWordsResponse struct {
WordList []Words
}
func getWords(speech string) (*GetWordsResponse, error) {
url := fmt.Sprintf("http://api.wordnik.com/v4/words.json/randomWords?hasDictionaryDef=false&includePartOfSpeech=%s&minCorpusCount=0&maxCorpusCount=-1&minDictionaryCount=1&maxDictionaryCount=-1&minLength=5&maxLength=-1&limit=10&api_key=api_key", speech)
res, err := http.Get(url)
if err != nil {
panic(err.Error())
}
body, err := ioutil.ReadAll(res.Body)
if err != nil {
panic(err.Error())
}
var s = new(GetWordsResponse)
var arr []string
_ = json.Unmarshal([]byte(body), &arr)
log.Printf("Unmarshaled: %v", arr)
return s, err
}
func main() {
dictionary := &password.Dictionary{
Adjectives: []string{"beautiful", "homely", "magical", "posh", "excellent"},
Subjects: []string{"mermaids", "unicorns", "lions", "piranhas"},
Verbs: []string{"love", "fancy", "eat", "bring", "fear", "aggravate"},
Adverbs: []string{"cuddling", "slapping", "shouting", "jumping"},
Objects: []string{"teddy-bears", "diamonds", "buckets", "boxes"},
}
generator := password.NewGenerator(dictionary, time.Now().UnixNano())
pass := generator.Generate()
fmt.Printf("%s", pass)
getWords("Verb")
}
As you can see, what I am trying to do is use WordNik API to request Adverbs, nouns, etc and then make a dictionary based off those words to generate a password. I am horrible with arrays and handling data.
As pointed out in the comments you need to export the fields so they can be unmarshaled
The encoding/json package relies on reflection and since it is in another package, it cannot access unexported fields. ( in go, fields, methods or functions that start with a small letter are unexported whereas with capital letters they are exported )
Then your example json does not contain the WordList at all, so what you want is unmarshaling directly into the array of Words. Also a words object only consists of Id and Word and not of arrays themselves.
type Words struct {
Id int64
Word string
}
func main() {
....
var words []Words
// you don't need to read the whole body first, you can decode in the same turn like this
err := json.NewDecoder(req.Body).Decode(&words)
if nil != err {
log.Fatal(err)
}
...
}
Another very important thing is that you should not ignore errors. This would have helped you to debug the issue. ( What I mean is _ = json.Unmarshal)
As for beginning with go, you could implement a simple test to see if your code works as intended.
https://play.golang.org/p/nuz9uXdka5S < check this working example for reference.

Unable to Unmarshall the data correctly in golang from a map to a structure

I am currently unable to unmarshall the data correctly from a map into a structure. Following is a code snippet (Brief Code at playground):
Request you to kindly provide the reason of getting default values on unmarshlling back the data.
package main
import (
"fmt"
"encoding/json"
"os"
)
func main() {
fmt.Println("Hello, playground")
type PDPOffer struct {
cart_value int `json:"cart_value"`
discount_amount_default int `json:"discount_amount_default"`
max_discount string `json:"max_discount"`
}
a:= map[string]interface{} {
"cart_value" : 1,
"max_discount" : 2,
}
var pdf PDPOffer
b, err := json.Marshal(a)
if err != nil {
fmt.Println("error:", err)
}
os.Stdout.Write(b)//working
err1 := json.Unmarshal(b, &pdf)
if err1 != nil {
fmt.Println("error:", err)
}
fmt.Printf("%+v", pdf)//displaying just the defualt values????????
}
json.Marshal and json.Unmarshal can only work on exported struct fields. Your fields aren't exported and aren't visible to the json code.
The reason for your unmarshal not working is that you need to expose fields of the struct and for doing that you need to start field name with Capital letters. Something has below:
type PDPOffer struct {
Cart_value int `json:"cart_value"`
Discount_amount_default int `json:"discount_amount_default"`
Max_discount string `json:"max_discount"`
}
Also, you were trying to marshal an int value into a string for "max_discount", you need to store it as a string in your map that you're marshaling:
a := map[string]interface{}{
"cart_value": 1,
"max_discount": "2",
}
The error handling had a bug checking for err1 != nil then printing err which was hiding the message error: json: cannot unmarshal number into Go value of type string
Working example with all the fixes: http://play.golang.org/p/L8VC-531nS

how to access deeply nested json keys and values

I'm writing a websocket client in Go. I'm receiving the following JSON from the server:
{"args":[{"time":"2013-05-21 16:57:17"}],"name":"send:time"}
I'm trying to access the time parameter, but just can't grasp how to reach deep into an interface type:
package main;
import "encoding/json"
import "log"
func main() {
msg := `{"args":[{"time":"2013-05-21 16:56:16", "tzs":[{"name":"GMT"}]}],"name":"send:time"}`
u := map[string]interface{}{}
err := json.Unmarshal([]byte(msg), &u)
if err != nil {
panic(err)
}
args := u["args"]
log.Println( args[0]["time"] ) // invalid notation...
}
Which obviously errors, since the notation is not right:
invalid operation: args[0] (index of type interface {})
I just can't find a way to dig into the map to grab deeply nested keys and values.
Once I can get over grabbing dynamic values, I'd like to declare these messages. How would I write a type struct to represent such complex data structs?
You may like to consider the package github.com/bitly/go-simplejson
See the doc: http://godoc.org/github.com/bitly/go-simplejson
Example:
time, err := json.Get("args").GetIndex(0).String("time")
if err != nil {
panic(err)
}
log.Println(time)
The interface{} part of the map[string]interface{} you decode into will match the type of that field. So in this case:
args.([]interface{})[0].(map[string]interface{})["time"].(string)
should return "2013-05-21 16:56:16"
However, if you know the structure of the JSON, you should try defining a struct that matches that structure and unmarshal into that. Ex:
type Time struct {
Time time.Time `json:"time"`
Timezone []TZStruct `json:"tzs"` // obv. you need to define TZStruct as well
Name string `json:"name"`
}
type TimeResponse struct {
Args []Time `json:"args"`
}
var t TimeResponse
json.Unmarshal(msg, &t)
That may not be perfect, but should give you the idea
I'm extremely new to Golang coming from Python, and have always struggled with encode/decoding json. I found gjson at https://github.com/tidwall/gjson, and it helped me immensely:
package main
import "github.com/tidwall/gjson"
func main() {
msg := (`{"args":[{"time":"2013-05-21 16:56:16", "tzs":[{"name":"GMT"}]}],"name":"send:time"}`)
value := gjson.Get(msg, "args.#.time")
println(value.String())
}
-----------------------
["2013-05-21 16:56:16"]
Additionally, I noticed the comment of how to convert into Struct
package main
import (
"encoding/json"
"fmt"
)
type msgFormat struct {
Time string `json:"time"`
Tzs msgFormatTzs `json:"tzs"`
Name string `json:"name"`
}
type msgFormatTzs struct {
TzsName string `json:"name"`
}
func main() {
msg := (`{"args":[{"time":"2013-05-21 16:56:16", "tzs":[{"name":"GMT"}]}],"name":"send:time"}`)
r, err := json.Marshal(msgFormatTzs{msg})
if err != nil {
panic(err)
}
fmt.Printf("%v", r)
}
Try on Go playground

Resources