Unmarshaling values of JSON objects - go

If given the string, from a MediaWiki API request:
str = ` {
"query": {
"pages": {
"66984": {
"pageid": 66984,
"ns": 0,
"title": "Main Page",
"touched": "2012-11-23T06:44:22Z",
"lastrevid": 1347044,
"counter": "",
"length": 28,
"redirect": "",
"starttimestamp": "2012-12-15T05:21:21Z",
"edittoken": "bd7d4a61cc4ce6489e68c21259e6e416+\\"
}
}
}
}`
What can be done to get the edittoken, using Go's json package (keep in mind the 66984 number will continually change)?

When you have a changing key like this the best way to deal with it is with a map. In the example below I've used structs up until the point we reach a changing key. Then I switched to a map format after that. I linked up a working example as well.
http://play.golang.org/p/ny0kyafgYO
package main
import (
"fmt"
"encoding/json"
)
type query struct {
Query struct {
Pages map[string]interface{}
}
}
func main() {
str := `{"query":{"pages":{"66984":{"pageid":66984,"ns":0,"title":"Main Page","touched":"2012-11-23T06:44:22Z","lastrevid":1347044,"counter":"","length":28,"redirect":"","starttimestamp":"2012-12-15T05:21:21Z","edittoken":"bd7d4a61cc4ce6489e68c21259e6e416+\\"}}}}`
q := query{}
err := json.Unmarshal([]byte(str), &q)
if err!=nil {
panic(err)
}
for _, p := range q.Query.Pages {
fmt.Printf("edittoken = %s\n", p.(map[string]interface{})["edittoken"].(string))
}
}

Note that if you use the &indexpageids=true parameter in the API request URL, the result will contain a "pageids" array, like so:
str = ` {
"query": {
"pageids": [
"66984"
],
"pages": {
"66984": {
"pageid": 66984,
"ns": 0,
"title": "Main Page",
"touched": "2012-11-23T06:44:22Z",
"lastrevid": 1347044,
"counter": "",
"length": 28,
"redirect": "",
"starttimestamp": "2012-12-15T05:21:21Z",
"edittoken": "bd7d4a61cc4ce6489e68c21259e6e416+\\"
}
}
}
}`
so you can use pageids[0] to access the continually changing number, which will likely make things easier.

Related

How to get the number of items from a structure sub field using reflect in go

I have some issues when getting the number of items from a sub field in a slice struct through reflect package.
This is how I'm trying to get the number of items from Items
func main() {
type Items struct {
Name string `json:"name"`
Present bool `json:"present"`
}
type someStuff struct {
Fields string `json:"fields"`
Items []Items `json:"items"`
}
type Stuff struct {
Stuff []someStuff `json:"stuff"`
}
some_stuff := `{
"stuff": [
{
"fields": "example",
"items": [
{ "name": "book01", "present": true },
{ "name": "book02", "present": true },
{ "name": "book03", "present": true }
]
}
]
}`
var variable Stuff
err := json.Unmarshal([]byte(some_stuff), &variable)
if err != nil {
panic(err)
}
//I want to get the number of items in my case 3
NumItems := reflect.ValueOf(variable.Stuff.Items)
}
This is the error:
variable.Items undefined (type []Stuff has no field or method Items)
I'm unsure if I can retrieve the number of items like that.
I have already fixed the issue.
In order to get the number of sub fields we can make use of Len() from reflect.ValueOf.
The code now is getting the number of Items:
package main
import (
"encoding/json"
"fmt"
"reflect"
)
func main() {
type Items struct {
Name string `json:"name"`
Present bool `json:"present"`
}
type someStuff struct {
Fields string `json:"fields"`
Items []Items `json:"items"`
}
type Stuff struct {
Stuff []someStuff `json:"stuff"`
}
some_stuff := `{
"stuff": [
{
"fields": "example",
"items": [
{ "name": "book01", "present": true },
{ "name": "book02", "present": true },
{ "name": "book03", "present": true }
]
}
]
}`
var variable Stuff
err := json.Unmarshal([]byte(some_stuff), &variable)
if err != nil {
panic(err)
}
//I want to get the number of items in my case 3
t := reflect.ValueOf(variable.Stuff[0].Items)
fmt.Println(t.Len())
}
Output: 3

Unmarshal Nested JSON Object With Go

I'm making requests to an API that returns a JSON structure that is difficult to parse, especially in Go. Here is the gist of the data in tabular form:
Ticker
Jan 3
Jan 4
Jan 5
Jan 6
AAPL
182.01
179.70
174.92
172
NFLX
597.37
591.15
567.52
553.29
However, the requested data is completely unordered and unnecessarily nested:
{
"records": [
{
"id": "NFLX",
"data": {
"date": "2022-01-04",
"price": 591.15
}
},
{
"id": "AAPL",
"data": {
"date": "2022-01-03",
"price": 182.01
}
},
{
"id": "AAPL",
"data": {
"date": "2022-01-06",
"price": 172
}
},
{
"id": "NFLX",
"data": {
"date": "2022-01-05",
"price": 567.52
}
},
{
"id": "AAPL",
"data": {
"date": "2022-01-05",
"price": 174.92
}
},
{
"id": "NFLX",
"data": {
"date": "2022-01-06",
"price": 553.29
}
},
{
"id": "NFLX",
"data": {
"date": "2022-01-03",
"price": 597.37
}
},
{
"id": "AAPL",
"data": {
"date": "2022-01-04",
"price": 179.7
}
}
]
}
My goal is to create a map using Ticker as the key and a numpy-like array of prices - ordered by descending date - as the value, such as:
[{Ticker:NFLX Prices:[553.29 567.52 591.15 597.37]} {Ticker:AAPL Prices:[172 174.92 179.7 182.01]}]
To accomplish this, I'm creating several intermediate structs, performing small operations on each along the way to get it into this final format. I can't help but to think there is a better solution. But for now, here is my current implementation:
func NewRecord(id string, prices []float64) Record {
return Record{
Ticker: id,
Prices: prices,
}
}
type Record struct {
Ticker string
Prices []float64
}
type Records []Record
func (rs *Records) UnmarshalJSON(bs []byte) error {
type entry struct {
Date string `json:"date"`
Price float64 `json:"price"`
}
type payload struct {
EntrySlice []struct {
Ticker string `json:"id"`
Entry entry `json:"data"`
} `json:"records"`
}
var p payload
if err := json.Unmarshal(bs, &p); err != nil {
return err
}
// Create a DataFrame Like Map Using Ticker as the Key
dataframe := make(map[string][]entry)
for _, x := range p.EntrySlice {
dataframe[x.Ticker] = append(dataframe[x.Ticker], entry{
x.Entry.Date,
x.Entry.Price,
})
}
// Sort Entries (DataFrame Values) By Date Descending (i.e. Most Recent First)
for _, entry := range dataframe {
sort.Slice(entry, func(i, j int) bool { return entry[i].Date > entry[j].Date })
}
// Drop dates to create an array-like structure indexed by ticker
var records Records
for id, data := range dataframe {
var fx []float64
for _, entry := range data {
fx = append(fx, entry.Price)
}
records = append(records, NewRecord(id, fx))
}
*rs = records
return nil
}
This code works. But as someone relatively new to Go, I feel like I'm doing a lot more than what is necessary.
I'd like to learn what others are doing in these situations to find a method that is both idiomatic and parsimonious.
playground: https://play.golang.com/p/p3Ft0Ts4oSN

Unmarshal unknown JSON fields without structs

I am trying to unmarshal a JSON object which has an optional array, I am doing this without an array and this is what I got so far:
import (
"encoding/json"
"fmt"
)
func main() {
jo := `
{
"given_name": "Akshay Raj",
"name": "Akshay",
"country": "New Zealand",
"family_name": "Gollahalli",
"emails": [
"name#example.com"
]
}
`
var raw map[string]interface{}
err := json.Unmarshal([]byte(jo), &raw)
if err != nil {
panic(err)
}
fmt.Println(raw["emails"][0])
}
The emails field might or might not come sometime. I know I can use struct and unmarshal it twice for with and without array. When I try to get the index 0 of raw["emails"][0] I get the following error
invalid operation: raw["emails"][0] (type interface {} does not support indexing)
Is there a way to get the index of the emails field?
Update 1
I can do something like this fmt.Println(raw["emails"].([]interface{})[0]) and it works. Is this the only way?
The easiest way is with a struct. There's no need to unmarshal twice.
type MyStruct struct {
// ... other fields
Emails []string `json:"emails"`
}
This will work, regardless of whether the JSON input contains the emails field. When it is missing, your resulting struct will just have an uninitialized Emails field.
You can use type assertions. The Go tutorial on type assertions is here.
A Go playground link applying type assertions to your problem is here. For ease of reading, that code is replicated below:
package main
import (
"encoding/json"
"fmt"
)
func main() {
jo := `
{
"given_name": "Akshay Raj",
"name": "Akshay",
"country": "New Zealand",
"family_name": "Gollahalli",
"emails": [
"name#example.com"
]
}
`
var raw map[string]interface{}
err := json.Unmarshal([]byte(jo), &raw)
if err != nil {
panic(err)
}
emails, ok := raw["emails"]
if !ok {
panic("do this when no 'emails' key")
}
emailsSlice, ok := emails.([]interface{})
if !ok {
panic("do this when 'emails' value is not a slice")
}
if len(emailsSlice) == 0 {
panic("do this when 'emails' slice is empty")
}
email, ok := (emailsSlice[0]).(string)
if !ok {
panic("do this when 'emails' slice contains non-string")
}
fmt.Println(email)
}
As always you can use additional libraries for work with your json data. For example with gojsonq package it will like so:
package main
import (
"fmt"
"github.com/thedevsaddam/gojsonq"
)
func main() {
json := `
{
"given_name": "Akshay Raj",
"name": "Akshay",
"country": "New Zealand",
"family_name": "Gollahalli",
"emails": [
"name#example.com"
]
}
`
first := gojsonq.New().JSONString(json).Find("emails.[0]")
if first != nil {
fmt.Println(first.(string))
} else {
fmt.Println("There isn't emails")
}
}

golang unmarshal map[string]interface{} to a struct containing an array with meta

I have the following json data coming through an API. I want to unmarshal this data into a different way of structure as it is defined below. How can I do it in an elegant way?
{
"_meta": {
"count": 2,
"total": 2
},
"0": {
"key": "key0",
"name": "name0"
},
"1": {
"key": "key1",
"name": "name1"
},
"2": {
"key": "key2",
"name": "name2"
}
// It goes on..
}
type Data struct {
Meta Meta `json:"_meta,omitempty"`
Elements []Element
}
type Element struct {
Key string
Name string
}
type Meta struct{
Count int
Total int
}
This can be quite tricky because you have a json object that holds everything. So i went with the approach of unmarshalling to map of string to *json.RawMessage and then fixing the struct from there.
To do that you will be using a custom Unmarshaler and the benefit of it is that you delay the actual parsing of the inner messages until you need them.
So for example if your meta field was wrong or the numbers it said didn't match the length of the map-1 you could exit prematurely.
package main
import (
"encoding/json"
"fmt"
)
type jdata map[string]*json.RawMessage
type data struct {
Meta Meta
Elements []Element
}
//Element is a key val assoc
type Element struct {
Key string
Name string
}
//Meta holds counts and total of elems
type Meta struct {
Count int
Total int
}
var datain = []byte(`
{
"_meta": {
"count": 2,
"total": 2
},
"0": {
"key": "key0",
"name": "name0"
},
"1": {
"key": "key1",
"name": "name1"
},
"2": {
"key": "key2",
"name": "name2"
}
}`)
func (d *data) UnmarshalJSON(buf []byte) (err error) {
var (
meta *json.RawMessage
ok bool
)
jdata := new(jdata)
if err = json.Unmarshal(buf, jdata); err != nil {
return
}
if meta, ok = (*jdata)["_meta"]; !ok {
return fmt.Errorf("_meta field not found in JSON")
}
if err = json.Unmarshal(*meta, &d.Meta); err != nil {
return
}
for k, v := range *jdata {
if k == "_meta" {
continue
}
elem := &Element{}
if err = json.Unmarshal(*v, elem); err != nil {
return err
}
d.Elements = append(d.Elements, *elem)
}
return nil
}
func main() {
data := &data{}
if err := data.UnmarshalJSON(datain); err != nil {
panic(err)
}
fmt.Printf("decoded:%v\n", data)
}

Parsing nested JSON in go language

How could I parse and extract certain values from below JSON
Here is the sample JSON response
{
"success":true,
"endpoint":"https://api.abcxyz.com",
"info":{
"Guestconnected":134,
"Guestratio":100000.06963,
"symbol1":{
"code":"NY",
"symbol":"*",
"name":"newyear",
"codev":391.78161,
"symbolAppearsAfter":false,
"local":true
},
"symbol2":{
"code":"HNY",
"symbol":"#",
"name":"HappyNewYear",
"codev":1000000.0960,
"symbolAppearsAfter":true,
"local":false
},
"latest":{
"value":1597509,
"autovalue":"00099cf8da58a36c08f2ef98650ff6043ddfb",
"height":474696,
"time":1499527696
}
},
"Allguest":{
"all":4,
"filtered":4,
"total_invitations":15430,
"sent_invitations":15430,
"final_invitations":0
},
"Guestlist":[
{
"GuestCode":"369AR",
"all":2,
"total_invitations":5430,
"sent_invitations":5430,
"final_invitations":0,
"change":0,
"accounts":0
},
{
"GuestCode":"6POIA96TY",
"all":2,
"total_invitations":10000,
"sent_invitations":10000,
"final_invitations":0,
"change":0,
"accounts":0
}
]
}
My Code is :
package main
import (
"encoding/json"
"fmt"
"log"
"net/http"
)
type object struct {
Success bool `json:"success"`
Endpoint string `json:"endpoint"`
Allguest struct {
All int `json:"all"`
Filtered int `json:"filtered"`
TotalInvitations int `json:"total_invitations"`
SentInvitations int `json:"sent_invitations"`
FinalInvitations int `json:"final_invitations"`
} `json:"Allguest"`
Guestlist []struct {
GuestCode string `json:"GuestCode"`
All int `json:"all"`
TotalInvitations int `json:"total_invitations"`
SentInvitations int `json:"sent_invitations"`
FinalInvitations int `json:"final_invitations"`
Change int `json:"change"`
Accounts int `json:"accounts"`
} `json:"Guestlist"`
}
func main() {
uri := "https://siteurl.com/api?lists=1"
res, err := http.Get(uri)
fmt.Println(uri)
if err != nil {
fmt.Println("Error:", err)
log.Fatal(err)
}
defer res.Body.Close()
var s object
err := json.NewDecoder(res.Body).Decode(&s)
if err != nil {
log.Fatal(err)
fmt.Println("Error:", err)
}
fmt.Println(s.Success)
fmt.Println(s.Allguest.TotalInvitations)
for i := 0; i < 6; i++ {
fmt.Println(s.Guestlist[i].TotalInvitations)
)
}
The problem is :
If the response is null it is giving index out of range error how can we avoid this ?
Applying condition if TotalInvitations value is greater than 100 then save in 100.csv else save in others.csv
If you need only certain entries from JSON you can define a structure with only those fields you are interested in. And if value can be null it is better to declare a pointer in the structure. Please take a look at an example at go playground: https://play.golang.org/p/mEwSXvPg3D
use gjson pkg to get value from json document an simple way
example:
package main
import (
"fmt"
"github.com/tidwall/gjson"
)
func main() {
data := []byte(`{
"success": true,
"endpoint": "https://api.abcxyz.com",
"info": {
"Guestconnected": 134,
"Guestratio": 100000.06963,
"symbol1": {
"code": "NY",
"symbol": "*",
"name": "newyear",
"codev": 391.78161,
"symbolAppearsAfter": false,
"local": true
},
"symbol2": {
"code": "HNY",
"symbol": "#",
"name": "HappyNewYear",
"codev": 1000000.096,
"symbolAppearsAfter": true,
"local": false
},
"latest": {
"value": 1597509,
"autovalue": "00099cf8da58a36c08f2ef98650ff6043ddfb",
"height": 474696,
"time": 1499527696
}
},
"Allguest": {
"all": 4,
"filtered": 4,
"total_invitations": 15430,
"sent_invitations": 15430,
"final_invitations": 0
},
"Guestlist": [
{
"GuestCode": "369AR",
"all": 2,
"total_invitations": 5430,
"sent_invitations": 5430,
"final_invitations": 0,
"change": 0,
"accounts": 0
},
{
"GuestCode": "6POIA96TY",
"all": 2,
"total_invitations": 10000,
"sent_invitations": 10000,
"final_invitations": 0,
"change": 0,
"accounts": 0
}
]
}
`)
r := gjson.GetBytes(data, "Allguest.total_invitations")
fmt.Println(r.Value()) // output 15430
r = gjson.GetBytes(data, "Guestlist.#.total_invitations")
fmt.Println(r.Value()) // output [5430 10000]
}
If the response is null it is giving index out of range error how can
we avoid this?
Unless I'm misunderstanding you, the response being null shouldn't matter. If you receive 2 Guestlist items, you can only loop over those 2, meaning for i := 0; i < 6; i++ is wrong. Change it to for i := 0; i < len(s.Guestlist); i++ or for i := range s.Guestlist.
Applying condition if TotalInvitations value is greater than 100 then
save in 100.csv else save in others.csv
You will probably want the os package's Create function to create a new file for writing or the OpenFile function, the O_CREATE, O_WRONLY, and O_APPEND file-open flags, and you can pass 0755 as the permission mode (or another set of octal permissions if you want to change access of a file).
You can then use csvWriter := csv.NewWriter(csvFile) to wrap the resulting io.Writer (assuming there is no error opening the file). From there, you can just write whatever information you require to the file as CSV records. Don't forget to call csvWriter.Flush to flush the output buffer and csvWriter.Error to check whether there was an error flushing the output buffer. Also don't forget to close the file you opened.

Resources