I'm making requests to an API that returns a JSON structure that is difficult to parse, especially in Go. Here is the gist of the data in tabular form:
Ticker
Jan 3
Jan 4
Jan 5
Jan 6
AAPL
182.01
179.70
174.92
172
NFLX
597.37
591.15
567.52
553.29
However, the requested data is completely unordered and unnecessarily nested:
{
"records": [
{
"id": "NFLX",
"data": {
"date": "2022-01-04",
"price": 591.15
}
},
{
"id": "AAPL",
"data": {
"date": "2022-01-03",
"price": 182.01
}
},
{
"id": "AAPL",
"data": {
"date": "2022-01-06",
"price": 172
}
},
{
"id": "NFLX",
"data": {
"date": "2022-01-05",
"price": 567.52
}
},
{
"id": "AAPL",
"data": {
"date": "2022-01-05",
"price": 174.92
}
},
{
"id": "NFLX",
"data": {
"date": "2022-01-06",
"price": 553.29
}
},
{
"id": "NFLX",
"data": {
"date": "2022-01-03",
"price": 597.37
}
},
{
"id": "AAPL",
"data": {
"date": "2022-01-04",
"price": 179.7
}
}
]
}
My goal is to create a map using Ticker as the key and a numpy-like array of prices - ordered by descending date - as the value, such as:
[{Ticker:NFLX Prices:[553.29 567.52 591.15 597.37]} {Ticker:AAPL Prices:[172 174.92 179.7 182.01]}]
To accomplish this, I'm creating several intermediate structs, performing small operations on each along the way to get it into this final format. I can't help but to think there is a better solution. But for now, here is my current implementation:
func NewRecord(id string, prices []float64) Record {
return Record{
Ticker: id,
Prices: prices,
}
}
type Record struct {
Ticker string
Prices []float64
}
type Records []Record
func (rs *Records) UnmarshalJSON(bs []byte) error {
type entry struct {
Date string `json:"date"`
Price float64 `json:"price"`
}
type payload struct {
EntrySlice []struct {
Ticker string `json:"id"`
Entry entry `json:"data"`
} `json:"records"`
}
var p payload
if err := json.Unmarshal(bs, &p); err != nil {
return err
}
// Create a DataFrame Like Map Using Ticker as the Key
dataframe := make(map[string][]entry)
for _, x := range p.EntrySlice {
dataframe[x.Ticker] = append(dataframe[x.Ticker], entry{
x.Entry.Date,
x.Entry.Price,
})
}
// Sort Entries (DataFrame Values) By Date Descending (i.e. Most Recent First)
for _, entry := range dataframe {
sort.Slice(entry, func(i, j int) bool { return entry[i].Date > entry[j].Date })
}
// Drop dates to create an array-like structure indexed by ticker
var records Records
for id, data := range dataframe {
var fx []float64
for _, entry := range data {
fx = append(fx, entry.Price)
}
records = append(records, NewRecord(id, fx))
}
*rs = records
return nil
}
This code works. But as someone relatively new to Go, I feel like I'm doing a lot more than what is necessary.
I'd like to learn what others are doing in these situations to find a method that is both idiomatic and parsimonious.
playground: https://play.golang.com/p/p3Ft0Ts4oSN
Related
I have some issues when getting the number of items from a sub field in a slice struct through reflect package.
This is how I'm trying to get the number of items from Items
func main() {
type Items struct {
Name string `json:"name"`
Present bool `json:"present"`
}
type someStuff struct {
Fields string `json:"fields"`
Items []Items `json:"items"`
}
type Stuff struct {
Stuff []someStuff `json:"stuff"`
}
some_stuff := `{
"stuff": [
{
"fields": "example",
"items": [
{ "name": "book01", "present": true },
{ "name": "book02", "present": true },
{ "name": "book03", "present": true }
]
}
]
}`
var variable Stuff
err := json.Unmarshal([]byte(some_stuff), &variable)
if err != nil {
panic(err)
}
//I want to get the number of items in my case 3
NumItems := reflect.ValueOf(variable.Stuff.Items)
}
This is the error:
variable.Items undefined (type []Stuff has no field or method Items)
I'm unsure if I can retrieve the number of items like that.
I have already fixed the issue.
In order to get the number of sub fields we can make use of Len() from reflect.ValueOf.
The code now is getting the number of Items:
package main
import (
"encoding/json"
"fmt"
"reflect"
)
func main() {
type Items struct {
Name string `json:"name"`
Present bool `json:"present"`
}
type someStuff struct {
Fields string `json:"fields"`
Items []Items `json:"items"`
}
type Stuff struct {
Stuff []someStuff `json:"stuff"`
}
some_stuff := `{
"stuff": [
{
"fields": "example",
"items": [
{ "name": "book01", "present": true },
{ "name": "book02", "present": true },
{ "name": "book03", "present": true }
]
}
]
}`
var variable Stuff
err := json.Unmarshal([]byte(some_stuff), &variable)
if err != nil {
panic(err)
}
//I want to get the number of items in my case 3
t := reflect.ValueOf(variable.Stuff[0].Items)
fmt.Println(t.Len())
}
Output: 3
package main
import (
"encoding/json"
"fmt"
)
type employee struct {
Name string `json:"name"`
Id int `json:"id,omitempty"`
ManagerId int `json:"managerid"`
Reporters []employee `json:"reporters,omitempty"`
}
// type employees struct {
// employees []employee `json:"employee"`
// }
type EmployeeList struct {
employees map[int]employee
root employee
}
func NewEmployeeList() *EmployeeList {
var el EmployeeList
el.employees = make(map[int]employee)
return &el
}
func (el *EmployeeList) CreateMap(employeesInfo []employee) {
for _ , emp := range employeesInfo {
e := employee{
Name : emp.Name,
Id: emp.Id,
ManagerId: emp.ManagerId,
}
el.employees[emp.Id] = e
if(emp.ManagerId == 0){
el.root = e
fmt.Println("CreateMap",el.root)
}
}
fmt.Println("CreateMap2",el.root,el.employees)
}
func (el *EmployeeList) getReportersById(empId int) []employee {
reporters := []employee{}
for _ , employee := range el.employees {
if(employee.ManagerId == empId){
reporters = append(reporters, employee)
}
}
return reporters
}
func (el *EmployeeList) maketree(e *employee) {
//e := root
e.Reporters = el.getReportersById(e.Id)
if(true){
fmt.Println("maketree",e.Id,e.Name,e.Reporters)
}
// e.Reporters = reporters
if(len(e.Reporters) == 0){
return
}
for _ , reporterEmployee := range e.Reporters {
el.maketree(&reporterEmployee);
}
}
func (el EmployeeList) print(root employee, level int) {
for i:= 0; i<level;i++ {
fmt.Print("\t");
}
fmt.Println(root.Name);
for _, reporter := range root.Reporters {
el.print(reporter, level + 1)
}
}
func main() {
//1. Read JSON File
myJsonString := `[{ "name": "Rob", "id": 7, "managerid": 3 }, { "name": "Rex", "id": 6, "managerid": 2 }, { "name": "Jake", "id": 5, "managerid": 2 }, { "name": "Paul", "id": 4, "managerid": 1 }, { "name": "Oliver", "id": 3, "managerid": 1 }, { "name": "John", "id": 2, "managerid": 1 }, { "name": "Britney", "id": 1, "managerid": 0 }]`
//2. Create class and sent file data
emplist := NewEmployeeList()
rawEmployeesInfo := []employee{}
_ = json.Unmarshal([]byte(myJsonString),&rawEmployeesInfo);
//fmt.Println(rawEmployeesInfo);
emplist.CreateMap(rawEmployeesInfo);
//fmt.Println(emplist.employees,emplist.root);
fmt.Println("Main1",emplist.root)
emplist.maketree(&emplist.root);
//fmt.Println(emplist.root)
fmt.Println("Main2",emplist.root)
emplist.print(emplist.root,0)
}
I am trying to create a tree from a json to store employee list. The problem is that while creating the tree values are stored correctly but somehow they lose value in the main function when I want to print the hierarchy tree. Can someone please help me why the value is not stored? I have added logs also to check.
Your CreateMap method:
func (el EmployeeList) CreateMap(employeesInfo []employee) {
...
}
should be:
func (el *EmployeeList) CreateMap(employeesInfo []employee) {
//---^
....
}
so that el is a pointer. Otherwise you're operating on a copy of emplist when you say:
emplist.CreateMap(rawEmployeesInfo);
so you do a bunch of work to create your map and then throw it all away.
I have the following json data coming through an API. I want to unmarshal this data into a different way of structure as it is defined below. How can I do it in an elegant way?
{
"_meta": {
"count": 2,
"total": 2
},
"0": {
"key": "key0",
"name": "name0"
},
"1": {
"key": "key1",
"name": "name1"
},
"2": {
"key": "key2",
"name": "name2"
}
// It goes on..
}
type Data struct {
Meta Meta `json:"_meta,omitempty"`
Elements []Element
}
type Element struct {
Key string
Name string
}
type Meta struct{
Count int
Total int
}
This can be quite tricky because you have a json object that holds everything. So i went with the approach of unmarshalling to map of string to *json.RawMessage and then fixing the struct from there.
To do that you will be using a custom Unmarshaler and the benefit of it is that you delay the actual parsing of the inner messages until you need them.
So for example if your meta field was wrong or the numbers it said didn't match the length of the map-1 you could exit prematurely.
package main
import (
"encoding/json"
"fmt"
)
type jdata map[string]*json.RawMessage
type data struct {
Meta Meta
Elements []Element
}
//Element is a key val assoc
type Element struct {
Key string
Name string
}
//Meta holds counts and total of elems
type Meta struct {
Count int
Total int
}
var datain = []byte(`
{
"_meta": {
"count": 2,
"total": 2
},
"0": {
"key": "key0",
"name": "name0"
},
"1": {
"key": "key1",
"name": "name1"
},
"2": {
"key": "key2",
"name": "name2"
}
}`)
func (d *data) UnmarshalJSON(buf []byte) (err error) {
var (
meta *json.RawMessage
ok bool
)
jdata := new(jdata)
if err = json.Unmarshal(buf, jdata); err != nil {
return
}
if meta, ok = (*jdata)["_meta"]; !ok {
return fmt.Errorf("_meta field not found in JSON")
}
if err = json.Unmarshal(*meta, &d.Meta); err != nil {
return
}
for k, v := range *jdata {
if k == "_meta" {
continue
}
elem := &Element{}
if err = json.Unmarshal(*v, elem); err != nil {
return err
}
d.Elements = append(d.Elements, *elem)
}
return nil
}
func main() {
data := &data{}
if err := data.UnmarshalJSON(datain); err != nil {
panic(err)
}
fmt.Printf("decoded:%v\n", data)
}
Following are my 2 structs
type Attempt struct {
StartTime string `json:"startTime"`
EndTime string `json:"endTime"`
LastUpdated string `json:"lastUpdated"`
Duration uint32 `json:"duration"`
SparkUser string `json:"sparkUser"`
IsCompleted bool `json:"completed"`
LastUpdatedEpoch int64 `json:"lastUpdatedEpoch"`
StartTimeEpoch int64 `json:"startTimeEpoch"`
EndTimeEpoch int64 `json:"EndTimeEpoch"`
}
type Apps struct {
Id string `json:"id"`
Name string `json:"name"`
Attempts []Attempt `json:"attempts"`
}
The following test parses a json string into this apps := &[]Apps{}. When accessing the members of apps, I am getting the following error
invalid operation: apps[0] (type *[]Apps does not support indexing)
The test
func TestUnmarshalApps(t *testing.T) {
appsJson := `[
{
"id": "app-20161229224238-0001",
"name": "Spark shell",
"attempts": [
{
"startTime": "2016-12-30T03:42:26.828GMT",
"endTime": "2016-12-30T03:50:05.696GMT",
"lastUpdated": "2016-12-30T03:50:05.719GMT",
"duration": 458868,
"sparkUser": "esha",
"completed": true,
"endTimeEpoch": 1483069805696,
"lastUpdatedEpoch": 1483069805719,
"startTimeEpoch": 1483069346828
},
{
"startTime": "2016-12-30T03:42:26.828GMT",
"endTime": "2016-12-30T03:50:05.696GMT",
"lastUpdated": "2016-12-30T03:50:05.719GMT",
"duration": 458868,
"sparkUser": "esha",
"completed": true,
"endTimeEpoch": 1483069805696,
"lastUpdatedEpoch": 1483069805719,
"startTimeEpoch": 1483069346828
}
]
},
{
"id": "app-20161229222707-0000",
"name": "Spark shell",
"attempts": [
{
"startTime": "2016-12-30T03:26:50.679GMT",
"endTime": "2016-12-30T03:38:35.882GMT",
"lastUpdated": "2016-12-30T03:38:36.013GMT",
"duration": 705203,
"sparkUser": "esha",
"completed": true,
"endTimeEpoch": 1483069115882,
"lastUpdatedEpoch": 1483069116013,
"startTimeEpoch": 1483068410679
}
]
}
]`
apps := &[]Apps{}
err := json.Unmarshal([]byte(appsJson), apps)
if err != nil {
t.Fatal(err)
}
if len(*apps) != 2 {
t.Fail()
}
if len(apps[0].Attempts) != 2 {
t.Fail()
}
}
How to access the fields Attempts, Id etc.?
apps := &[]Apps{}
apps has type *[]Apps (pointer to slice of Apps objects).
Are you sure you didn't mean to use the type []*Apps (slice of pointers to Apps objects)?
Assuming *[]Apps really is the type you intended, you'd need to use (*apps)[i] to access every element of apps. That type is also the reason why you also need to use len(*apps) instead of len(apps) (and *apps for pretty much everything actually).
If given the string, from a MediaWiki API request:
str = ` {
"query": {
"pages": {
"66984": {
"pageid": 66984,
"ns": 0,
"title": "Main Page",
"touched": "2012-11-23T06:44:22Z",
"lastrevid": 1347044,
"counter": "",
"length": 28,
"redirect": "",
"starttimestamp": "2012-12-15T05:21:21Z",
"edittoken": "bd7d4a61cc4ce6489e68c21259e6e416+\\"
}
}
}
}`
What can be done to get the edittoken, using Go's json package (keep in mind the 66984 number will continually change)?
When you have a changing key like this the best way to deal with it is with a map. In the example below I've used structs up until the point we reach a changing key. Then I switched to a map format after that. I linked up a working example as well.
http://play.golang.org/p/ny0kyafgYO
package main
import (
"fmt"
"encoding/json"
)
type query struct {
Query struct {
Pages map[string]interface{}
}
}
func main() {
str := `{"query":{"pages":{"66984":{"pageid":66984,"ns":0,"title":"Main Page","touched":"2012-11-23T06:44:22Z","lastrevid":1347044,"counter":"","length":28,"redirect":"","starttimestamp":"2012-12-15T05:21:21Z","edittoken":"bd7d4a61cc4ce6489e68c21259e6e416+\\"}}}}`
q := query{}
err := json.Unmarshal([]byte(str), &q)
if err!=nil {
panic(err)
}
for _, p := range q.Query.Pages {
fmt.Printf("edittoken = %s\n", p.(map[string]interface{})["edittoken"].(string))
}
}
Note that if you use the &indexpageids=true parameter in the API request URL, the result will contain a "pageids" array, like so:
str = ` {
"query": {
"pageids": [
"66984"
],
"pages": {
"66984": {
"pageid": 66984,
"ns": 0,
"title": "Main Page",
"touched": "2012-11-23T06:44:22Z",
"lastrevid": 1347044,
"counter": "",
"length": 28,
"redirect": "",
"starttimestamp": "2012-12-15T05:21:21Z",
"edittoken": "bd7d4a61cc4ce6489e68c21259e6e416+\\"
}
}
}
}`
so you can use pageids[0] to access the continually changing number, which will likely make things easier.