I'm trying to place a json array into a struct from Google Analytics API.
EG:
"rows": [
[
"female",
"18-24",
"1308"
],
[
"female",
"25-34",
"741"
]
]
Typically I'd have key/value so I can put json:"gender" but there are no keys to associate with, so the values that it would search for change.
The struct would be:
type Row struct {
Gender string `json:"gender"`
AgeRange string `json:"blah"`
Count string `json:"blah"`
}
If I do len(jResp.Rows) I can see that it's grabbing all twelve rows/arrays but the fields are empty.
I don't think it's possible, with encoding/json to directly decode that json into a slice of structs without first implementing a UnmarshalJSON method on your Row type.
func (r *Row) UnmarshalJSON(data []byte) error {
var s []string
if err := json.Unmarshal(data, &s); err != nil {
return err
}
if len(s) >= 3 {
r.Gender = s[0]
r.AgeRange = s[1]
r.Count = s[2]
}
return nil
}
// make sure it's a slice of pointers to Row
type Resp struct {
Rows []*Row `json:"rows"`
}
Edit: fixed the code a little to make it actually compile. Heres a working example https://play.golang.org/p/eqVQj65xJv.
You could also just decode the data first into a slice of strings and then loop over the result to build you struct values.
type Resp struct {
Rows [][]string `json:"rows"`
}
type Row struct {
Gender string `json:"gender"`
AgeRange string `json:"blah"`
Count string `json:"blah"`
}
var resp jResp
if err := json.Unmarshal(data, &resp); err != nil {
panic(err)
}
var rows = make([]Row, len(resp.Rows))
for i, r := range resp.Rows {
rows[i] = Row{
Gender: r[0],
AgeRange: r[1],
Count: r[2],
}
}
Edit: fixed this one as well. https://play.golang.org/p/Otb7iULSh3
Related
I'm trying to add string "Employee" to my existing JSON response. Also, we need to be able to generate this version of json based on an user condition. Only if the user condition is met, I need to generate second version of json with string "Employee" added. If not the first version without string "Employee" should be generated. How can I achieve it with out updating the existing struct and how can I check this with if clause to check for the condition and then generate json based on it?
Below is my existing json response in go
[
{
"EmpId":{
"String":"ABCD",
"Valid":true
},
"Department":{
"Float64":0,
"Valid":true
}
}
]
How can I get my json response like below with out changing existing struct based on input parameter?
{
"Employee":[
{
"EmpId":{
"String":"ABCD",
"Valid":true
},
"Department":{
"Float64":0,
"Valid":true
}
}
]
}
Below is my code:
Step 1: model folder
type EmployeeWithRoot struct {
Employee []Employee
}
type Employee struct {
EmpNbr sql.NullString `json:"EmpNbr"`
DateofJoin sql.NullString `json:"DateofJoin"`
DeptId sql.NullString `json:"DeptId"`
DeptName sql.NullString `json:"DeptName"`
}
Step 2: code folder
func GetEmpDetails(logRequestId string, logNestedLevel int, EmpNbr string, DateofJoin string) ([]model.EmployeeWithRoot, error) {
logFunctionFunctionName := "code.GetEmpDetails"
logStartTime := time.Now()
logNestedLevel++
defer configurations.TimeTrack(logFunctionFunctionName, logRequestId, logStartTime, logNestedLevel)
rows, err := db.Query(utils.SELECT_OF_EMP_AGGR, EmpNbr, DateofJoin, DeptId, DeptName)
if err != nil {
return nil, err
}
defer rows.Close()
var e []model.EmployeeWithRoot
for rows.Next() {
var x model.EmployeeWithRoot
err := rows.Scan(&x.Employee.EmpNbr, &x.Employee.DateofJoin, &x.Employee.DeptId,&x.Employee.DeptName)
if err != nil {
return nil, err
}
e = append(e, x)
}
err = rows.Err()
if err != nil {
return nil, err
}
return e, nil
}
STEP 3: API folder
Employee, err := code.GetEmpDetails(logRequestId, logNestedLevel, EmpNbr, DateofJoin)
if err != nil {
log.Panic(err)
}
marshalDataForRequestContentType(logRequestId, logNestedLevel, w, r, Employee)
I'm getting the below error.
x.Employee.EmpNbr undefined (type []model.Employee has no field or method EmpNbr)
x.Employee.DateofJoin undefined (type []model.Employee has no field or method DateofJoin)enter code here
x.Employee.DeptId undefined (type []model.Employee has no field or method DeptId)
x.Employee.DeptName undefined (type []model.Employee has no field or method DeptName)
Considering you're just wrapping it it an outer object, I don't see any reason you'd need to change the existing struct, just wrap it in a new one. I'll have to make some guesses/assumptions here since you've only shown the JSON and not the Go code that produces it, but assuming your existing JSON is produced by marshaling something like var response []Employee, the desired JSON could be produced in your condition by marshaling instead:
json.Marshal(struct{Employee []Employee}{response})
Working example: https://go.dev/play/p/vwDvxnQ96G_2
Use string concatenation:
func addRoot(json string) string {
return `{ "Employee":` + json + `}`
}
Run an example on the GoLang playground.
Here's the code if you are working with []byte instead of string:
func addRoot(json []byte) []byte {
const prefix = `{ "Employee":`
const suffix = `}`
result := make([]byte, 0, len(prefix)+len(json)+len(suffix))
return append(append(append(result, prefix...), json...), suffix...)
}
Run this example on the GoLang playground.
If you have some JSON in a byte slice ([]byte) then you can just add the outer element directly - e.g. (playground):
existingJSON := []byte(`[
{
"EmpId":{
"String":"ABCD",
"Valid":true
},
"Department":{
"Float64":0,
"Valid":true
}
}
]`)
b := bytes.NewBufferString(`{"Employee":`)
b.Write(existingJSON)
b.WriteString(`}`)
fmt.Println(string(b.Bytes()))
If this is not what you are looking for please add further details to your question (ideally your attempt as a minimal, reproducible, example)
I'm making an http request in golang to an external api. It gives a general response of {"error":[]string, "result":changing interface{}}. depending on the function that is making the request, the Result field changes. Since I know the structure of the Result field for each function I run, I want to be able to change the value of Result before unmarshalling to json. I've tried to do this with the following code:
func GetAssets(output *Resp, resultType interface{}) error {
return publicRequest("/Assets", output, resultType)
}
func publicRequest(endPoint string, output *Resp, resultType interface{}) error {
url := Rest_url + Pub_rest_url + endPoint //"https://api.kraken.com/0/public/Assets in this case
resp, err := http.Get(url)
if err != nil {
return err
}
defer resp.Body.Close()
output.Result = resultType
return json.NewDecoder(resp.Body).Decode(&output)
}
Here is how it's being ran in main
type Resp struct {
Error []string `json:"error"`
Result interface{} `json:"result"`
}
type AssetInfo struct {
Aclass string `json:"aclass"`
Altname string `json:"altname"`
Decimals int `json:"decimals"`
Display int `json:"display_decimals"`
}
func main() {
var result map[string]AssetInfo
jsonData := Resp{}
rest_api_client.GetAssets(&jsonData, result)
fmt.Println(jsonData)
}
The issue is that it doesn't unmarshal correctly. A map is created for each asset, but the data contained inside of each asset is also being stored inside of a map. I'm not sure if I explained this well, but here is the current response after unmarshalling to understand what I mean.
Here is the data type of Resp.Result: map[string]interface {}
{[] map[1INCH:map[aclass:currency altname:1INCH decimals:10 display_decimals:5] AAVE:map[aclass:currency altname:AAVE decimals:10 display_decimals:5] ACA:map[aclass:currency altname:ACA decimals:10 display_decimals:5] ADA:map[aclass:currency altname:ADA decimals:8 display_decimals:6]...}
The response type I'm looking for is map[string]AssetInfo. Hopefully it could be unmarshalled like this:
{[] map[1INCH:{currency 1INCH 10 5} AAVE:{currency AAVE 10 5} ACA:{currency ACA 10 5} ADA:{currency ADA 8 6} ADA.S:{currency ADA.S 8 6}...}
Any help? I'd rather keep the Resp struct as generic as possible and just change the value of the Result field (if this is even possible to do correctly) since I plan to have multiple functions that call different endpoints of the api, and they'll all have the same underlying response type of the Resp struct with different Result types
You can view a working example in the following repo:
https://github.com/alessiosavi/GoArbitrage/blob/e107af466852b1ed30c2413eb4401595f7412b4f/markets/kraken/kraken.go
Basically, I've defined the following structure:
type Tickers struct {
Error []interface{} `json:"error"`
Result map[string]Ticker `json:"result"`
}
type Ticker struct {
Aclass string `json:"aclass"`
Altname string `json:"altname"`
Decimals int `json:"decimals"`
DisplayDecimals int `json:"display_decimals"`
}
Than I execute the request in the following way:
const KRAKEN_TICKERS_URL string = `https://api.kraken.com/0/public/Assets`
type Kraken struct {
PairsNames []string `json:"pairs_name"`
Pairs map[string]datastructure.KrakenPair `json:"pairs"`
OrderBook map[string]datastructure.KrakenOrderBook `json:"orderbook"`
MakerFee float64 `json:"maker_fee"`
TakerFees float64 `json:"taker_fee"`
// FeePercent is delegated to save if the fee is in percent or in coin
FeePercent bool `json:"fee_percent"`
Tickers []string
}
// Init is delegated to initialize the maps for the kraken
func (k *Kraken) Init() {
k.Pairs = make(map[string]datastructure.KrakenPair)
k.OrderBook = make(map[string]datastructure.KrakenOrderBook)
k.SetFees()
}
// SetFees is delegated to initialize the fee type/amount for the given market
func (k *Kraken) SetFees() {
k.MakerFee = 0.16
k.TakerFees = 0.26
k.FeePercent = true
}
func (k *Kraken) GetTickers() error {
res := datastructure.Tickers{}
var err error
var request req.Request
var data []byte
var tickers []string
resp := request.SendRequest(KRAKEN_TICKERS_URL, "GET", nil, nil, false, 10*time.Second)
if resp.Error != nil {
zap.S().Debugw("Error during http request. Err: " + resp.Error.Error())
return resp.Error
}
if resp.StatusCode != 200 {
zap.S().Warnw("Received a non 200 status code: " + strconv.Itoa(resp.StatusCode))
return errors.New("NON_200_STATUS_CODE")
}
data = resp.Body
if err = json.Unmarshal(data, &res); err != nil {
zap.S().Warn("ERROR! :" + err.Error())
return err
}
zap.S().Infof("Data: %v", res.Result)
tickers = make([]string, len(res.Result))
i := 0
for key := range res.Result {
tickers[i] = res.Result[key].Altname
i++
}
k.Tickers = tickers
return nil
}
I have JSON like this that I need to parse into a golang type:
{
name: "something"
rules: [
{
"itemTypeBasedConditions": [["containsAny", ["first_match", "second_match"]]],
"validity": "INVALID"
}]
}
The problem is that each array of the array in itemTypeBasedConditions contains a mix of strings (always first element) and another array (second element), and I am not sure how to parse all of that into an object that I could then manipulate.
I got to:
type RulesFile struct {
Name string
Rules []RulesItem
}
type RulesItem struct {
itemTypeBasedConditions [][]interface{}
validity bool
}
And then I guess I have to convert elements one by one from interface{} to either string (containsAny) or an array of strings ("first_match", "second_match")
Is there a better way of approaching this JSON parsing?
I would do something like this, you can probably alter this to your needs.
package main
import (
"encoding/json"
"fmt"
"os"
"reflect"
)
type RulesFile struct {
Name string `json:"name"`
Rules []RulesItem `json:"rules"`
}
type RulesItem struct {
ItemTypeBasedConditions [][]Condition `json:"itemTypeBasedConditions"`
Validity bool `json:"validity"`
}
type Condition struct {
Value *string
Array *[]string
}
func (c Condition) String() string {
if c.Value != nil {
return *c.Value
}
return fmt.Sprintf("%v", *c.Array)
}
func (c *Condition) UnmarshalJSON(data []byte) error {
var y interface{}
err := json.Unmarshal(data, &y)
if err != nil {
return err
}
switch reflect.TypeOf(y).String() {
case "string":
val := fmt.Sprintf("%v", y)
c.Value = &val
return nil
case "[]interface {}":
temp := y.([]interface{})
a := make([]string, len(temp))
for i, v := range temp {
a[i] = fmt.Sprint(v)
}
c.Array = &a
return nil
}
return fmt.Errorf("cannot unmarshall into string or []string: %v", y)
}
var input string = `
{
"name": "something",
"rules": [
{
"itemTypeBasedConditions": [["containsAny",["first_match", "second_match"]]],
"validity": false
}
]
}`
func main() {
var ruleFile RulesFile
err := json.Unmarshal([]byte(input), &ruleFile)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
fmt.Printf("%+v\n", ruleFile)
}
You can implement the json.Unmarshaler interface. Have that implementation first unmarshal the json into a slice of json.RawMessage, then, once you've done that, you can unmarshal the individual elements to their corresponding types.
type Cond struct {
Name string
Args []string
}
func (c *Cond) UnmarshalJSON(data []byte) error {
// unmarshal into a slice of raw json
var raw []json.RawMessage
if err := json.Unmarshal(data, &raw); err != nil {
return err
} else if len(raw) != 2 {
return errors.New("unsupported number of elements in condition")
}
// unmarshal the first raw json element into a string
if err := json.Unmarshal(raw[0], &c.Name); err != nil {
return err
}
// unmarshal the second raw json element into a slice of string
return json.Unmarshal(raw[1], &c.Args)
}
https://go.dev/play/p/-tbr73TvX0d
I defined a struct named Student and a map named score.
Data structure is shown below:
type Student struct {
CountryID int
RegionID int
Name string
}
stu := Student{111, 222, "Tom"}
score := make(map[Student]int64)
score[stu] = 100
i am using json.Marshal to marshal score into json, but i cannot use json.Unmarshal to unmarshal this json. Below is my code. i am using function GetMarshableObject to translate struct Student into string which is marshable.
Could anyone tell me how to deal with this json to unmarshal it back to map score.
package main
import (
"encoding/json"
"fmt"
"os"
"reflect"
)
type Student struct {
CountryID int
RegionID int
Name string
}
func GetMarshableObject(src interface{}) interface{} {
t := reflect.TypeOf(src)
v := reflect.ValueOf(src)
kind := t.Kind()
var result reflect.Value
switch kind {
case reflect.Map:
//Find the map layer count
layer := 0
cur := t.Elem()
for reflect.Map == cur.Kind() {
layer++
cur = cur.Elem()
}
result = reflect.MakeMap(reflect.MapOf(reflect.TypeOf("a"), cur))
for layer > 0 {
result = reflect.MakeMap(reflect.MapOf(reflect.TypeOf("a"), result.Type()))
layer--
}
keys := v.MapKeys()
for _, k := range keys {
value := reflect.ValueOf(GetMarshableObject(v.MapIndex(k).Interface()))
if value.Type() != result.Type().Elem() {
result = reflect.MakeMap(reflect.MapOf(reflect.TypeOf("a"), value.Type()))
}
result.SetMapIndex(reflect.ValueOf(fmt.Sprintf("%v", k)), reflect.ValueOf(GetMarshableObject(v.MapIndex(k).Interface())))
}
default:
result = v
}
return result.Interface()
}
func main() {
stu := Student{111, 222, "Tom"}
score := make(map[Student]int64)
score[stu] = 100
b, err := json.Marshal(GetMarshableObject(score))
if err != nil {
fmt.Println("error:", err)
}
os.Stdout.Write(b) //{"{111 222 Tom}":100}
scoreBak := make(map[Student]int64)
if err = json.Unmarshal(b, &scoreBak); nil != err {
fmt.Println("error: %v", err) // get error here: cannot unmarshal object into Go value of type map[main.Student]int64
}
}
From the docs:
The map's key type must either be a string, an integer type, or
implement encoding.TextMarshaler.
func (s Student) MarshalText() (text []byte, err error) {
type noMethod Student
return json.Marshal(noMethod(s))
}
func (s *Student) UnmarshalText(text []byte) error {
type noMethod Student
return json.Unmarshal(text, (*noMethod)(s))
}
As an example I'm using encoding/json to turn a Student value into a json object key, however that is not required and you can choose your own format.
https://play.golang.org/p/4BgZn4Y37Ww
I'm looking to iterate over the string fields of a struct so I can do some clean-up/validation (with strings.TrimSpace, strings.Trim, etc).
Right now I have a messy switch-case that's not really scalable, and as this isn't in a hot spot of my application (a web form) it seems leveraging reflect is a good choice here.
I'm at a bit of a roadblock for how to implement this however, and the reflect docs are a little confusing to me (I've been digging through some other validation packages, but they're way too heavyweight + I'm using gorilla/schema for the unmarshalling part already):
Iterate over the struct
For each field of type string, apply whatever I need to from the strings package i.e. field = strings.TrimSpace(field)
If there exists a field.Tag.Get("max"), we'll use that value (strconv.Atoi, then unicode.RuneCountInString)
Provide an error slice that's also compatible with the error interface type
type FormError []string
type Listing struct {
Title string `max:"50"`
Location string `max:"100"`
Description string `max:"10000"`
ExpiryDate time.Time
RenderedDesc template.HTML
Contact string `max:"255"`
}
// Iterate over our struct, fix whitespace/formatting where possible
// and return errors encountered
func (l *Listing) Validate() error {
typ := l.Elem().Type()
var invalid FormError
for i = 0; i < typ.NumField(); i++ {
// Iterate over fields
// For StructFields of type string, field = strings.TrimSpace(field)
// if field.Tag.Get("max") != "" {
// check max length/convert to int/utf8.RuneCountInString
if max length exceeded, invalid = append(invalid, "errormsg")
}
if len(invalid) > 0 {
return invalid
}
return nil
}
func (f FormError) Error() string {
var fullError string
for _, v := range f {
fullError =+ v + "\n"
}
return "Errors were encountered during form processing: " + fullError
}
Thanks in advance.
What you want is primarily the methods on reflect.Value called NumFields() int and Field(int). The only thing you're really missing is the string check and SetString method.
package main
import "fmt"
import "reflect"
import "strings"
type MyStruct struct {
A,B,C string
I int
D string
J int
}
func main() {
ms := MyStruct{"Green ", " Eggs", " and ", 2, " Ham ", 15}
// Print it out now so we can see the difference
fmt.Printf("%s%s%s%d%s%d\n", ms.A, ms.B, ms.C, ms.I, ms.D, ms.J)
// We need a pointer so that we can set the value via reflection
msValuePtr := reflect.ValueOf(&ms)
msValue := msValuePtr.Elem()
for i := 0; i < msValue.NumField(); i++ {
field := msValue.Field(i)
// Ignore fields that don't have the same type as a string
if field.Type() != reflect.TypeOf("") {
continue
}
str := field.Interface().(string)
str = strings.TrimSpace(str)
field.SetString(str)
}
fmt.Printf("%s%s%s%d%s%d\n", ms.A, ms.B, ms.C, ms.I, ms.D, ms.J)
}
(Playground link)
There are two caveats here:
You need a pointer to what you're going to change. If you have a value, you'll need to return the modified result.
Attempts to modify unexported fields generally will cause reflect to panic. If you plan on modifying unexported fields, make sure to do this trick inside the package.
This code is rather flexible, you can use switch statements or type switches (on the value returned by field.Interface()) if you need differing behavior depending on the type.
Edit: As for the tag behavior, you seem to already have that figured out. Once you have field and have checked that it's a string, you can just use field.Tag.Get("max") and parse it from there.
Edit2: I made a small error on the tag. Tags are part of the reflect.Type of a struct, so to get them you can use (this is a bit long-winded) msValue.Type().Field(i).Tag.Get("max")
(Playground version of the code you posted in the comments with a working Tag get).
I got beat to the punch, but since I went to the work, here's a solution:
type FormError []*string
type Listing struct {
Title string `max:"50"`
Location string `max:"100"`
Description string `max:"10000"`
ExpiryDate time.Time
RenderedDesc template.HTML
Contact string `max:"255"`
}
// Iterate over our struct, fix whitespace/formatting where possible
// and return errors encountered
func (l *Listing) Validate() error {
listingType := reflect.TypeOf(*l)
listingValue := reflect.ValueOf(l)
listingElem := listingValue.Elem()
var invalid FormError = []*string{}
// Iterate over fields
for i := 0; i < listingElem.NumField(); i++ {
fieldValue := listingElem.Field(i)
// For StructFields of type string, field = strings.TrimSpace(field)
if fieldValue.Type().Name() == "string" {
newFieldValue := strings.TrimSpace(fieldValue.Interface().(string))
fieldValue.SetString(newFieldValue)
fieldType := listingType.Field(i)
maxLengthStr := fieldType.Tag.Get("max")
if maxLengthStr != "" {
maxLength, err := strconv.Atoi(maxLengthStr)
if err != nil {
panic("Field 'max' must be an integer")
}
// check max length/convert to int/utf8.RuneCountInString
if utf8.RuneCountInString(newFieldValue) > maxLength {
// if max length exceeded, invalid = append(invalid, "errormsg")
invalidMessage := `"`+fieldType.Name+`" is too long (max allowed: `+maxLengthStr+`)`
invalid = append(invalid, &invalidMessage)
}
}
}
}
if len(invalid) > 0 {
return invalid
}
return nil
}
func (f FormError) Error() string {
var fullError string
for _, v := range f {
fullError = *v + "\n"
}
return "Errors were encountered during form processing: " + fullError
}
I see you asked about how to do the tags. Reflection has two components: a type and a value. The tag is associated with the type, so you have to get it separately than the field: listingType := reflect.TypeOf(*l). Then you can get the indexed field and the tag from that.
I don't know if it's a good way, but I use it like this.
https://play.golang.org/p/aQ_hG2BYmMD
You can send the address of a struct to this function.
Sorry for My English is not very good.
trimStruct(&someStruct)
func trimStruct(v interface{}) {
bytes, err := json.Marshal(v)
if err != nil {
fmt.Println("[trimStruct] Marshal Error :", err)
}
var mapSI map[string]interface{}
if err := json.Unmarshal(bytes, &mapSI); err != nil {
fmt.Println("[trimStruct] Unmarshal to byte Error :", err)
}
mapSI = trimMapStringInterface(mapSI).(map[string]interface{})
bytes2, err := json.Marshal(mapSI)
if err != nil {
fmt.Println("[trimStruct] Marshal Error :", err)
}
if err := json.Unmarshal(bytes2, v); err != nil {
fmt.Println("[trimStruct] Unmarshal to b Error :", err)
}
}
func trimMapStringInterface(data interface{}) interface{} {
if values, valid := data.([]interface{}); valid {
for i := range values {
data.([]interface{})[i] = trimMapStringInterface(values[i])
}
} else if values, valid := data.(map[string]interface{}); valid {
for k, v := range values {
data.(map[string]interface{})[k] = trimMapStringInterface(v)
}
} else if value, valid := data.(string); valid {
data = strings.TrimSpace(value)
}
return data
}