Different data structures coming from a json - go

My question is,
I have a JSON that returns me a list of objects.
But sometimes this list returns me a single object in case the det.
I try to use the pattern below
Det []struct {
NItem string `json:"-nItem"`
Prod struct {
CProd string `json:"cProd"`
CEST string `json:"CEST"`
Cfop string `json:"CFOP"`
UCom string `json:"uCom"`
QCom string `json:"qCom"`
IndTot string `json:"indTot"`
VProd string `json:"vProd"`
CEANTrib string `json:"cEANTrib"`
UTrib string `json:"uTrib"`
NCM string `json:"NCM"`
CEAN string `json:"cEAN"`
XProd string `json:"xProd"`
VUnCom string `json:"vUnCom"`
QTrib string `json:"qTrib"`
VUnTrib string `json:"vUnTrib"`
} `json:"prod"`
Imposto struct {
VTotTrib string `json:"vTotTrib"`
ICMS struct {
ICMS00 struct {
Orig string `json:"orig"`
CST string `json:"CST"`
ModBC string `json:"modBC"`
VBC string `json:"vBC"`
PICMS string `json:"pICMS"`
VICMS string `json:" vICMS"`
} `json:"ICMS00"`
} `json:"ICMS"`
IPI struct {
CEnq string `json:"cEnq"`
IPITrib struct {
CST string `json:"CST"`
VBC string `json:"vBC"`
PIPI string `json:"pIPI"`
VIPI string `json:"vIPI"`
} `json:"IPITrib"`
} `json:"IPI"`
PIS struct {
PISAliq struct {
CST string `json:"CST"`
VBC string `json:"vBC"`
PPIS string `json:"pPIS"`
VPIS string `json:"vPIS"`
} `json:"PISAliq"`
} `json:"PIS"`
COFINS struct {
COFINSAliq struct {
CST string `json:"CST "`
VBC string `json:"vBC"`
PCOFINS string `json:"pCOFINS"`
VCOFINS string `json:"vCOFINS"`
} `json:"COFINSAliq"`
} `json:"COFINS"`
} `json:"imposto"`
} `json:"det"`
but works perfectly, but when the det object receives only element the unmarshal does not work, only works if I remove the slice of struct like that
Det struct {
NItem string `json:"-nItem"`
Prod struct {
CProd string `json:"cProd"`
CEST string `json:"CEST"`
Cfop string `json:"CFOP"`
UCom string `json:"uCom"`
QCom string `json:"qCom"`
IndTot string `json:"indTot"`
VProd string `json:"vProd"`
CEANTrib string `json:"cEANTrib"`
UTrib string `json:"uTrib"`
NCM string `json:"NCM"`
CEAN string `json:"cE
The whole structure looks like this.
SomeStruct struct {
ID string json:"-Id"
Ide struct {...} json:"ide"
Total struct {
ICMSTot struct {...} json:"ICMSTot"
} json:"total"
Cobr struct { Dup struct {....} json:"dup" } json:"cobr"
Versao string json:"-versao"
Emit struct { ... } json:"enderEmit"
Det []struct {
NItem string json:"-nItem"
Prod struct {
CProd string json:"cProd"
CEST string json:"CEST"
Cfop string json:"CFOP"
UCom string json:"uCom"
QCom string json:"qCom"
IndTot string json:"indTot"
VProd string json:"vProd"
CEANTrib string json:"cEANTrib"
UTrib string json:"uTrib"
NCM string json:"NCM"
CEAN string json:" cEAN"
XProd string json:"xProd"
VUnCom string json:"vUnCom"
QTrib string json:"qTrib"
VUnTrib string json:"vUnTrib"
} json:"prod"

Decode the JSON as a slice if the first non-whitespace byte in the input is [. Otherwise, decode the JSON as a struct.
func decode(data []byte) ([]Det, error) {
data = bytes.TrimSpace(data)
if len(data) == 0 {
return nil, errors.New("empty")
}
if data[0] == '[' {
var v []Det
err := json.Unmarshal(data, &v)
return v, err
}
var v [1]Det
err := json.Unmarshal(data, &v[0])
return v[:], err
}
Playground example
If the Det value is a member of a struct, then use RawMessage to capture the JSON and then use the function above.
type SomeStruct struct {
....
DetRaw json.RawMessage `json:"Det"`
Det []Det `json:"-"`
....
}
var v SomeStruct
if err := json.Unmarshal(data, &v); err != nil {
// handle error
}
v.Det, err = decode([]byte(v.DetRaw))
if err != nil {
// handle error
}

Related

Please tell me how to bind multi-array to struct

type _getData struct {
Title string `json:"title" form:"title"`
Date string `json:"date" form:"date"`
Pages []struct {
Order int `json:"order" form:"title"`
Description string `json:"description" form:"description"`
} `json:"pages" form:"pages"`
func CreateDiary(c echo.Context) error {
var getData _getData
c.Bind(&getData)
fmt.Print(getData)
...
Receive the following data through c.FormParams command, please tell me how to bind it to _getData struct,
map[address:[미국 캘리포니아 산타클라라 카운티 쿠퍼티노 ] date:[2021-10-05] location:[37.32779072192643 -122.01981157064436] map_id:[0] pages[0][description]:[123123] pages[0][order]:[0] pages[1][description]:[123123] pages[1][order]:[1] tags[0][id]:[12] tags[0][tag_name]:[sdf] title:[123123]]
I want to get the data of pages as an array, but I am getting []
You can use 3rd party lib.
import "github.com/monoculum/formam/v3"
type MyFormData struct {
Pages []struct {
Order int `formam:"order"`
Description string `formam:"description"`
} `formam:"pages"`
Tags []struct {
TagName string `formam:"tag_name"`
Id string `formam:"id"`
} `formam:"tags"`
Title string `formam:"title"`
}
func HttpHandler(c echo.Context) error {
myFormData := MyFormData{}
form, err := c.FormParams()
if err != nil {
return err
}
dec := formam.NewDecoder(&formam.DecoderOptions{TagName: "formam"})
dec.Decode(form, &myFormData)
return c.JSON(200, myFormData)
}

Function doesn't recognize Yaml nested struct

I have the following struct YAML:
type YamlConfig struct {
Items struct {
RiskyRoles []struct {
Name string `yaml:"name"`
Rules []struct{
Verbs []string `yaml:"verbs"`
ResourceOperator string `yaml:"resourcesOperator"`
Resources []string `yaml:"resources"`
}
} `yaml:"RiskyRoles"`
} `yaml:"Items"`
}
I have a function that parse a YAML file to an object and then I want to send the Rules struct object to a function named DoingStuff(..):
yamlFile, err := ioutil.ReadFile("actionItems.yaml")
if err != nil {
fmt.Printf("Error reading YAML file: %s\n", err)
} else{
var yamlConfig YamlConfig
err = yaml.Unmarshal(yamlFile, &yamlConfig)
if err != nil {
fmt.Printf("Error parsing YAML file: %s\n", err)
}
for _, yamlRole := range yamlConfig.Items.RiskyRoles{
DoingStuff(yamlRole.Rules)
}
}
But inside the function DoingStuff, the struct object Rules is not recognized:
func DoingStuff(yamlRules []struct{}) {
// Not recognize ****
for _, rule := range yamlRules {
fmt.Print(rule.ResourceOperator)
}
}
How can I convert it to:
Rules []struct{
Verbs []string `yaml:"verbs"`
ResourceOperator string `yaml:"resourcesOperator"`
Resources []string `yaml:"resources"`
}
Should I re-declare this struct again?
Or cast using interfaces ?
EDIT:
I added new struct and used it inside the YamlConfig struct but the parse failed to parse the Rules:
type RulesStruct struct {
Rules []struct{
Verbs []string `yaml:"verbs"`
ResourceOperator string `yaml:"resourcesOperator"`
Resources []string `yaml:"resources"`
}
}
type YamlConfig struct {
Items struct {
RiskyRoles []struct {
Name string `yaml:"name"`
Message string `yaml:"message"`
Priority string `yaml:"priority"`
Rules []RulesStruct
} `yaml:"RiskyRoles"`
} `yaml:"Items"`
}
Thanks to #mkporiva help, I changed the structs like that:
type RulesStruct struct {
Verbs []string `yaml:"verbs"`
ResourceOperator string `yaml:"resourcesOperator"`
Resources []string `yaml:"resources"`
}
type YamlConfig struct {
Items struct {
RiskyRoles []struct {
Name string `yaml:"name"`
Message string `yaml:"message"`
Priority string `yaml:"priority"`
Rules []RulesStruct
} `yaml:"RiskyRoles"`
} `yaml:"Items"`
}
Now it works fine.

why the "switch t := policy.Parameter.(type)" does not work with viper.Unmarshal()

I want to unmarshal several types from JSON and use the interface to represent the actual struct that it is different. But when I send the struct as interface{} it converts it to a map. The animal.json is:
"{"type":"cat","policies":[{"name":"kitty","parameter":{"duration":600,"percent":90}}]}"
package main
import (
"reflect"
log "github.com/sirupsen/logrus"
"github.com/spf13/viper"
)
func main() {
var err error
animal := New()
viper.SetConfigType("json")
viper.SetConfigName("animal")
viper.AddConfigPath("~/Desktop/")
viper.AddConfigPath(".")
if err := viper.ReadInConfig(); err != nil {
return
}
if err = viper.Unmarshal(&animal); err != nil {
return
}
for _, policy := range animal.Policies {
log.Info(policy.Name)
log.Info(policy.Parameter)
//INFO[0000] map[duration:600 percent:90]
log.Info(reflect.TypeOf(policy.Parameter))
//INFO[0000] map[string]interface {}, Why is it not an interface{} and how do I get it?
switch t := policy.Parameter.(type) {
//why does the switch not work?
case *CatParameter:
log.Info("cat", t)
case *DogParameter:
log.Info("dog", t)
}
}
}
func New() *Animal {
var animal Animal
animal.Type = "cat"
return &animal
}
type Animal struct {
Type string `json:"type" form:"type"`
Policies []Policy `json:"policies" form:"policies"`
}
type CatParameter struct {
Duration int `json:"duration"`
Percent int `json:"percent"`
}
type DogParameter struct {
Percent int `json:"percent"`
Duration int `json:"duration"`
Operation string `json:"operation"`
}
type Policy struct {
Name string `json:"name"`
Parameter interface{} `json:"parameter"`
}
It's json unmarshal feature
If you use an interface{} as a decoder, the default json object for interface{} is map[string]interface{}
You can see it here:
https://godoc.org/encoding/json#Unmarshal
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
So in t := policy.Parameter.(type), the t is map[string]interface{}
For solving your problem, you can try to define another field to distinguish CatParameter or DogParameter
Maybe:
type Policy struct {
Name string `json:"name"`
Parameter Parameter `json:"parameter"`
}
type Parameter struct {
Name string `json:"name"` // cat or dog
Percent int `json:"percent,omitempty"`
Duration int `json:"duration,omitempty"`
Operation string `json:"operation,omitempty"`
}

How to create structure for nested data set in golang?

New to golang & trying to make a script for making bulk upload to Elasticsearch server. My json data set is something like this...
{
product_displayname: "LG Stylus 2 Plus K535D (16 GB, Brown)",
product_price: "24000.00",
popularity: "0.00",
barcode: "",
exclusive_flag: "0",
product_id: "176982",
product_name: "Stylus 2 Plus K535D (Brown)",
brand_name: "LG",
brand_id: "1",
product_spec : {
display_spec: [{
spec_id: "103",
sdv: "24000",
snv: "24000.0000"
}, {
spec_id: "104",
sdv: "GSM",
snv: "0.0000"
}],
filter_spec: [{
spec_id: "103",
sdv: "24000",
snv: "24000.0000"
}, {
spec_id: "105",
sdv: "Touch Screen",
snv: "0.0000"
}]
}
}
Golang Structure I made(by refering google & other online info) for the above dataset is like this...
type Product struct {
product_displayname string `json:"product_displayname"`
product_price string `json:"product_price"`
popularity string `json:"popularity"`
barcode string `json:"barcode"`
exclusive_flag string `json:"exclusive_flag"`
product_id string `json:"product_id"`
product_name string `json:"product_name"`
brand_name string `json:"brand_name"`
brand_id string `json:"brand_id"`
product_spec
}
type product_spec struct {
display_spec []display_speclist
filter_spec []filter_speclist
}
type display_speclist struct {
spec_id string `json:"spec_id"`
sdv string `json:"sdv"`
snv string `json:"snv"`
}
type filter_speclist struct {
spec_id string `json:"spec_id"`
sdv string `json:"sdv"`
snv string `json:"snv"`
}
But, whenever I'm trying to use above Structure with sample data in my bulk upload script I'm getting following error
github.com/crazyheart/elastic-bulk-upload/main.go:70: syntax error: missing operand
github.com/crazyheart/elastic-bulk-upload/main.go:70: unknown escape sequence
github.com/crazyheart/elastic-bulk-upload/main.go:71: syntax error: non-declaration statement outside function body
I feel like I'm making some mistake in mapping that nested field display_spec & filter_spec in golang structure. But can't able to figure out what it is.
main.go
package main
import (
"fmt"
"golang.org/x/net/context"
"gopkg.in/olivere/elastic.v5"
"strconv"
)
type Product struct {
ProductDisplayname string `json:"product_displayname"`
ProductPrice string `json:"product_price"`
Popularity string `json:"popularity"`
Barcode string `json:"barcode"`
ExclusiveFlag string `json:"exclusive_flag"`
ProductID string `json:"product_id"`
ProductName string `json:"product_name"`
BrandName string `json:"brand_name"`
BrandID string `json:"brand_id"`
ProductSpec struct {
DisplaySpec []struct {
SpecID string `json:"spec_id"`
Sdv string `json:"sdv"`
Snv string `json:"snv"`
} `json:"display_spec"`
FilterSpec []struct {
SpecID string `json:"spec_id"`
Sdv string `json:"sdv"`
Snv string `json:"snv"`
} `json:"filter_spec"`
} `json:"product_spec"`
}
func main() {
// Create a context
ctx := context.Background()
client, err := elastic.NewClient()
if err != nil {
fmt.Println("%v", err)
}
// Bulk upload code
n := 0
for i := 0; i < 1000; i++ {
bulkRequest := client.Bulk()
for j := 0; j < 10000; j++ {
n++
product_data := Product{product_displayname:"LG Stylus 2 Plus K535D (16 GB, Brown)",product_price:"24000.00",popularity:"0.00",barcode:"",exclusive_flag:"0",product_id:"17698276",product_name:"Stylus 2 Plus K535D (Brown)",brand_name:"LG",brand_id:"1",product_spec:{display_spec:[{spec_id:"103",sdv:"24000",snv:"24000.0000"},{spec_id:"104",sdv:"GSM",snv:"0.0000"}],filter_spec:[{spec_id:"103",sdv:"24000",snv:"24000.0000"},{spec_id:"105",sdv:"Touch Screen",snv:"0.0000"}]} }
req := elastic.NewBulkIndexRequest().Index("shopfront").Type("products").Id(strconv.Itoa(n)).Doc(product_data)
bulkRequest = bulkRequest.Add(req)
}
bulkResponse, err := bulkRequest.Do(ctx)
if err != nil {
fmt.Println(err)
}
if bulkResponse != nil {
fmt.Println(bulkResponse)
}
fmt.Println(i)
}
}
Workflow
1.- Validate your json (the one you posted is invalid).
2.- Build a proper struct, you can help yourself using this nice tool.
For your case
The structs appears to be fine except you're not exporting the struct fields by capitalizing the initial letter (Thanks #ANisus).
This (shorted) seems more natural.
type Product struct {
ProductDisplayname string `json:"product_displayname"`
ProductSpec struct {
DisplaySpec []struct {
SpecID string `json:"spec_id"`
Sdv string `json:"sdv"`
Snv string `json:"snv"`
} `json:"display_spec"`
FilterSpec []struct {
SpecID string `json:"spec_id"`
Sdv string `json:"sdv"`
Snv string `json:"snv"`
} `json:"filter_spec"`
} `json:"product_spec"`
}

Unmarshal dynamic XML

I have been using unmarshal without any problems until I came across a situation where the XML tag name is dynamic.
XML Could look like:
<unit_amount_in_cents>
<USD type="integer">4000</USD>
</unit_amount_in_cents>
<setup_fee_in_cents>
<USD type="integer">4000</USD>
</setup_fee_in_cents>
or
<unit_amount_in_cents>
<GBP type="integer">4000</GBP>
</unit_amount_in_cents>
<setup_fee_in_cents>
<GBP type="integer">4000</GBP>
</setup_fee_in_cents>
or could have both (or more)
<unit_amount_in_cents>
<USD type="integer">4000</USD>
<GBP type="integer">4000</GBP>
</unit_amount_in_cents>
<setup_fee_in_cents>
<USD type="integer">4000</USD>
<GBP type="integer">4000</USD>
</setup_fee_in_cents>
I can marshal to xml w/o problems by assigning the XML.Name.Local to what I need it to be but can't unmarshal it.
Here is what the struct looks like
type Plan struct {
XMLName xml.Name `xml:"plan"`
Name string `xml:"name,omitempty"`
PlanCode string `xml:"plan_code,omitempty"`
Description string `xml:"description,omitempty"`
SuccessUrl string `xml:"success_url,omitempty"`
CancelUrl string `xml:"cancel_url,omitempty"`
DisplayDonationAmounts bool `xml:"display_donation_amounts,omitempty"`
DisplayQuantity bool `xml:"display_quantity,omitempty"`
DisplayPhoneNumber bool `xml:"display_phone_number,omitempty"`
BypassHostedConfirmation bool `xml:"bypass_hosted_confirmation,omitempty"`
UnitName string `xml:"unit_name,omitempty"`
PaymentPageTOSLink string `xml:"payment_page_tos_link,omitempty"`
PlanIntervalLength int `xml:"plan_interval_length,omitempty"`
PlanIntervalUnit string `xml:"plan_interval_unit,omitempty"`
AccountingCode string `xml:"accounting_code,omitempty"`
CreatedAt *time.Time `xml:"created_at,omitempty"`
SetupFeeInCents CurrencyArray `xml:"setup_fee_in_cents,omitempty"`
UnitAmountInCents CurrencyArray `xml:"unit_amount_in_cents,omitempty"`
}
type CurrencyArray struct {
CurrencyList []Currency
}
func (c *CurrencyArray) AddCurrency(currency string, amount int) {
newc := Currency{Amount:fmt.Sprintf("%v",amount)}
newc.XMLName.Local = currency
c.CurrencyList = append(c.CurrencyList, newc)
}
func (c *CurrencyArray) GetCurrencyValue(currency string) (value int, e error) {
for _, v := range c.CurrencyList {
if v.XMLName.Local == currency {
value, e = strconv.Atoi(v.Amount)
return
}
}
e = errors.New(fmt.Sprintf("%s not found",currency))
return
}
type Currency struct {
XMLName xml.Name `xml:""`
Amount string `xml:",chardata"`
}
You need the tag xml:",any" on your CurrencyList field.
http://play.golang.org/p/i23w03z6R4

Resources