I have a struct which contains a type based on an enum. I am trying to render it to a user friendly string. Here's minimum viable code:
package main
import (
"fmt"
"gopkg.in/yaml.v3"
)
type Job struct {
Engine Engine `json:"Engine" yaml:"Engine"`
}
//go:generate stringer -type=Engine --trimprefix=Engine
type Engine int
const (
engineUnknown Engine = iota // must be first
EngineDocker
engineDone // must be last
)
func main() {
j := Job{Engine: EngineDocker}
fmt.Printf("%+v\n\n", j)
out, _ := yaml.Marshal(j)
fmt.Println(string(out))
}
Here's the generated code:
// Code generated by "stringer -type=Engine --trimprefix=Engine"; DO NOT EDIT.
package main
import "strconv"
func _() {
// An "invalid array index" compiler error signifies that the constant values have changed.
// Re-run the stringer command to generate them again.
var x [1]struct{}
_ = x[engineUnknown-0]
_ = x[EngineDocker-1]
_ = x[engineDone-2]
}
const _Engine_name = "engineUnknownDockerengineDone"
var _Engine_index = [...]uint8{0, 13, 19, 29}
func (i Engine) String() string {
if i < 0 || i >= Engine(len(_Engine_index)-1) {
return "Engine(" + strconv.FormatInt(int64(i), 10) + ")"
}
return _Engine_name[_Engine_index[i]:_Engine_index[i+1]]
}
Here's the output:
{Engine:1}
Engine: 1
Here's what I'd like the output to be:
{Engine:Docker}
Engine: Docker
I thought the String() in the generated file would accomplish this. Is there any way to do this? Thanks!
yaml marshaler doesn't use String method. Instead YAML uses encoding.TextMarshaler and encoding.TextUnmarshaler interfaces. Actually, all other codec schemes - JSON, XML, TOML, etc. - use those interfaces to read/write the values. So, if you implement those methods for your type, you will receive all other codecs for free.
Here is an example how to make a human-readable encoding for your enum: https://go.dev/play/p/pEcBmAM-oZJ
type Engine int
const (
engineUnknown Engine = iota // must be first
EngineDocker
engineDone // must be last
)
var engineNames []string
var engineNameToValue map[string]Engine
func init() {
engineNames = []string{"Unknown", "Docker"}
engineNameToValue = make(map[string]Engine)
for i, name := range engineNames {
engineNameToValue[strings.ToLower(name)] = Engine(i)
}
}
func (e Engine) String() string {
if e < 0 || int(e) >= len(engineNames) {
panic(fmt.Errorf("Invalid engine code: %d", e))
}
return engineNames[e]
}
func ParseEngine(text string) (Engine, error) {
i, ok := engineNameToValue[strings.ToLower(text)]
if !ok {
return engineUnknown, fmt.Errorf("Invalid engine name: %s", text)
}
return i, nil
}
func (e Engine) MarshalText() ([]byte, error) {
return []byte(e.String()), nil
}
func (e *Engine) UnmarshalText(text []byte) (err error) {
name := string(text)
*e, err = ParseEngine(name)
return
}
How it works:
func main() {
j := Job{Engine: EngineDocker}
fmt.Printf("%#v\n\n", j)
out, err := yaml.Marshal(j)
if err != nil {
panic(err)
}
fmt.Printf("YAML: %s\n", string(out))
var jj Job
err = yaml.Unmarshal(out, &jj)
if err != nil {
panic(err)
}
fmt.Printf("%#v\n\n", jj)
// == JSON ==
out, err = json.Marshal(j)
if err != nil {
panic(err)
}
fmt.Printf("JSON: %s\n", string(out))
var jjs Job
err = json.Unmarshal(out, &jjs)
if err != nil {
panic(err)
}
fmt.Printf("%#v\n\n", jjs)
}
the output
main.Job{Engine:1}
YAML: Engine: Docker
main.Job{Engine:1}
JSON: {"Engine":"Docker"}
main.Job{Engine:1}
See? It writes and reads strings to both YAML and JSON without any extra effort.
I have JSON like this that I need to parse into a golang type:
{
name: "something"
rules: [
{
"itemTypeBasedConditions": [["containsAny", ["first_match", "second_match"]]],
"validity": "INVALID"
}]
}
The problem is that each array of the array in itemTypeBasedConditions contains a mix of strings (always first element) and another array (second element), and I am not sure how to parse all of that into an object that I could then manipulate.
I got to:
type RulesFile struct {
Name string
Rules []RulesItem
}
type RulesItem struct {
itemTypeBasedConditions [][]interface{}
validity bool
}
And then I guess I have to convert elements one by one from interface{} to either string (containsAny) or an array of strings ("first_match", "second_match")
Is there a better way of approaching this JSON parsing?
I would do something like this, you can probably alter this to your needs.
package main
import (
"encoding/json"
"fmt"
"os"
"reflect"
)
type RulesFile struct {
Name string `json:"name"`
Rules []RulesItem `json:"rules"`
}
type RulesItem struct {
ItemTypeBasedConditions [][]Condition `json:"itemTypeBasedConditions"`
Validity bool `json:"validity"`
}
type Condition struct {
Value *string
Array *[]string
}
func (c Condition) String() string {
if c.Value != nil {
return *c.Value
}
return fmt.Sprintf("%v", *c.Array)
}
func (c *Condition) UnmarshalJSON(data []byte) error {
var y interface{}
err := json.Unmarshal(data, &y)
if err != nil {
return err
}
switch reflect.TypeOf(y).String() {
case "string":
val := fmt.Sprintf("%v", y)
c.Value = &val
return nil
case "[]interface {}":
temp := y.([]interface{})
a := make([]string, len(temp))
for i, v := range temp {
a[i] = fmt.Sprint(v)
}
c.Array = &a
return nil
}
return fmt.Errorf("cannot unmarshall into string or []string: %v", y)
}
var input string = `
{
"name": "something",
"rules": [
{
"itemTypeBasedConditions": [["containsAny",["first_match", "second_match"]]],
"validity": false
}
]
}`
func main() {
var ruleFile RulesFile
err := json.Unmarshal([]byte(input), &ruleFile)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
fmt.Printf("%+v\n", ruleFile)
}
You can implement the json.Unmarshaler interface. Have that implementation first unmarshal the json into a slice of json.RawMessage, then, once you've done that, you can unmarshal the individual elements to their corresponding types.
type Cond struct {
Name string
Args []string
}
func (c *Cond) UnmarshalJSON(data []byte) error {
// unmarshal into a slice of raw json
var raw []json.RawMessage
if err := json.Unmarshal(data, &raw); err != nil {
return err
} else if len(raw) != 2 {
return errors.New("unsupported number of elements in condition")
}
// unmarshal the first raw json element into a string
if err := json.Unmarshal(raw[0], &c.Name); err != nil {
return err
}
// unmarshal the second raw json element into a slice of string
return json.Unmarshal(raw[1], &c.Args)
}
https://go.dev/play/p/-tbr73TvX0d
I have a case where I'm I have to support multiple versions. Each one has different data so I create 2 structs. Based on the version I will return 1 of the structs. Once I identify which struct, I then would request the data and Unmarshal into the struct. However, Since that struct satisfies an interface, I dont think the unmarshal is working correctly. I always get the zero value for the sctruct
package main
import (
"encoding/json"
"fmt"
)
// Ten300 ...
type Ten300 struct {
Map string `json:"map"`
Enabled string `json:"enabled"`
}
// Ten400 ...
type Ten400 struct {
Block1 int `json:"block_1"`
Block2 int `json:"block_2"`
}
// NET ...
type NET struct {
CMD Commands
S TenIft
}
// Commands ...
type Commands struct {
Get string
}
// TenIft ...
type TenIft interface {
Get(string) error
}
// Get ...
func (n *Ten300) Get(module string) error {
fmt.Println("Ten300", "Get()")
return nil
}
// Get ...
func (n *Ten400) Get(module string) error {
fmt.Println("Ten400", "Get()")
n.Block2 = 100
return nil
}
// TenGen easy to read fun type
type TenGen func() NET
// VersionSetup is a map of possible version
var VersionSetup = map[string]TenGen{
"3.0.0": func() NET {
return NET{
CMD: Commands{
Get: "getConfig",
},
S: &Ten300{},
}
},
"4.0.0": func() NET {
return NET{
CMD: Commands{
Get: "getSettings",
},
S: &Ten400{},
}
},
}
const input300 = `{
"map": "one",
"enabled": "two"
}`
const input400 = `{
"block_1": 1,
"block_2": 2
}`
// Setup ...
func (n *NET) Setup() error {
// This Switch is just to use hard coded data
var input string
switch n.S.(type) {
case *Ten400:
input = input400
case *Ten300:
input = input300
}
err := json.Unmarshal([]byte(input), n.S)
if err != nil {
fmt.Println("returning err:", err)
return err
}
fmt.Printf("n.s type: %T\nn.s value: %+v\n", n.S, n.S)
n.S.Get("xxx")
return nil
}
func main() {
version := "3.0.0"
if f, ok := VersionSetup[version]; ok {
net := f()
err := net.Setup()
if err != nil {
panic(err)
}
}
version = "4.0.0"
if f, ok := VersionSetup[version]; ok {
net := f()
err := net.Setup()
if err != nil {
panic(err)
}
}
}
Added go-playground ... this seems to work, but is this the best solution?
I try to understand why both functions return the same output.
As far as I understood, the point of omit empty is to not add that key to the result struct.
I wrote this example, I was expecting the first output not to have the "Empty" key, but for some reason its value still shows as 0.
package main
import (
"encoding/json"
"fmt"
"strings"
)
type agentOmitEmpty struct {
Alias string `json:"Alias,omitempty"`
Skilled bool `json:"Skilled,omitempty"`
FinID int32 `json:"FinId,omitempty"`
Empty int `json:"Empty,omitempty"`
}
type agent struct {
Alias string `json:"Alias"`
Skilled bool `json:"Skilled"`
FinID int32 `json:"FinId"`
Empty int `json:"Empty"`
}
func main() {
jsonString := `{
"Alias":"Robert",
"Skilled":true,
"FinId":12345
}`
fmt.Printf("output with omit emtpy: %v\n", withEmpty(strings.NewReader(jsonString)))
// output with omit emtpy: {Robert true 12345 0}
fmt.Printf("output regular: %v\n", withoutEmpty(strings.NewReader(jsonString)))
// output without omit: {Robert true 12345 0}
}
func withEmpty(r *strings.Reader) agentOmitEmpty {
dec := json.NewDecoder(r)
body := agentOmitEmpty{}
err := dec.Decode(&body)
if err != nil {
panic(err)
}
return body
}
func withoutEmpty(r *strings.Reader) agent {
dec := json.NewDecoder(r)
body := agent{}
err := dec.Decode(&body)
if err != nil {
panic(err)
}
return body
}
You need to define Empty as *int so it will be replaced with nil when there is no value. Then it will not be saved in the database.
This question already has answers here:
Unmarshal 2 different structs in a slice
(3 answers)
Closed 3 years ago.
i'm struggling to create a data structure for unmarshal the following json:
{
"asks": [
["2.049720", "183.556", 1576323009],
["2.049750", "555.125", 1576323009],
["2.049760", "393.580", 1576323008],
["2.049980", "206.514", 1576322995]
],
"bids": [
["2.043800", "20.691", 1576322350],
["2.039080", "755.396", 1576323007],
["2.036960", "214.621", 1576323006],
["2.036930", "700.792", 1576322987]
]
}
If I use the following struct with interfaces, there is no problem:
type OrderBook struct {
Asks [][]interface{} `json:"asks"`
Bids [][]interface{} `json:"bids"`
}
But i need a more strict typing, so i've tried with:
type BitfinexOrderBook struct {
Pair string `json:"pair"`
Asks [][]BitfinexOrder `json:"asks"`
Bids [][]BitfinexOrder `json:"bids"`
}
type BitfinexOrder struct {
Price string
Volume string
Timestamp time.Time
}
But unfortunately i had not success.
This is the code that I have used for parse the Kraken API for retrieve the order book:
// loadKrakenOrderBook is delegated to load the data related to pairs info
func loadKrakenOrderBook(data []byte) (datastructure.BitfinexOrderBook, error) {
var err error
// Creating the maps for the JSON data
m := map[string]interface{}{}
var orderbook datastructure.BitfinexOrderBook
// Parsing/Unmarshalling JSON
err = json.Unmarshal(data, &m)
if err != nil {
zap.S().Debugw("Error unmarshalling data: " + err.Error())
return orderbook, err
}
a := reflect.ValueOf(m["result"])
if a.Kind() == reflect.Map {
key := a.MapKeys()[0]
log.Println("KEY: ", key)
strct := a.MapIndex(key)
log.Println("MAP: ", strct)
m, _ := strct.Interface().(map[string]interface{})
log.Println("M: ", m)
data, err := json.Marshal(m)
if err != nil {
zap.S().Warnw("Panic on key: ", key.String(), " ERR: "+err.Error())
return orderbook, err
}
log.Println("DATA: ", string(data))
err = json.Unmarshal(data, &orderbook)
if err != nil {
zap.S().Warnw("Panic on key: ", key.String(), " during unmarshal. ERR: "+err.Error())
return orderbook, err
}
return orderbook, nil
}
return orderbook, errors.New("UNABLE_PARSE_VALUE")
}
The data that i use for test are the following:
{
"error": [],
"result": {
"LINKUSD": {
"asks": [
["2.049720", "183.556", 1576323009],
["2.049750", "555.125", 1576323009],
["2.049760", "393.580", 1576323008],
["2.049980", "206.514", 1576322995]
],
"bids": [
["2.043800", "20.691", 1576322350],
["2.039080", "755.396", 1576323007],
["2.036960", "214.621", 1576323006],
["2.036930", "700.792", 1576322987]
]
}
}
}
EDIT
NOTE: the data that i receive in input is the latest json that i've post, not the array of bids and asks.
I've tried to integrate the solution proposed by #chmike. Unfortunately there is a few preprocessing to be made, cause the data is the latest json that i've post.
So i've changed to code as following in order to extract the json data related to asks and bids.
func order(data []byte) (datastructure.BitfinexOrderBook, error) {
var err error
// Creating the maps for the JSON data
m := map[string]interface{}{}
var orderbook datastructure.BitfinexOrderBook
// var asks datastructure.BitfinexOrder
// var bids datastructure.BitfinexOrder
// Parsing/Unmarshalling JSON
err = json.Unmarshal(data, &m)
if err != nil {
zap.S().Warn("Error unmarshalling data: " + err.Error())
return orderbook, err
}
// Extract the "result" json
a := reflect.ValueOf(m["result"])
if a.Kind() == reflect.Map {
key := a.MapKeys()[0]
log.Println("KEY: ", key)
log.Println()
strct := a.MapIndex(key)
log.Println("MAP: ", strct)
m, _ := strct.Interface().(map[string]interface{})
log.Println("M: ", m)
log.Println("Asks: ", m["asks"])
log.Println("Bids: ", m["bids"])
// Here i retrieve the asks array
asks_data, err := json.Marshal(m["asks"])
log.Println("OK: ", err)
log.Println("ASKS: ", string(asks_data))
var asks datastructure.BitfinexOrder
// here i try to unmarshal the data into the struct
asks, err = UnmarshalJSON(asks_data)
log.Println(err)
log.Println(asks)
}
return orderbook, errors.New("UNABLE_PARSE_VALUE")
}
Unfortunately, i receive the following error:
json: cannot unmarshal array into Go value of type json.Number
As suggested by #Flimzy, you need a custom Unmarshaler. Here it is.
Note that the BitfinexOrderBook definition is slightly different from yours. There was an error in it.
// BitfinexOrderBook is a book of orders.
type BitfinexOrderBook struct {
Asks []BitfinexOrder `json:"asks"`
Bids []BitfinexOrder `json:"bids"`
}
// BitfinexOrder is a bitfinex order.
type BitfinexOrder struct {
Price string
Volume string
Timestamp time.Time
}
// UnmarshalJSON decode a BifinexOrder.
func (b *BitfinexOrder) UnmarshalJSON(data []byte) error {
var packedData []json.Number
err := json.Unmarshal(data, &packedData)
if err != nil {
return err
}
b.Price = packedData[0].String()
b.Volume = packedData[1].String()
t, err := packedData[2].Int64()
if err != nil {
return err
}
b.Timestamp = time.Unix(t, 0)
return nil
}
Note also that this custom unmarshaler function allows you to convert the price or volume to a float, which is probably what you want.
While you can hack your way by using reflex, or maybe even write your own parser, the most efficient way is to implement a json.Unmarshaler.
There are a few problem remaining, though.
You are transforming a json array to the struct, not just interface{} elements in it, so it should be: Asks []BitfinexOrder and Bids []BitfinexOrder.
You need to wrap the struct BitfinexOrderBook to get it work with its data. It is trivial and much simpler than using reflex.
By default, json.Unmarshal unmarshals a json number into a float64, which is not a good thing when parsing timestamp. You can use json.NewDecoder to get a decoder and then use Decoder.UseNumber to force use a string.
For example,
func (bo *BitfinexOrder) UnmarshalJSON(data []byte) error {
dec := json.NewDecoder(bytes.NewReader(data))
dec.UseNumber()
var x []interface{}
err := dec.Decode(&x)
if err != nil {
return errParse(err.Error())
}
if len(x) != 3 {
return errParse("length is not 3")
}
price, ok := x[0].(string)
if !ok {
return errParse("price is not string")
}
volume, ok := x[1].(string)
if !ok {
return errParse("volume is not string")
}
number, ok := x[2].(json.Number)
if !ok {
return errParse("timestamp is not number")
}
tint64, err := strconv.ParseInt(string(number), 10, 64)
if err != nil {
return errParse(fmt.Sprintf("parsing timestamp: %s", err))
}
*bo = BitfinexOrder{
Price: price,
Volume: volume,
Timestamp: time.Unix(tint64, 0),
}
return nil
}
and main func (wrapping the struct):
func main() {
x := struct {
Result struct{ LINKUSD BitfinexOrderBook }
}{}
err := json.Unmarshal(data, &x)
if err != nil {
log.Fatalln(err)
}
bob := x.Result.LINKUSD
fmt.Println(bob)
}
Playground link: https://play.golang.org/p/pC124F-3M_S .
Note: the playground link use a helper function to create errors. Some might argue it is best to name the helper function NewErrInvalidBitfinexOrder or rename the error. That is not the scope of this question and I think for the sake of typing, I will keep the short name for now.