I´m trying to read a json file and parse into jsonObject in my Go class.
The json has a random names and number of elements when I receive it.
For example:
{"707514313":1505680270,"1568212945":1505676950,"732898933":1505681884}
So all the examples that I´ve seen that use an struct to define the interface for the unmarshal, where they put the names of the json values, but in my case I cannot do it since I dont know how many and the name of the values of the json.
var settings struct {
Name1 string `json:"707514313"`
Name2 string `json:"1568212945"`
Who knows how many more and with which names?!
}
So I end up unmarshalling with the default interface
func loadFileToJson(filename string) {
plan, _ := ioutil.ReadFile(filename)
var data interface{}
checkError(json.Unmarshal(plan, &data))
fmt.Println("Data %s ", data)
}
That load in data a (map[String]interface{})
Any idea how to achieve what I want.
EDIT:
I create this struct
type Structure struct {
Name map[string]uint64
}
And changing the old default by
var jsonObject []Structure
checkError(json.Unmarshal(plan, &jsonObject))
Is giving me this error
json: cannot unmarshal object into Go value of type []main.StructureData %s []
As #Anzel pointed out your data appears to be perfect for a map[string]uint64. This assumes a couple things, namely that your object keys are always strings (as in your example) and that the values are always uint64 (again as your sample data suggested). As such, unmarshal into that data type instead of interface{}
plan := []byte(`{"707514313":1505680270,"1568212945":1505676950,"732898933":1505681884}`)
var data map[string]uint64
json.Unmarshal(plan, &data)
fmt.Printf("Data is %+v\n", data)
OUTPUT
Data is map[1568212945:1505676950 732898933:1505681884 707514313:1505680270]
As commented, you just need to set the field type as map[string]uint64 and implement a few methods to parse the file and get the map value.
See in this playground for some pseudo code:
playground
However, depending on your map values, you may need to define the field type as map[string]uint64 or whatever reflecting the json structure, e.g. map[string]interface{} or even a separate embedded struct with nested structure.
Hope this helps.
Related
I am connecting to an API which gives a rather large json payload. I need to add a key and value to the root object.Once I do ioutil.Readall from the package "net/http" the JSON is a byte array.
My goal is to just simply add to the structure and move on. As an example, the following the pretty similar to what I am doing: https://tutorialedge.net/golang/consuming-restful-api-with-go/
So how can I simply add to a JSON structure another element (key: value)?
If all you want to do is add a key and value to the root object and produce new JSON, and you don't care about having the data in a structure, you can unmarshal into map[string]interface{}, add your value, and then marshal again:
var m map[string]interface{}
err := json.Unmarshal(data, &m)
m["new_key"] = newValue
newData, err := json.Marshal(m)
(I'm not checking for errors, but you should do that of course.) Take a look at https://golang.org/pkg/encoding/json/ for more information about how to deal with JSON in Go.
Here's how you can do it in an efficient way while preserving the order of keys in the original JSON. The idea is to use json.RawMessage
// original JSON
bytes := []byte(`{"name": "Adele", "age": 24}`)
// let's try to add the kv pair "country": "USA" to it
type Person struct {
Bio json.RawMessage `json:"bio"`
Country string `json:"country"`
}
p := Person{
Bio: json.RawMessage(bytes),
Country: "USA",
}
// ignoring error for brevity
modifiedBytes, _ := json.Marshal(p)
fmt.Println(string(modifiedBytes))
Output:
{"bio":{"name":"Adele","age":24},"country":"USA"}
You can see that the ordering of original JSON is preserved, which wouldn't have been the case if you marshalled the JSON to a map[string]interface{}. This is also more efficient when you're dealing with huge JSONs since there's no reflection involved.
Complete code - https://play.golang.org/p/3hAPVbrAo_w
Since you have a byte data, you need to parse it and store the result in a variable that has your json structure using json.Marshal.
Then after, to add a new key value pair, you can define a new struct with the key and its data type
var variable type1
// Unmarshal the byte data in a variable
json.Unmarshall(data, &variable)
// to add a new key value you can define a new type
type type2 struct {
type1
key type
}
// and you can add
variable2 := type2{variable, newValueToAdd}
While deserializing & reserializing is the more "correct" approach, it may be overkill for just adding a value at the root, which can be done with simple string manipulation (really byte slice manipulation, but the semantics are similar and it's arguably easier):
data := []byte(`{"foo": 1, "bar": true}`)
ins := []byte(`, "baz": "hello"`) // Note the leading comma.
closingBraceIdx := bytes.LastIndexByte(data, '}')
data = append(data[:closingBraceIdx], ins...)
data = append(data, '}')
This is more error-prone, because it is unaware of JSON syntax entirely, but it's safe enough for most cases and for large JSON documents it is more efficient than a parse, insert, and reserialize.
Playground example: https://play.golang.org/p/h8kL4Zzp7rq
if you want to add key-value of json bytes to new json object, you can use json.RawMessage.
type Res struct {
Data interface{}
Message string
}
var row json.RawMessage
row = []byte(`{"Name":"xxx","Sex":1}`)
res := Res{Data:row,Message:"user"}
resBytes ,err := json.Marshal(res)
println(string(resBytes))
//print result："Data":{"Name":"xxx","Sex":1},"Message":"user"}
SJSON package is another way to modify JSON values.
In this example :
json:= `{
"name": {"first":"James","last":"Bond"},
"age' :40,
"license" {"Diamond","Gold","Silver"}
}`
To replace a "Diamond" with "Ultimate"
value, _ := sjson.Set(json, "license.0", "Ultimate")
fmt.Println(value)
json:= `{
"name": {"first":"James","last":"Bond"},
"age' :40,
"license" {"Ultimate","Gold","Silver"}
}`
Or to add a value "Ultimate" in the end:
value, _ := sjson.Set(json, "license.-1", "Ultimate")
fmt.Println(value)
json:= `{
"name": {"first":"James","last":"Bond"},
"age' :40,
"license" {"Diamond","Gold","Silver","Ultimate"}
}`
I'm writing a package in go to send messages between services, using a specific type of transport.
I'd like the package to not understand the type of messages being sent.
My first thought is to serialize the message object into json, send that, deserialize on the receiving end, and pass the go object (as an interface{}) to the subscribing code.
The serialization isn't a problem, but I don't see how the generic package code can deserialize the message since it doesn't know the type. I thought of using reflect TypeOf(), and passing that value as part of the message. But I don't see how to accomplish this since Type is an interface and the implementing rtype is not exported.
If the receiving app gets an interface{}, it is going to have to check the type anyways, so maybe it should just do the deserialization. Or the receiver could provide a reflect Type so the package can deserialize?
Or it could give the receiver a map of field name to value, but I'd prefer the actual type.
Any suggestions?
Let me add an example:
I have a go channel for sending change notifications of different types of objects. Since go doesn't support tagged unions, I define the channel type as:
type UpdateInfo struct {
UpdateType UpdateType
OldObject interface{}
NewObject interface{}
}
The receiving end of the channel gets an UpdateInfo with OldObject and NewObject as the actual concrete object types that were sent.
I want to extend this to work between applications, where the transport will be via a message queue to support pub/sub, multiple consumers, etc.
TL;DR
Just use json.Unmarshal. You can wrap it lightly, using your transport, and call json.Unmarshal (or with a json.Decoder instance, use d.Decode) on your prebuilt JSON bytes and the v interface{} argument from your caller.
Somewhat longer, with an example
Consider how json.Unmarshal does its own magic. Its first argument is the JSON (data []byte), but its second argument is of type interface{}:
func Unmarshal(data []byte, v interface{}) error
As the documentation goes on to say, if v really is just an interface{}:
To unmarshal JSON into an interface value, Unmarshal stores one of these in the interface value:
bool, for JSON booleans
float64, for JSON numbers
string, for JSON strings
[]interface{}, for JSON arrays
map[string]interface{}, for JSON objects
nil for JSON null
but if v has an underlying concrete type, such as type myData struct { ... }, it's much fancier. It only does the above if v's underlying type is interface{}.
Its actual implementation is particularly complex because it's optimized to do the de-JSON-ification and the assignment into the target object at the same time. In principle, though, it's mostly a big type-switch on the underlying (concrete) type of the interface value.
Meanwhile, what you are describing in your question is that you will first deserialize into generic JSON—which really means a variable of type interface{}—and then do your own assignment out of this pre-decoded JSON into another variable of type interface{}, where the type signature of your own decoder would be:
func xxxDecoder(/* maybe some args here, */ v interface{}) error {
var predecoded interface{}
// get some json bytes from somewhere into variable `data`
err := json.Unmarshal(data, &predecoded)
// now emulate json.Unmarshal by getting field names and assigning
... this is the hard part ...
}
and you would then call this code by writing:
type myData struct {
Field1 int `xxx:"Field1"`
Field2 string `xxx:"Field2"`
}
so that you know that JSON object key "Field1" should fill in your Field1 field with an integer, and JSON object key "Field2" should fill in your Field2 field with a string:
func whatever() {
var x myData
err := xxxDecode(..., &x)
if err != nil { ... handle error ... }
... use x.Field1 and x.Field2 ...
}
But this is silly. You can just write:
type myData struct {
Field1 int `json:"Field1"`
Field2 string `json:"Field2"`
}
(or even omit the tags since the field's names are the default json tags), and then do this:
func xxxDecode(..., v interface{}) error {
... get data bytes as before ...
return json.Unmarshal(data, v)
}
In other words, just let json.Unmarshal do all the work by providing json tags in the data structures in question. You still get—and transmit across your special transport—the JSON data bytes from json.Marshal and json.Unmarshal. You do the transmitting and receiving. json.Marshal and json.Unmarshal do all the hard work: you don't have to touch it!
It's still fun to see how Json.Unmarshal works
Jump down to around line 660 of encoding/json/decode.go, where you will find the thing that handles a JSON "object" ({ followed by either } or a string that represents a key), for instance:
func (d *decodeState) object(v reflect.Value) error {
There are some mechanics to handle corner cases (including the fact that v might not be settable and/or might be a pointer that should be followed), then it makes sure that v is either a map[T1]T2 or struct, and if it is a map, that it's suitable—that both T1 and T2 will work when decoding the "key":value items in the object.
If all goes well, it gets into the JSON key-and-value scanning loop starting at line 720 (for {, which will break or return as appropriate). On each trip through this loop, the code reads the JSON key first, leaving the : and value part for later.
If we're decoding into a struct, the decoder now uses the struct's fields—names and json:"..." tags—to find a reflect.Value that we'll use to store right into the field.1 This is subv, found by calling v.Field(i) for the right i, with some slightly complicated goo to handle embedded anonymous structs and pointer-following. The core of this is just subv = v.Field(i), though, where i is whichever field this key names, within the struct. So subv is now a reflect.Value that represents the actual struct instance's value, which we should set once we've decoded the value part of the JSON key-value pair.
If we're decoding into a map, we will decode the value into a temporary first, then store it into the map after decoding. It would be nice to share this with the struct-field storing, but we need a different reflect function to do the store into the map: v.SetMapIndex, where v is the reflect.Value of the map. That's why for a map, subv points to a temporary Elem.
We're now ready to convert the actual value to the target type, so we go back to the JSON bytes and consume the colon : character and read the JSON value. We get the value and store it into our storage location (subv). This is the code starting at line 809 (if destring {). The actual assigning is done through the decoder functions (d.literalStore at line 908, or d.value at line 412) and these actually decode the JSON value while doing the storing. Note that only d.literalStore really stores the value—d.value calls on d.array, d.object, or d.literalStore to do the work recursively if needed.
d.literalStore therefore contains many switch v.Kind()s: it parses a null or a true or false or an integer or a string or an array, then makes sure it can store the resulting value into v.Kind(), and chooses how to store that resulting value into v.Kind() based on the combination of what it just decoded, and the actual v.Kind(). So there's a bit of a combinatorial explosion here, but it gets the job done.
If all that worked, and we're decoding to a map, we may now need to massage the type of the temporary, find the real key, and store the converted value into the map. That's what lines 830 (if v.Kind() == reflect.Map {) through the final close brace at 867 are about.
1To find fields, we first look over at encoding/json/encode.go to find cachedTypeFields. It is a caching version of typeFields. This is where the json tags are found and put into a slice. The result is cached via cachedTypeFields in a map indexed by the reflect-type value of the struct type. So what we get is a slow lookup the first time we use a struct type, then a fast lookup afterwards, to get a slice of information about how to do the decoding. This slice-of-information maps from json-tag-or-field name to: field; type; whether it's a sub-field of an anonymous structure; and so on: everything we will need to know to decode it properly—or to encode it, on the encoding side. (I didn't really look closely at this code.)
You can encode/decode several message on the same buffer, whether that be a "gob" or "json" or some other encoding.
Assuming there's a limited set of concrete types that you want to support, you can always encode a type tag as the first thing, then encode the actual object. This way the decode can decode the type tag first, and depending on its value, decide how to decode the next item.
// encoder side
enc := json.NewEncoder(buffer) // or gob.NewEncoder(buffer)
enc.Encode("player")
enc.Encode(playerInstance)
// decoder side
dec := json.NewDecoder(buffer) // or gob.NewDecoder(buffer)
var tag string
dec.Decode(&tag)
switch tag {
case "player":
var playerInstance Player
dec.Decode(&player)
// do something with it
case "somethingelse":
// decode something else
}
Try dynobuffers instead of struct. It provides get and set by name ability for byte array. Also it is much more faster than json.
I'm using the Golang validate library to do some input error checking as part of an API (a silly demo API for learning purposes).
When one performs the validation a slice of errors is returned. In reality, the slice is made up of the validate library's struct BadField, which looks like this:
type BadField struct {
Field string
Err error
}
func (b BadField) Error() string {
return fmt.Sprintf("field %s is invalid: %v", b.Field, b.Err)
}
I'd like to pass around a more-specific slice, so rather than []error I would like have []BadField so that I can access the Field value.
So far I can't find a way of casting/converting from one to the other. Maybe there isn't one (due to the nature of go and slices). Maybe there's a package that will do this for me.
My initial implementation
The way I've come up with is to loop through the slice and cast each element individually.
errors := valueWithBadStuff.Validate()
validationErrors := make([]validate.BadField, len(errors))
for _, err := range errors {
validationError, ok := err.(validate.BadField)
if !ok {
panic("badarghfiremyeyes") // S/O purposes only
}
validationErrors = append(validationErrors, validationError)
}
Which feels kinda long for something "simple" but perhaps there's a more go idiomatic way? Or a nicer way?
For background, my intention (at the moment) is to take the slice of validation errors and pipe it back to the client as an array of JSON objects with the Field name and the error message (i.e. for a fictional age field: ["field_name": "age", "Cannot be less than 0"])
Just after the loop above I do more conversion to generate a slice of structs that are tagged with json that will actually go the client. The extra conversion may be duplication and pointless but, right now, it's purely a learning piece and I'll probably refactor it in an hour or two.
There's not really a "nicer" way of doing this. To convert a slice you have to basically do what you've already discovered.
If you are simply returning these errors to a client, you could probably avoid the need to typecast this at all.
Implement the JSON Marshaler interface and you can make your type will automatically output the JSON in the format you desire. For example, for the format you gave above this would be:
func (e BadField) MarshalJSON() ([]byte, error) {
return json.Marshal([]string{"field_name",e.Field,e.Err.Error()})
}
I suspect however that you would probably rather have a response something like:
[
{
"field":"age",
"error":"msg1"
},
{
"field":"name",
"error":"msg2"
}
]
To do this, you could simply add the JSON tags to the struct definition, e.g.
type BadField struct {
Field string `json:"field"`
Err error `json:"error"`
}
This would mean calling json.Marshal on a slice of []error which contains BadField instances would result in the JSON above.
It might be helpful to read more about JSON & Go
PS Consider if you want your methods to be value or pointer receivers
I'm following up on Golang Decoding Generic JSON Objects to One of Many Formats as a way to unmarshal generic json. I'm going to have a multitude of different types tho which can be added by others, so hardcoding case statements is not feasible.
I also don't want to hardcode the type as a string, but let the ones using the library chose the "lookup" name, in case they want to rename their underlying structs later.
I am basically looking for something like this:
type myInterface interface {
Something() // irrelevant, just to show you It's not about interface{}
}
type myBar struct {} // fulfils myInterface
type mySomething struct {} // fulfils myInterface
var types = make(map[string]type) // <--- Obvious Pseudo code ;)
types["foo:bar"] = myBar // done by whoever uses the library
types["1230988"] = mySomething // ...
type storageWrapper struct {
Type string
Data json.RawMessage
}
func loadSomething(id string) myInterface {
buf := db.load(id) // pseudo code but you get the idea
sw := &storageWrapper{}
json.Unmarshal(buf, sw)
// now the interesting part
targetType := types[sw.Type]
thing := &targetType{}
json.Unmarshal(sw.Data, thing)
return thing
}
I have this feeling that I'm overthinking the whole Problem. Or that I'm trying to bend Go into something that conflicts with its underlying philosophy. I'm very open and thankful for any advice that suggests a different approach to the whole Problem
Have types be a map[string]myInterface, and to register a type, have callers store an empty value of that type (not a reference) into the map. Then, to unmarshal, you can "get the type" by copying the empty value out of the map, unmarshaling into it, and returning it (or a reference to it). The interface value will do the job of identifying which type is wanted. Plus, if users want to default some fields to non-zero/empty values in case they're not provided in the JSON, they can actually do that by storing those values within the struct in the type map.
I am trying to create a function that takes a []byte and an interface{} (standing for the struct) and returns an interface{} as the struct type passed into the func.
Something like this:
package main
import (
"encoding/json"
)
func UnmarshalFromJSONArray(sms []byte,tt string) (interface{}) {
var ts = new(tt)
err := json.Unmarshal(sms,&ts)
if(err != nil) {
fmt.Println(err)
}
return sms
}
So that method would run something like this:
// let's say a struct has the following definition:
type MyStructType struct {
Id int
Name string
Desc string
}
// we can some how get its fully qualified class name (this may require reflection?) or pass it into the UnMarshal method direction some how.
mst := "package.MyStructType",
// and then assume a byte array ba that has JSON format for
ba := []byte(`{"Id":"3","Name":"Jack","Desc":"the man"}`)
stct := UnmarshalFromJSONArray(ba,mst)
MyStructureType out := stct
// leaving "stct" being the unmarshalled byte array which can be used like any other struct of type "MyStructureType"
The key being that I never need to know what the fields of MyStructureType are before unmarshalling. All I need are the name of the struct and some way to instance one and then populate it with JSON byte array data that matches its fields. Hopefully that is possible (it is trivial in java using reflection). So I want to basically unmarshal an anonymous struct type by it's name without needing to know what fields it has.
Any suggestions?
The short answer is that this is impossible. There is no string to type translator in Go. You can make a map of strings to reflect.Type's, but you would need to know the possible options ahead of time or you need to provide the caller with a way to register types (perhaps in init).
Assuming you have found a way to resolve the string to its reflect.Type, you can simply call reflect.New(typ).Interface() to get the pointer you need to pass to json.Unmarshal().
The best answer is to avoid trying this all together. Writing idiomatic Java in Go isn't really possible. If I knew more about your problem, I could give you a more idiomatic Go solution.