I am connecting to an API which gives a rather large json payload. I need to add a key and value to the root object.Once I do ioutil.Readall from the package "net/http" the JSON is a byte array.
My goal is to just simply add to the structure and move on. As an example, the following the pretty similar to what I am doing: https://tutorialedge.net/golang/consuming-restful-api-with-go/
So how can I simply add to a JSON structure another element (key: value)?
If all you want to do is add a key and value to the root object and produce new JSON, and you don't care about having the data in a structure, you can unmarshal into map[string]interface{}, add your value, and then marshal again:
var m map[string]interface{}
err := json.Unmarshal(data, &m)
m["new_key"] = newValue
newData, err := json.Marshal(m)
(I'm not checking for errors, but you should do that of course.) Take a look at https://golang.org/pkg/encoding/json/ for more information about how to deal with JSON in Go.
Here's how you can do it in an efficient way while preserving the order of keys in the original JSON. The idea is to use json.RawMessage
// original JSON
bytes := []byte(`{"name": "Adele", "age": 24}`)
// let's try to add the kv pair "country": "USA" to it
type Person struct {
Bio json.RawMessage `json:"bio"`
Country string `json:"country"`
}
p := Person{
Bio: json.RawMessage(bytes),
Country: "USA",
}
// ignoring error for brevity
modifiedBytes, _ := json.Marshal(p)
fmt.Println(string(modifiedBytes))
Output:
{"bio":{"name":"Adele","age":24},"country":"USA"}
You can see that the ordering of original JSON is preserved, which wouldn't have been the case if you marshalled the JSON to a map[string]interface{}. This is also more efficient when you're dealing with huge JSONs since there's no reflection involved.
Complete code - https://play.golang.org/p/3hAPVbrAo_w
Since you have a byte data, you need to parse it and store the result in a variable that has your json structure using json.Marshal.
Then after, to add a new key value pair, you can define a new struct with the key and its data type
var variable type1
// Unmarshal the byte data in a variable
json.Unmarshall(data, &variable)
// to add a new key value you can define a new type
type type2 struct {
type1
key type
}
// and you can add
variable2 := type2{variable, newValueToAdd}
While deserializing & reserializing is the more "correct" approach, it may be overkill for just adding a value at the root, which can be done with simple string manipulation (really byte slice manipulation, but the semantics are similar and it's arguably easier):
data := []byte(`{"foo": 1, "bar": true}`)
ins := []byte(`, "baz": "hello"`) // Note the leading comma.
closingBraceIdx := bytes.LastIndexByte(data, '}')
data = append(data[:closingBraceIdx], ins...)
data = append(data, '}')
This is more error-prone, because it is unaware of JSON syntax entirely, but it's safe enough for most cases and for large JSON documents it is more efficient than a parse, insert, and reserialize.
Playground example: https://play.golang.org/p/h8kL4Zzp7rq
if you want to add key-value of json bytes to new json object, you can use json.RawMessage.
type Res struct {
Data interface{}
Message string
}
var row json.RawMessage
row = []byte(`{"Name":"xxx","Sex":1}`)
res := Res{Data:row,Message:"user"}
resBytes ,err := json.Marshal(res)
println(string(resBytes))
//print result:"Data":{"Name":"xxx","Sex":1},"Message":"user"}
SJSON package is another way to modify JSON values.
In this example :
json:= `{
"name": {"first":"James","last":"Bond"},
"age' :40,
"license" {"Diamond","Gold","Silver"}
}`
To replace a "Diamond" with "Ultimate"
value, _ := sjson.Set(json, "license.0", "Ultimate")
fmt.Println(value)
json:= `{
"name": {"first":"James","last":"Bond"},
"age' :40,
"license" {"Ultimate","Gold","Silver"}
}`
Or to add a value "Ultimate" in the end:
value, _ := sjson.Set(json, "license.-1", "Ultimate")
fmt.Println(value)
json:= `{
"name": {"first":"James","last":"Bond"},
"age' :40,
"license" {"Diamond","Gold","Silver","Ultimate"}
}`
Related
I want to write a mockData method which can accept several types of parameter and return correspond objects based on its json data. The code as below:
func MockData(jsonPath string,v interface{})(interface{},error){
var ret interface{}
data,_ := ioutil.ReadFile(jsonPath)
switch v.(type) {
case Req:
ret = Req{}
fmt.Printf("\n===before Unmarshal==%T===\n",ret)
err = json.Unmarshal(data,&ret)
if err!=nil{...}
fmt.Printf("======after unmarshal===%T\n",ret)
case ...
default:
fmt.Printf("error===not match")
}
return ret,err
}
However, it panics when I use it. The code as below:
func main(){
reqJsonPath := /xx/yy/req.json
obj,err:=test.MockData(jsonFile,Req{})
if err!=nil{...}
require := obj.(Req) //panic cant []interface{} to Req
}
and the output of MockData is:
===before Unmarshal==Req===
======after unmarshal===[]interface{}
the type of object changed after unmarshal. and some more strange is that if I replace:
ret = Req{}
with
ret = &Req{}
the output will be same as below:
===before Unmarshal==*Req===
======after unmarshal===*Req
To reproduce the problem more conveniently I give the Require struct as below:
type Req []*Ele
type Ele struct {
ID int
Level int
}
summary:
Can I achieve expected function which produces different types of objects based on its json and type?
Why does the type of object changed after unmarshal, and why it not changed after I add &?
Can I achieve expected function which produces different types of objects based on its json and type?
func MockData(filename string, v interface{}) (interface{}, error) {
data, _ := ioutil.ReadFile(filename)
switch t := v.(type) {
case Req:
// t at this point is a Req{}
err := json.Unmarshal(data, &t)
return t, err
}
return nil, errors.New("unknown type")
}
I don't really know your motivation why you you need to pass an actual struct rather than a pointer. Check this demonstration
Why does the type of object changed after unmarshal, and why it not changed after I add &?
When you unmarshal using &ret where ret is an interface, you are getting the address of the interface. Hence, json.Unmarshal() will see that the backing data is a interface rather than a pointer to a struct. The default data type that json.Unmarshal() will use is map[string]interface{} for objects and []interface{} for arrays.
Now if you unmarshal using ret where ret is &Req{}, json.Unmarshal() will check that the backing data is a struct, hence it can do it's unmarshaling using the struct's fields.
Edit:
You seem to be confused by pointer to an interface which is different to an interface which has a pointer. Try this code and you'll see the difference.
var x interface{} = Req{}
var y interface{} = &x
var z interface{} = &Req{}
fmt.Printf("%T\n", y)
fmt.Printf("%T\n", z)
Remember that interfaces are just normal values and they also take memory. Now if you take an address of that memory, you get the pointer to the interface rather than the pointer to the data the interface is referring to.
Can I achieve expected function which produces different types of objects based on its json and type?
Yes, but you'll have to convert it back using a type assertion at the calling end ie
MyFoo:=MockData("foo.json", Foo{}).(Foo)
(or have multiple return ret.(Foo) return ret.(Bar) in the func)
Why does the type of object changed after unmarshal, and why it not changed after I add &?
There are some helpful comments in the top of the Unmarshal source
namely
// To unmarshal JSON into a pointer, Unmarshal first handles the case of
// the JSON being the JSON literal null. In that case, Unmarshal sets
// the pointer to nil. Otherwise, Unmarshal unmarshals the JSON into
// the value pointed at by the pointer. If the pointer is nil, Unmarshal
// allocates a new value for it to point to.
and
// To unmarshal JSON into an interface value,
// Unmarshal stores one of these in the interface value:
//
// bool, for JSON booleans
// float64, for JSON numbers
// string, for JSON strings
// []interface{}, for JSON arrays
// map[string]interface{}, for JSON objects
// nil for JSON null
So in the first case you are unmarshalling into an interface value (ret is declared as an interface{})
In the second case there is a pointer to a struct so that's what you get
Team,
new to Programming.
I have data available after unmarshaling the Json as shown below, which has nested Key values. flat key values I am able to access, how do I access nested key values.
Here is the byte slice data shown below after unmarshaling —>
tables:[map[name:basic__snatpool_members] map[name:net__snatpool_members] map[name:optimizations__hosts] map[columnNames:[name] name:pool__hosts rows:[map[row:[ry.hj.com]]]] traffic_group:/Common/traffic-group-1
Flat key values I am able to access by using the following code
p.TrafficGroup = m[“traffic_group”].(string)
here is the complete function
func dataToIapp(name string, d *schema.ResourceData) bigip.Iapp {
var p bigip.Iapp
var obj interface{}
jsonblob := []byte(d.Get("jsonfile").(string))
err := json.Unmarshal(jsonblob, &obj)
if err != nil {
fmt.Println("error", err)
}
m := obj.(map[string]interface{}) // Important: to access property
p.Name = m[“name”].(string)
p.Partition = m[“partition”].(string)
p.InheritedDevicegroup = m[“inherited_devicegroup”].(string)
}
Note: This may not work with your JSON structure. I inferred what it would be based on your question but without the actual structure, I cannot guarantee this to work without modification.
If you want to access them in a map, you need to assert that the interface pulled from the first map is actually a map. So you would need to do this:
tmp := m["tables"]
tables, ok := tmp.(map[string]string)
if !ok {
//error handling here
}
r.Name = tables["name"].(string)
But instead of accessing the unmarshaled JSON as a map[string]interface{}, why don't you create structs that match your JSON output?
type JSONRoot struct {
Name string `json:"name"`
Partition string `json:"partition"`
InheritedDevicegroup string `json:"inherited_devicegroup"`
Tables map[string]string `json:"tables"` //Ideally, this would be a map of structs
}
Then in your code:
func dataToIapp(name string, d *schema.ResourceData) bigip.Iapp {
var p bigip.Iapp
var obj &JSONRoot{}
jsonblob := []byte(d.Get("jsonfile").(string))
err := json.Unmarshal(jsonblob, &obj)
if err != nil {
fmt.Println("error", err)
}
p.Name = obj.Name
p.Partition = obj.Partition
p.InheritedDevicegroup = obj.InheritedDevicegroup
p.Name = obj.Tables["name"]
}
JSON objects are unmarshaled into map[string]interface{}, JSON arrays into []interface{}, same applies for nested objects/arrays.
So for example if a key/index maps to a nested object you need to type assert the value to map[string]interface{} and if the key/index maps to an array of objects you first need to assert the value to []interface{} and then each element to map[string]interface{}.
e.g. (for brevity this code is not guarding against panic)
tables := obj.(map[string]interface{})["tables"]
table1 := tables.([]interface{})[0]
name := table1.(map[string]interface{})["name"]
namestr := name.(string)
However, if it's the case that the json you are parsing is not dynamic but instead has a specific structure you should define a struct type that mirrors that structure and unmarshal the json into that.
All you have to do is repeatedly accessing the map via type-switching or assertion:
for _, table := range m["tables"] {
switch val := table {
case string:
fmt.Println("table is string")
case int:
fmt.Println("table is integer")
// This is your case, since JSON is unmarshaled to type []interface{} and map[string]interface{}
case []interface{}:
fmt.Println("table is a slice of interface{}")
for _, tb := range value {
if m, ok := tb.(map[string]interface{}); ok {
// Now it's accessible
fmt.Println(m["name"])
}
}
default:
fmt.Println("unknown type")
}
}
You might want to handle errors better than this.
To read more, check out my writing from a while ago https://medium.com/code-zen/dynamically-creating-instances-from-key-value-pair-map-and-json-in-go-feef83ab9db2.
I´m trying to read a json file and parse into jsonObject in my Go class.
The json has a random names and number of elements when I receive it.
For example:
{"707514313":1505680270,"1568212945":1505676950,"732898933":1505681884}
So all the examples that I´ve seen that use an struct to define the interface for the unmarshal, where they put the names of the json values, but in my case I cannot do it since I dont know how many and the name of the values of the json.
var settings struct {
Name1 string `json:"707514313"`
Name2 string `json:"1568212945"`
Who knows how many more and with which names?!
}
So I end up unmarshalling with the default interface
func loadFileToJson(filename string) {
plan, _ := ioutil.ReadFile(filename)
var data interface{}
checkError(json.Unmarshal(plan, &data))
fmt.Println("Data %s ", data)
}
That load in data a (map[String]interface{})
Any idea how to achieve what I want.
EDIT:
I create this struct
type Structure struct {
Name map[string]uint64
}
And changing the old default by
var jsonObject []Structure
checkError(json.Unmarshal(plan, &jsonObject))
Is giving me this error
json: cannot unmarshal object into Go value of type []main.StructureData %s []
As #Anzel pointed out your data appears to be perfect for a map[string]uint64. This assumes a couple things, namely that your object keys are always strings (as in your example) and that the values are always uint64 (again as your sample data suggested). As such, unmarshal into that data type instead of interface{}
plan := []byte(`{"707514313":1505680270,"1568212945":1505676950,"732898933":1505681884}`)
var data map[string]uint64
json.Unmarshal(plan, &data)
fmt.Printf("Data is %+v\n", data)
OUTPUT
Data is map[1568212945:1505676950 732898933:1505681884 707514313:1505680270]
As commented, you just need to set the field type as map[string]uint64 and implement a few methods to parse the file and get the map value.
See in this playground for some pseudo code:
playground
However, depending on your map values, you may need to define the field type as map[string]uint64 or whatever reflecting the json structure, e.g. map[string]interface{} or even a separate embedded struct with nested structure.
Hope this helps.
I have a listener which receives protobuf messages. However it doesn't know which type of message comes in when. So I tried to unmarshal into an interface{} so I can later type cast:
var data interface{}
err := proto.Unmarshal(message, data)
if err != nil {
log.Fatal("unmarshaling error: ", err)
}
log.Printf("%v\n", data)
However this code doesn't compile:
cannot use data (type interface {}) as type proto.Message in argument to proto.Unmarshal:
interface {} does not implement proto.Message (missing ProtoMessage method)
How can I unmarshal and later type cast an "unknown" protobuf message in go?
First, two words about the OP's question, as presented by them:
proto.Unmarshal can't unmarshal into an interface{}. The method signature is obvious, you must pass a proto.Message argument, which is an interface implemented by concrete protobuffer types.
When handling a raw protobuffer []byte payload that didn't come in an Any, ideally you have at least something (a string, a number, etc...) coming together with the byte slice, that you can use to map to the concrete protobuf message.
You can then switch on that and instantiate the appropriate protobuf concrete type, and only then pass that argument to Unmarshal:
var message proto.Message
switch atLeastSomething {
case "foo":
message = &mypb.Foo{}
case "bar":
message = &mypb.Bar{}
}
_ = proto.Unmarshal(data, message)
Now, what if the byte payload is truly unknown?
As a foreword, consider that this should seldom happen in practice. The schema used to generate the protobuffer types in your language of choice represents a contract, and by accepting protobuffer payloads you are, for some definitions of it, fulfilling that contract.
Anyway, if for some reason you must deal with a completely unknown, mysterious, protobuffer payload in wire format, you can extract some information from it with the protowire package.
Be aware that the wire representation of a protobuf message is ambiguous. A big source of uncertainty is the "length-delimited" type (2) being used for strings, bytes, repeated fields and... sub-messages (reference).
You can retrieve the payload content, but you are bound to have weak semantics.
The code
With that said, this is what a parser for unknown proto messages may look like. The idea is to leverage protowire.ConsumeField to read through the original byte slice.
The data model could be like this:
type Field struct {
Tag Tag
Val Val
}
type Tag struct {
Num int32
Type protowire.Type
}
type Val struct {
Payload interface{}
Length int
}
And the parser:
func parseUnknown(b []byte) []Field {
fields := make([]Field, 0)
for len(b) > 0 {
n, t, fieldlen := protowire.ConsumeField(b)
if fieldlen < 1 {
return nil
}
field := Field{
Tag: Tag{Num: int32(n), Type: t },
}
_, _, taglen := protowire.ConsumeTag(b[:fieldlen])
if taglen < 1 {
return nil
}
var (
v interface{}
vlen int
)
switch t {
case protowire.VarintType:
v, vlen = protowire.ConsumeVarint(b[taglen:fieldlen])
case protowire.Fixed64Type:
v, vlen = protowire.ConsumeFixed64(b[taglen:fieldlen])
case protowire.BytesType:
v, vlen = protowire.ConsumeBytes(b[taglen:fieldlen])
sub := parseUnknown(v.([]byte))
if sub != nil {
v = sub
}
case protowire.StartGroupType:
v, vlen = protowire.ConsumeGroup(n, b[taglen:fieldlen])
sub := parseUnknown(v.([]byte))
if sub != nil {
v = sub
}
case protowire.Fixed32Type:
v, vlen = protowire.ConsumeFixed32(b[taglen:fieldlen])
}
if vlen < 1 {
return nil
}
field.Val = Val{Payload: v, Length: vlen - taglen}
// fmt.Printf("%#v\n", field)
fields = append(fields, field)
b = b[fieldlen:]
}
return fields
}
Sample input and output
Given a proto schema like:
message Foo {
string a = 1;
string b = 2;
Bar bar = 3;
}
message Bar {
string c = 1;
}
initialized in Go as:
&test.Foo{A: "A", B: "B", Bar: &test.Bar{C: "C"}}
And by adding a fmt.Printf("%#v\n", field) statement at the end of the loop in the above code, it will output the following:
main.Field{Tag:main.Tag{Num:1, Type:2}, Val:main.Val{Payload:[]uint8{0x41}, Length:1}}
main.Field{Tag:main.Tag{Num:2, Type:2}, Val:main.Val{Payload:[]uint8{0x42}, Length:1}}
main.Field{Tag:main.Tag{Num:1, Type:2}, Val:main.Val{Payload:[]uint8{0x43}, Length:1}}
main.Field{Tag:main.Tag{Num:3, Type:2}, Val:main.Val{Payload:[]main.Field{main.Field{Tag:main.Tag{Num:1, Type:2}, Val:main.Val{Payload:[]uint8{0x43}, Length:1}}}, Length:3}}
About sub-messages
As you can see from the above the idea to deal with a protowire.BytesType that may or may not be a message field is to attempt to parse it, recursively. If it succeeds, we keep the resulting msg and store it in the field value, if it fails, we store the bytes as-is, which then may be a proto string or bytes. BTW, if I'm reading correctly, this seems what Marc Gravell does in the Protogen code.
About repeated fields
The code above doesn't deal with repeated fields explicitly, but after the parsing is done, repeated fields will have the same value for Field.Tag.Num. From that, packing the fields into a slice/array should be trivial.
About maps
The code above also doesn't deal with proto maps. I suspect that maps are semantically equivalent to a repeated k/v pair, e.g.:
message Pair {
string key = 1; // or whatever key type
string val = 2; // or whatever val type
}
If my assumption is correct, then maps can be parsed with the given code as sub-messages.
About oneofs
I haven't yet tested this, but I expect that information about the union type are completely lost. The byte payload will contain only the value that was actually set.
But what about Any?
The Any proto type doesn't fit in the picture. Contrary to what it may look like, Any is not analogous to, say, map[string]interface{} for JSON objects. And the reason is simple: Any is a proto message with a very well defined structure, namely (in Go):
type Any struct {
// unexported fields
TypeUrl string // struct tags omitted
Value []byte // struct tags omitted
}
So it is more similar to the implementation of a Go interface{} in that it holds some actual data and that data's type information.
It can hold itself arbitrary proto payloads (with their type information!) but it can not be used to decode unknown messages, because Any has exactly those two fields, type url and a byte payload.
To wrap up, this answer doesn't provide a full-blown production-grade solution, but it shows how to decode arbitrary payloads while preserving as much original semantics as possible. Hopefully it will point you in the right direction.
As you've seen, and the commenters have pointed out, you can't use proto.Unmarshal to interface{} because, the method expects a type Message which implements an interface MessageV1.
Protobuf messages are typed and correspond to method invocations ("comes in") and the implementation cannot take generic types of protobuf but specific protobufs:
func (s *server) M(ctx context.Context, _ *pb.Foo) (*pb.Bar, error)
The solution is to envelope your generic types as Any within a specific type perhaps Envelope:
message Envelope {
google.protobuf.Any content = 1;
...
}
The content is then transmitted as a []byte (see Golang anypb.Any) and the implementation (anypb) includes methods to pack|unpack these.
The 'trick' with Any is that messages include a [TypeURL] that uniquely identifies the message so that the receiver knows how to e.g. Unmarshal it.
I'm writing a test that a JSON list is empty.
{"matches": []}
The object has type map[string]interface{}, and I want to test that the list is empty.
var matches := response["matches"]
if len(matches) != 0 {
t.Errorf("Non-empty match list!")
}
However I'm told at compile time that this is invalid
invalid argument matches (type interface {}) for len
If I try casting to a list type:
matches := response["matches"].([]string)
I get a panic:
panic: interface conversion: interface is []interface {}, not []string [recovered]
What do I want to write here?
JSON parsing with maps in Go uses interfaces everywhere. Imagine you have the following JSON object:
{
"stuff" : [
"stuff1",
"stuff2",
"stuff3",
]
}
The Go JSON library will parse the outer object as a map from keys to values, as you've seen in your code. It maps variable names as keys to the values that correspond to those variable names. However, since it has no way of knowing ahead of time what those values are, the value type of the map is simply interface{}. So let's say you know there's a key called "stuff", and you know that its value is an array. You could do:
arr := myMap["stuff"]
And you know that it's an array type, so you can actually instead do:
arr := myMap["stuff"].([]interface{})
the problem here is that while you're right that it's an array, and the JSON library knows this, it has no way of knowing that every element will be of type string, so there's no way for it to decide that the array type should actually be []string. Imagine if you had done this instead:
{
"stuff" : [
"stuff1",
"stuff2",
3
]
}
Well "stuff" can't now be an array of strings because one of the elements isn't a string. In fact, it can't be an array of anything - there's no single type that would satisfy the types of all of the elements. So that's why the Go JSON library has no choice but to leave it as []interface{}. Luckily, since all you want is the length, you're already done. You can just do:
arr := myMap["stuff"].([]interface{})
l := len(arr)
Now that's all fine and good, but let's say that down the road you want to actually look at one of the elements. You could now take out an element and, knowing that it's a string, do:
arr := myMap["stuff"].([]interface{})
iv := arr[0] // interface value
sv := iv.(string) // string value
NOTE
When I say "array," I mean array in the JSON sense - these are JSON arrays. The data structure that represents them in Go is called a "slice" (Go has arrays too, but they're a separate thing - if you're used to arrays in languages like C or Java, Go slices are the closest analogue).
When dealing with JSON, you can add type declarations for array and object, then add methods as needed to help with conversion:
package main
import "encoding/json"
type (
array []interface{}
object map[string]interface{}
)
func (o object) a(s string) array {
return o[s].([]interface{})
}
func main() {
data := []byte(`{"matches": []}`)
var response object
json.Unmarshal(data, &response)
matches := response.a("matches")
mLen := len(matches)
println(mLen == 0)
}
https://golang.org/ref/spec#Type_declarations