Marshal into a bson.Raw - go

Using gopkg.in/mgo.v2/bson, I wonder how to marshal an interface{} value into a value of type bson.Raw.
The documentation for bson.Raw states:
Using this type it is possible to unmarshal or marshal values partially.
But I can't find a Marshal function that would return bson.Raw.
What am I missing?
Example of what I try to do:
package main
import (
"fmt"
"gopkg.in/mgo.v2/bson"
)
func main() {
// How to avoid a MarshalRaw help function?
raw, err := MarshalRaw("Hello world")
if err != nil {
panic(err)
}
fmt.Printf("%+v\n", raw)
}
func MarshalRaw(v interface{}) (*bson.Raw, error) {
bin, err := bson.Marshal(struct{ Raw interface{} }{v})
if err != nil {
return nil, err
}
var raw struct{ Raw bson.Raw }
err = bson.Unmarshal(bin, &raw)
if err != nil {
return nil, err
}
return &raw.Raw, nil
}
Output:
&{Kind:2 Data:[12 0 0 0 72 101 108 108 111 32 119 111 114 108 100 0]}

bson.Raw is used as a value both when marshaling and unmarshaling.
To transform an interface{} into a bson.Raw, the first thing to do is to marshal it so that you get the plain document data that represents whatever is being marshaled:
var value interface{} = bson.M{"some": "value"}
data, err := bson.Marshal(value)
if err != nil {
log.Fatal(err)
}
and then it may have one or more fields unmarshaled into bson.Raw values:
var doc struct{ Some bson.Raw }
err = bson.Unmarshal(data, &doc)
if err != nil {
log.Fatal(err)
}
or even the entire document:
var doc bson.Raw
err = bson.Unmarshal(data, &doc)
if err != nil {
log.Fatal(err)
}
If you want the entire document rather than just a field, you can also use this shortcut:
doc := bson.Raw{3, data}
The 3 constant represents a document in the bson specification, and it must of course match the provided data. Since BSON only supports documents at the top level, we know this must be correct.

I believe that bson.Raw is intended to be used as a type for a variable.
for example: (in play)
type Bar struct {
AnInt int
AString bson.Raw
}
The "AString" field will be kept as the bson.Raw struct your link mentions.
This is super-useful if you want to partially decode the top level of a nested structure to figure out its actualy type so you can parse the rest in to the proper datatype.
Note, the above is untested, not in front of a machine I can actually run go on at the moment. This is based on the assumption that it works like the standard encoding/json package.

Solution I was looking for
m := bson.M{"ns": bson.M{"coll": "test1", "db": "testdb"}}
data, _ := bson.Marshal(m)
r := bson.Raw(data)
coll := r.Lookup("ns", "coll").StringValue()
println(coll)
Playground

Related

Unable to Unmarshall to types.Struct with Protobuf

I am working on a legacy Go app that it's driving me crazy in terms of type definition.
At some point I need to return a value with the value types.Struct (from github.com/gogo/protobuf/types).
Here is the signature for the function:
GetDeviceConfig(ctx context.Context, in *DeviceID, opts ...grpc.CallOption) (*types.Struct, error)
I'm getting the value from a Postgres database (with GORM) using a string column called "config". It could be any type of JSON.
Just an example:
{"fields":{"A":{"Kind":{"string_value":"B"}},"C":{"Kind":{"string_value":"D"}}}}
When I try to unmarshall to a map[string]interface{} it works like a charm:
err := client.DB.Where("id = ?", id, 0).First(&device).Error
if err != nil && err != gorm.ErrRecordNotFound {
return nil, err
}
var dat map[string]interface{}
if err := json.Unmarshal([]byte(device.Config), &dat); err != nil {
panic(err)
}
but I need to conver it to types.Struct (valid protobuf signature that cannot be changed). My best guess was something like this:
// convert json to struct
s := types.Struct{}
if err := json.Unmarshal([]byte(device.Config), &s); err != nil {
panic(err)
}
// fmt.Println(s)
but s is empty. No data is populated.
Any help?
In order to unmarshal JSON into a Struct, you need to use Unmarshaler found in the jsonpb package. Something along the following lines should work:
err := jsonpb.UnmarshalString(device.Config, &s)
[...]
You can find further examples in jsonpb_test.go.

GoLang Unmarshal JSON From Elastic Search Result

I have data returned from Elasticsearch, using "github.com/olivere/elastic". That sort of works, when i add it to my struct and string it, like so,
data := Api {
Total: myTotal,
Data: string(result),
}
c.JSON(http.StatusOK, totalData)
the api is a struct like so,
type Api struct {
Total interface{}
Data interface{}
}
This returns data ok, from 1 to any number of results on request. How the results loaded into the data interface are not escaped or something, e.g.
"Data":"{\"CID\":\"XXXXXXXXXX\",\"Link\":\"XXXXXXXXX\",
So I have tried to unmarshal the data before adding it to the api struct.
var p DataApi
err := json.Unmarshal(result, &p)
if err != nil {
panic(err)
}
totalData := Api {
Total: myTotal,
Data: p,
}
c.JSON(http.StatusOK, totalData)
This sort of works fine, returns the data in the correct way, but only when loading one result. When 2 or more results are requested, I get this error from the unmarshal panic
invalid character '{' after top-level value
I have tried and google all over but can not find a solution to this? I am not sure what I am doing wrong? The DataApi is a nested set of structs, I was not sure if there was anything I should be being because of that?
This is being run within the Gin framework.
Thanks.
EDIT
So when I use fmt.Println on the string(result) I can print any number of results on the screen. How can I add this to the API struct and then I need the struct converted into JSON data. Is there some way of appending this string data on the JSON converted API struct?
Try to unmarshal multiple results into a slice:
var q []Api
err = json.Unmarshal(result, &q)
See on playground https://play.golang.org/p/D_bVAd4jBlI
package main
import (
"encoding/json"
"fmt"
)
type Api struct {
Total interface{}
Data interface{}
}
func main() {
data := Api{
Total: 1,
Data: "2",
}
result, err := json.Marshal(data)
if err != nil {
panic(err)
}
fmt.Printf("single data: %s\n", result)
var p Api
err = json.Unmarshal(result, &p)
if err != nil {
panic(err)
}
dataSlice := []Api{data}
result, err = json.Marshal(dataSlice)
if err != nil {
panic(err)
}
fmt.Printf("slice of data: %s\n", result)
var q []Api
err = json.Unmarshal(result, &q)
if err != nil {
panic(err)
}
}
Use json.RawMessage to store arbitrary JSON documents:
var p json.RawMessage
err := json.Unmarshal(result, &p)
if err != nil {
panic(err)
}
totalData := Api {
Total: myTotal,
Data: p,
}
c.JSON(http.StatusOK, totalData)
I have a working solution to my problem, I just use the Hits Hits From the data returned by elastic search, I would like just the source data but i think it does what I need it to do... for now.
Thanks.

Get value of struct with interfaces

I'm trying to parse this petition (https://www.binance.com/api/v1/depth?symbol=MDABTC&limit=500)
I was having tons of problems to create an struct for it, so I used an automated tool, this is what my struct looks like:
type orderBook struct {
Bids [][]interface{} `json:"Bids"`
Asks [][]interface{} `json:"Asks"`
}
I recover and parse the petition by doing:
url := "https://www.binance.com/api/v1/depth?symbol=MDABTC&limit=500"
resp, err := http.Get(url)
if err != nil {
panic(err)
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
panic(err)
}else{
book := orderBook{}
if err := json.Unmarshal(body, &book); err != nil {
panic(err)
}
But whenever I try to make an operation with the struct, like:
v := book.Asks[i][0] * book.Asks[i][1]
I get an error:
invalid operation: book.Asks[i][0] * book.Asks[i][1] (operator * not
defined on interface)
How do I define it? Do I need to create an struct for bids/asks, if so, how would that look like?
Sorry if this seems basic, I just started learning go.
In Golang Spec
For an expression x of interface type and a type T, the primary
expression
x.(T)
asserts that x is not nil and that the value stored in x is of type T.
The notation x.(T) is called a type assertion.
Fetching an underlying value of string type you need to type assert to string from interface.
books.Asks[0][0].(string)
For performing an arithmetic operation on same you needs to convert string into float64 to take into account decimal values
v := strconv.ParseFloat(books.Asks[0][0].(string), 64) * strconv.ParseFloat(books.Asks[0][1].(string), 64)
Checkout code on Go playground
Note that you could define proper structs that unmarshal from the JSON document by implementing the json.Unmarshaler interface.
For example (on the Go Playground):
type OrderBook struct {
Asks, Bids []Order
LastUpdateId int
}
type Order struct {
Price, Volume float64
}
func (o *Order) UnmarshalJSON(bs []byte) error {
values := make([]interface{}, 0, 3)
err := json.Unmarshal(bs, &values)
if err != nil {
return err
}
// TODO: more error checking re: len(values), and their types.
price, err := strconv.ParseFloat(values[0].(string), 10)
if err != nil {
return err
}
volume, err := strconv.ParseFloat(values[1].(string), 10)
if err != nil {
return err
}
*o = Order{price, volume}
return nil
}
As such, unmarshaling those documents looks idiomatic:
func main() {
book := OrderBook{}
err := json.Unmarshal([]byte(jsonstr), &book)
if err != nil {
panic(err)
}
fmt.Printf("Asks: %#v\n", book.Asks)
fmt.Printf("Bids: %#v\n", book.Bids)
fmt.Printf("Update: %#v\n", book.LastUpdateId)
// Asks: []main.Order{main.Order{Price:0.00013186, Volume:167}, main.Order{Price:0.00013187, Volume:128}, ...
// Bids: []main.Order{main.Order{Price:0.00013181, Volume:110}, main.Order{Price:0.00013127, Volume:502}, ...
// Update: 14069188
}

Receiving an empty map with no errors when trying to unmarshal a string read from a file (Answer: unmarshal into the data structure itself)

I have a file, 'test.txt', containing the following data. This file was created from the same structures from the code below using marshaling.
{"VLETXGJM":{"attrib1":"test1","attrib2":"test2"}}
I am trying to read it back from the file and unmarshal it into a map using the same structures. I can successfully read the data from the file. I receive no errors when I try to unmarshal it into the map. However, my map is empty.
The mutex is used to protect the map since my real implementation (this is an extracted test) needs to use a protected map for concurrency. I have tried this same code removing the sync library and received the same negative result.
The test code:
package main
import (
"encoding/json"
"fmt"
"sync"
"os"
)
type TestObject struct {
Attrib1 string `json:"attrib1"`
Attrib2 string `json:"attrib2"`
}
type TestMap map[string]TestObject
type TestList struct {
sync.RWMutex
list TestMap
}
func main() {
tl := TestList{ list: make(TestMap) }
// Read the list back out of the file
fi, err := os.Open("test.txt")
if os.IsNotExist(err) {
fmt.Println("data file does not exist")
panic(nil)
}
if err != nil {
panic(err)
}
defer func() {
if err := fi.Close(); err != nil {
panic(err)
}
}()
data := make([]byte, 1024 * 1024)
count, err := fi.Read(data)
if err != nil {
panic(err)
}
fmt.Printf("read from file: \"%s\"\n",data[:count])
tl.Lock()
err = json.Unmarshal(data[:count], &tl)
if err != nil {
panic(err)
}
tl.Unlock()
// List it out
tl.Lock()
if len(tl.list) == 0 {
fmt.Println("Empty list")
} else {
for key, _ := range tl.list {
fmt.Printf("%s: %s\n", tl.list[key].Attrib1, tl.list[key].Attrib2)
}
}
tl.Unlock()
}
The output of the run is:
read from file: "{"VLETXGJM":{"attrib1":"test1","attrib2":"test2"}}"
Empty list
Thank you for your help. I have searched for similar issues and not yet found an exact duplicate of this scenario.
I think you want to unmarshal into tl.list instead of tl:
err = json.Unmarshal(data[:count], &tl.list)
tl has no exported fields, so Unmarshal into tl won't do anything. tl.list (i.e., type TestMap) matches your data.

Converting struct with embedded interface into JSON

I have a struct that I want to marshal to JSON. It has a defined field called Foo (exported as foo) and a data interface field to which I want to pass a dynamic struct with additional JSON fields.
However when the data field is an interface instead of the specific struct it never gets exported as JSON. How can I make this work?
package main
import (
"encoding/json"
"fmt"
)
type data interface{}
type foo struct {
Foo string `json:"foo,omitempty"`
data
}
type bar struct {
Bar string `json:"bar,omitempty"`
}
func main() {
b := bar{"bar"}
f := foo{"foo", b}
byt, err := json.Marshal(f)
if err != nil {
fmt.Println(err)
return
}
fmt.Println(string(byt))
}
I need to output to look like this (it needs to be flat, not nested):
{"foo": "foo", "bar": "bar"}
You could do this with a custom json.Marshaler implementation and a little bit of byte slicing.
func (f foo) MarshalJSON() ([]byte, error) {
type goo foo
g := goo(f)
b1, err := json.Marshal(g)
if err != nil {
return nil, err
}
b2, err := json.Marshal(g.data)
if err != nil {
return nil, err
}
s1 := string(b1[:len(b1)-1])
s2 := string(b2[1:])
return []byte(s1 + ", " + s2), nil
}
https://play.golang.org/p/NYTNWIL-xu
Please note that this is not checking whether the bytes can actually be sliced and it does also not consider the possible case of the data field being a slice or an array, which i'm unsure how you would want that flattened anyway.
I would write a custom marshaller, like so:
func (f foo) MarshalJSON() ([]byte, error) {
type tmp foo
g := tmp(f)
first, err := json.Marshal(g)
if err != nil {
return nil, err
}
second, err := json.Marshal(f.data)
if err != nil {
return nil, err
}
data := make(map[string]interface{})
json.Unmarshal(first, &data)
json.Unmarshal(second, &data)
return json.Marshal(data)
//{"bar":"bar","foo":"foo"}
}
https://play.golang.org/p/TENiCe9nR0
2 options:
Set to it type json.RawMessage so it won’t be decoded automatically and left as an interface. https://golang.org/pkg/encoding/json/#RawMessage
Write custom unmarshaler on the structure.

Resources