I need to write a generic function which can store objects as gobjects.
func hash_store(data map[string]string) {
//initialize a *bytes.Buffer
m := new(bytes.Buffer)
//the *bytes.Buffer satisfies the io.Writer interface and can
//be used in gob.NewEncoder()
enc := gob.NewEncoder(m)
//gob.Encoder has method Encode that accepts data items as parameter
enc.Encode(data)
//the bytes.Buffer type has method Bytes() that returns type []byte,
//and can be used as a parameter in ioutil.WriteFile()
err := ioutil.WriteFile("dep_data", m.Bytes(), 0600)
if err != nil {
panic(err)
}
fmt.Printf("just saved all depinfo with %v\n", data)
n,err := ioutil.ReadFile("dep_data")
if err != nil {
fmt.Printf("cannot read file")
panic(err)
}
//create a bytes.Buffer type with n, type []byte
p := bytes.NewBuffer(n)
//bytes.Buffer satisfies the interface for io.Writer and can be used
//in gob.NewDecoder()
dec := gob.NewDecoder(p)
//make a map reference type that we'll populate with the decoded gob
//e := make(map[int]string)
e := make(map[string]string)
//we must decode into a pointer, so we'll take the address of e
err = dec.Decode(&e)
if err != nil {
fmt.Printf("cannot decode")
panic(err)
}
fmt.Println("after reading dep_data printing ",e)
}
In this function I know the data type to be stored in map[string]string . But I need to write a generic function where I don't know data type and still store it as a gobject in a file.
Change your concrete type (map[string]string) to the empty interface type (interface{}).
See this related question why this works.
Encoding:
func store(data interface{}) {
m := new(bytes.Buffer)
enc := gob.NewEncoder(m)
err := enc.Encode(data)
if err != nil { panic(err) }
err = ioutil.WriteFile("dep_data", m.Bytes(), 0600)
if err != nil { panic(err) }
}
Decoding:
func load(e interface{}) {
n,err := ioutil.ReadFile("dep_data")
if err != nil { panic(err) }
p := bytes.NewBuffer(n)
dec := gob.NewDecoder(p)
err = dec.Decode(e)
if err != nil { panic(err) }
}
The value you put in load must be a pointer of the type you stored in the file using gob.
Example for map[string]string:
org := map[string]string{"foo": "bar"}
store(org)
var loadedMap map[string]string
load(&loadedMap)
fmt.Println(loadedMap["foo"]) // bar
When you encode the data, give the Encoder a *interface{}, then you can decoded with a *interface{}
var to_enc interface{} = ...;
god.NewEncoder(...).Encode(&to_enc);
...
var to_dec interface{}
god.NewDecoder(...).Decode(&to_dec);
Related
I have a struct that I can successfully gob encode and decode as follows:
type Node struct {
Value int
Next *Node
}
myNode := Node{
Value: 1,
Next: &Node{
Value: 2,
},
}
var buf bytes.Buffer
enc := gob.NewEncoder(&buf)
dec := gob.NewDecoder(&buf)
if err := enc.Encode(&myNode); err != nil {
panic(err)
}
var decodedNode Node
if err := dec.Decode(&decodedNode); err != nil {
panic(err)
}
I'd like to now hide the fields of Node:
type Node struct {
value int
next *Node
}
Because the fields are no longer exported I have to now write custom GobEncode and GobDecode functions:
func (d *Node) GobEncode() ([]byte, error) {
var buf bytes.Buffer
encoder := gob.NewEncoder(&buf)
if err := encoder.Encode(d.value); err != nil {
return nil, err
}
if err := encoder.Encode(d.next); err != nil {
return nil, err
}
return buf.Bytes(), nil
}
func (d *Node) GobDecode(b []byte) error {
buf := bytes.NewBuffer(b)
decoder := gob.NewDecoder(buf)
if err := decoder.Decode(&d.value); err != nil {
return err
}
if err := decoder.Decode(&d.next); err != nil {
return err
}
return nil
}
This doesn't work because of a nil value in Node.next:
panic: gob: cannot encode nil pointer of type *main.Node [recovered]
panic: gob: cannot encode nil pointer of type *main.Node [recovered]
panic: gob: cannot encode nil pointer of type *main.Node
I'm re-using gob.NewEncoder inside my custom function because I'm trying to re-use as much of the encode package as possible.
Any suggestions for getting this working?
I'm trying to parse this petition (https://www.binance.com/api/v1/depth?symbol=MDABTC&limit=500)
I was having tons of problems to create an struct for it, so I used an automated tool, this is what my struct looks like:
type orderBook struct {
Bids [][]interface{} `json:"Bids"`
Asks [][]interface{} `json:"Asks"`
}
I recover and parse the petition by doing:
url := "https://www.binance.com/api/v1/depth?symbol=MDABTC&limit=500"
resp, err := http.Get(url)
if err != nil {
panic(err)
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
panic(err)
}else{
book := orderBook{}
if err := json.Unmarshal(body, &book); err != nil {
panic(err)
}
But whenever I try to make an operation with the struct, like:
v := book.Asks[i][0] * book.Asks[i][1]
I get an error:
invalid operation: book.Asks[i][0] * book.Asks[i][1] (operator * not
defined on interface)
How do I define it? Do I need to create an struct for bids/asks, if so, how would that look like?
Sorry if this seems basic, I just started learning go.
In Golang Spec
For an expression x of interface type and a type T, the primary
expression
x.(T)
asserts that x is not nil and that the value stored in x is of type T.
The notation x.(T) is called a type assertion.
Fetching an underlying value of string type you need to type assert to string from interface.
books.Asks[0][0].(string)
For performing an arithmetic operation on same you needs to convert string into float64 to take into account decimal values
v := strconv.ParseFloat(books.Asks[0][0].(string), 64) * strconv.ParseFloat(books.Asks[0][1].(string), 64)
Checkout code on Go playground
Note that you could define proper structs that unmarshal from the JSON document by implementing the json.Unmarshaler interface.
For example (on the Go Playground):
type OrderBook struct {
Asks, Bids []Order
LastUpdateId int
}
type Order struct {
Price, Volume float64
}
func (o *Order) UnmarshalJSON(bs []byte) error {
values := make([]interface{}, 0, 3)
err := json.Unmarshal(bs, &values)
if err != nil {
return err
}
// TODO: more error checking re: len(values), and their types.
price, err := strconv.ParseFloat(values[0].(string), 10)
if err != nil {
return err
}
volume, err := strconv.ParseFloat(values[1].(string), 10)
if err != nil {
return err
}
*o = Order{price, volume}
return nil
}
As such, unmarshaling those documents looks idiomatic:
func main() {
book := OrderBook{}
err := json.Unmarshal([]byte(jsonstr), &book)
if err != nil {
panic(err)
}
fmt.Printf("Asks: %#v\n", book.Asks)
fmt.Printf("Bids: %#v\n", book.Bids)
fmt.Printf("Update: %#v\n", book.LastUpdateId)
// Asks: []main.Order{main.Order{Price:0.00013186, Volume:167}, main.Order{Price:0.00013187, Volume:128}, ...
// Bids: []main.Order{main.Order{Price:0.00013181, Volume:110}, main.Order{Price:0.00013127, Volume:502}, ...
// Update: 14069188
}
I'm using GIN as GO framework, I'm having an issue when uploading file and directly convert image as byte so I can store it in my BLOB field inside db table, so I have my piece of code like this:
func (a *AppHandler) Upload(ctx *gin.Context) {
form := &struct {
Name string `form:"name" validate:"required"`
Token string `form:"token" validate:"required"`
AppCode string `form:"app_code" validate:"required"`
}{}
ctx.Bind(form)
if validationErrors := a.ValidationService.ValidateForm(form); validationErrors != nil {
httpValidationErrorResponse(ctx, validationErrors)
return
}
file, header, err := ctx.Request.FormFile("file")
and I'm trying to store it in db like this
app.SetFile(file)
a.AppStore.Save(app)
and it returns this kind of error:
cannot use file (type multipart.File) as type []byte
*multipart.File implements io.Reader interface so you could copy its content into a bytes.Buffer like this:
file, header, err := ctx.Request.FormFile("file")
defer file.Close()
if err != nil {
return nil, err
}
buf := bytes.NewBuffer(nil)
if _, err := io.Copy(buf, file); err != nil {
return nil, err
}
and then add to your app
app.SetFile(buf.Bytes())
Here is schematic example what I need:
func decode(data []byte, target interface{}) (interface{}, error) {
decoded := target.(reflect.TypeOf(target)) // pseudocode
err := json.Unmarshal(data, &decoded)
...
I found several similar questions on SO, but all solutions are about switch by target.(type). This is not the best solution what I look for.
Also I found solution with reflection:
decoded := reflect.ValueOf(target).Convert(reflect.TypeOf(target))
// fmt.Printf("%T", decoded) == reflect.Value
But I couldn't understand how to get struct from reflect.Value to pass it to json.Unmarshal function.
How this decode function will be used?
I have multiple requests in different structures. I can determine what struct I should use to decode request. I have mapping between request type and structure like this map[RequestMethod]interface{}.
Schematic version looks like this:
func hydrate(data []byte) (interface{}, error) {
var base baseResponse
if err := json.Unmarshal(data, &base); err != nil {
return nil, err
}
target, ok := methodsMap[Method(base.Method)]
if !ok {
return nil, errors.New("Trying to hydrate data with unknown method: " + base.Method)
}
decoded, err := decode(data, target) // Expected target type.
Added:
If we pass our target to json.Unmarshal without casting to it type we will obtain map. Example:
func decode(data []byte, target interface{}) (interface{}, error) {
err := json.Unmarshal(data, &target)
// fmt.Printf("%T", target) == map[string]interface{} not struct.
Solution
Thanks to #icza we found solution:
func hydrate(data []byte) (interface{}, error) {
var base baseResponse
if err := json.Unmarshal(data, &base); err != nil {
return nil, err
}
target, ok := methodsMap[Method(base.Method)]
if !ok {
return nil, errors.New("Trying to hydrate data with unknown method: " + base.Method)
}
// Clone request draft struct.
decoded := reflect.New(reflect.ValueOf(target).Type()).Interface()
err := decode(data, decoded)
...
Related links:
Golang interface{} type misunderstanding
How to copy an interface value in Go?
Use the following code to create a copy of the draft struct and unmarshal to it:
t := reflect.TypeOf(methodsMap[Method(base.Method)])
pv := reflect.New(t)
err := json.Unmarshal(p, pv.Interface())
return pv.Elem().Interface(), err // pv.Elem() dereferences ptr
playground example
I want to write a file cache in Go. I am using gob encoding, and saving to a file, but my get function has some problem:
package main
import (
"encoding/gob"
"fmt"
"os"
)
var (
file = "tmp.txt"
)
type Data struct {
Expire int64
D interface{}
}
type User struct {
Id int
Name string
}
func main() {
user := User{
Id: 1,
Name: "lei",
}
err := set(file, user, 10)
if err != nil {
fmt.Println(err)
return
}
user = User{}
err = get(file, &user)
if err != nil {
fmt.Println(err)
return
}
//user not change.
fmt.Println(user)
}
func set(file string, v interface{}, expire int64) error {
f, err := os.OpenFile(file, os.O_CREATE|os.O_WRONLY|os.O_TRUNC, 0600)
if err != nil {
return err
}
defer f.Close()
//wrapper data
//save v in data.D
data := Data{
Expire: expire,
D: v,
}
gob.Register(v)
enc := gob.NewEncoder(f)
err = enc.Encode(data)
if err != nil {
return err
}
return nil
}
func get(file string, v interface{}) error {
f, err := os.OpenFile(file, os.O_RDONLY, 0600)
if err != nil {
return err
}
defer f.Close()
var data Data
dec := gob.NewDecoder(f)
err = dec.Decode(&data)
if err != nil {
return err
}
//get v
v = data.D
fmt.Println(v)
return nil
}
The get function passes interface type and I want to change the value, but not change.
http://play.golang.org/p/wV7rBH028o
In order to insert an unknown value into v of type interface{}, you need to use reflection. This is somewhat involved, but if you want to support this in full, you can see how its done by walking through the decoding process in some of the encoding packages (json, gob).
To get you started, here's a basic version of your get function using reflection. This skips a number of checks, and will only decode something that was encoded as a pointer.
func get(file string, v interface{}) error {
f, err := os.OpenFile(file, os.O_RDONLY, 0600)
if err != nil {
return err
}
defer f.Close()
rv := reflect.ValueOf(v)
if rv.Kind() != reflect.Ptr || rv.IsNil() {
panic("need a non nil pointer")
}
var data Data
dec := gob.NewDecoder(f)
err = dec.Decode(&data)
if err != nil {
return err
}
dv := reflect.ValueOf(data.D)
if dv.Kind() != reflect.Ptr {
panic("didn't decode a pointer")
}
rv.Elem().Set(dv.Elem())
return nil
}
I would actually suggest an easier way to handle this in your own code, which is to have the Get function return an interface{}. Since you will know what the possible types are at that point, you can use a type switch to assert the correct value.
An alternative approach is to return directly the value from the file:
func get(file string) (interface{}, error) {
f, err := os.OpenFile(file, os.O_RDONLY, 0600)
if err != nil {
return nil, err
}
defer f.Close()
var data Data
dec := gob.NewDecoder(f)
err = dec.Decode(&data)
if err != nil {
return nil,err
}
fmt.Println(data.D)
return data.D,nil
}
full working example: http://play.golang.org/p/178U_LVC5y