Error unmarshaling when getting array of struct from redis - go

I am using redigo to save some structs in redis. The thing is that for the same key I need to append new structs, but when I am trying to recover them I cannot unmarshal to an array.
Ie: (ignoring the errors intentionally)
type ADTO struct {
Value string
}
func main() {
pool := redis.Pool{
Dial: func() (conn redis.Conn, e error) {
return redis.Dial("tcp", "localhost:6379")
},
MaxIdle: 80,
MaxActive: 12000,
}
conn := pool.Get()
defer conn.Close()
key := "some-key"
defer conn.Do("DEL", key)
a := ADTO{Value: "a"}
bytes, _ := json.Marshal(a)
conn.Do("APPEND", key, bytes)
b := ADTO{Value: "b"}
bytes, _ = json.Marshal(b)
conn.Do("APPEND", key, bytes)
c := ADTO{Value: "c"}
bytes, _ = json.Marshal(c)
conn.Do("APPEND", key, bytes)
bytes, _ = redis.Bytes(conn.Do("GET", key))
adtos := make([]ADTO, 0)
// the following does not work
if err := json.Unmarshal(bytes, &adtos); err != nil {
return
}
}
If I only append a single struct and retrieving it, then it is working fine
I have tried with redis.ByteSlices with no luck

APPEND will only append to a string, it will not make a JSON array. After first insert, you'll have
{"Value":"a"}
Then after the second one, you'll have
{"Value":"a"}{"Value":"b"}
That's not a JSON array.
You can try using json.Decoder, and do something like:
b, _ = redis.Bytes(conn.Do("GET", key))
dec := json.NewDecoder(bytes.NewReader(b))
items := []ADTO{}
var x ADTO
for dec.Decode(&x) == nil {
items = append(items, x)
}

Related

How can I save and retrieve a map into redis using redigo?

I have a map like this, which I want to save/retrive from redis using redigo:
animals := map[string]bool{
"cat": true,
"dog": false,
"fox": true,
}
The length of the map may vary.
I tried these function:
func SetHash(key string, value map[string]bool) error {
conn := Pool.Get()
defer conn.Close()
_, err := conn.Do("HMSET", key, value)
if err != nil {
return fmt.Errorf("error setting key %s to %s: %v", key, value, err)
}
return err
}
func GetHash(key string) (map[string]bool, error) {
conn := Pool.Get()
defer conn.Close()
val, err := conn.Do("HGETALL", key)
if err != nil {
fmt.Errorf("error setting key %s to %s: %v", key, nil, err)
return nil, err
}
return val, err
}
But can not make GetHash correctly. I've checked the docs examples and it was not helpful. So appreciate your help to have a working example.
HMSET is deprecated, use HSET instead, no effect here though.
The map[string]bool may be flattened with AddFlat() for SetHash().
c.Do("HSET", redis.Args{}.Add("key").AddFlat(value)...)
For GetHash(), use Values(). You may use ScanStruct() to map to a struct or loop through the values to create a map dynamically.
v, err := redis.Values(c.Do("HGETALL", key))
redis.ScanStruct(v, &myStruct);
See example from redigo tests in scan_test.go.
The application is responsible for converting structured types to and from the types understood by Redis.
Flatten the map into a list of arguments:
func SetHash(key string, value map[string]bool) error {
conn := Pool.Get()
defer conn.Close()
// Create arguments: key field value [field value]...
var args = []interface{}{key}
for k, v := range value {
args = append(args, k, v)
}
_, err := conn.Do("HMSET", args...)
if err != nil {
return fmt.Errorf("error setting key %s to %v: %v", key, value, err)
}
return err
}
Convert the returned field value pairs to a map:
func GetHash(key string) (map[string]bool, error) {
conn := Pool.Get()
defer conn.Close()
values, err := redis.Strings(conn.Do("HGETALL", key))
if err != nil {
return nil, err
}
// Loop through [field value]... and parse value as bool.
m := map[string]bool{}
for i := 0; i < len(values); i += 2 {
b, err := strconv.ParseBool(value)
if err != nil {
return nil, errors.New("value not a bool")
}
m[key] = b
}
return m, nil
}

golang reflect into []interface{}

i wanna create a mini framework which takes a simple struct and creates a full crud out of it. i already started and the "findOne, update,create,delete" is working. not i have a problem to create a findAll method. to be more clear, i dont know how to use reflect to address my ptr to an array of struct.
Here a small example for the findOne function.
type company struct {
Id int
Name string
}
comp.InitModel(newDbConnection(), &comp)
In InitModel i can fill the pointer to the company with the following:
//m.caller = pointer to the ptr to comp (struct)
callerV := reflect.ValueOf(m.caller)
CallerField := callerV.Elem()
var values []interface{}
for _, e := range m.columns {
values = append(values, CallerField.FieldByName(e.name).Addr().Interface())
}
err := r.Scan(values...)
if err != nil {
return err
}
Now i wanna create a findAll method which would be called like this
var companies []company
comp.InitModel(newDbConnection(), &comp)
comp.FindAll(&companies) //in this is the db query and scan
fmt.Println(companies) //here should be the result
But i have a problem to get the reflect with a []interface working.
func (m *Model) FindAll(test []interface{}, c *Condition) error {
//get the sql statement from the struct
stmt := PrepairStmt(m, c)
rows, err := m.db.Query(stmt.selectParse(), c.arguments...)
if err != nil {
return err
}
defer rows.Close()
callerV := reflect.ValueOf(m.caller)
CallerField := callerV.Elem()
for rows.Next() {
var values []interface{}
for _, e := range m.columns {
values = append(values, CallerField.FieldByName(e.name).Addr().Interface())
}
err = rows.Scan(values...)
if err != nil {
return err
}
valuePtr := reflect.New(reflect.TypeOf(test).Elem())
test = reflect.Append(test,reflect.ValueOf(values))
}
return nil
}
Thats were my latest tries.
maybe someone can help me with this.
i would be really thankful
Use interface{} instead of []interface{} as the argument type:
func (m *Model) FindAll(result interface{}, c *Condition) error {
stmt := PrepairStmt(m, c)
rows, err := m.db.Query(stmt.selectParse(), c.arguments...)
if err != nil {
return err
}
defer rows.Close()
// resultv is the result slice
resultv := reflect.ValueOf(result).Elem()
// rowt is the struct type
rowt := resultv.Type().Elem()
// allocate a value for the row
rowv := reflect.New(rowt).Elem()
// collect values for scan
var values []interface{}
for _, e := range m.columns {
values = append(values, rowv.FieldByName(e.name).Addr().Interface())
}
for rows.Next() {
err = rows.Scan(values...)
if err != nil {
return err
}
// Append struct to result slice. Because the struct
// is copied in append, we can reuse the struct in
// this loop.
resultv.Set(reflect.Append(resultv, rowv))
}
return nil
}
Use it like this:
var companies []company
comp.InitModel(newDbConnection(), &comp)
comp.FindAll(&companies) //in this is the db query and scan

Go can't set string to struct

This code is get all objects from s3 and delete objects.
getAllObjects is called from DeletePhotosFromS3.
I cloud get 2 different keys in objects that is in DeletePhotosFromS3.
But deleteObjects have 2 same keys. ex [{Key: 1}, {Key: 1}].
Why deleteObjects have 2 same keys and how to set objects in []*s3.ObjectIdentifier?
func getAllObject(userID string) (*[]string, error) {
var objects []string
svc := initS3()
config := model.NewConfig()
input := &s3.ListObjectsInput{
Bucket: aws.String(config.AWSS3Bucket),
Prefix: aws.String(userID),
MaxKeys: aws.Int64(2), // default 1000
}
result, err := svc.ListObjects(input)
if err != nil {
return &objects, err
}
for _, v := range result.Contents {
objects = append(objects, *v.Key)
}
return &objects, nil
}
func DeletePhotosFromS3(userID string) (error) {
var deleteObjects []*s3.ObjectIdentifier
svc := initS3()
config := model.NewConfig()
objects, err := getAllObject(userID) // called getAllObject
for _, v := range *objects {
deleteObjects = append(deleteObjects, &s3.ObjectIdentifier{Key: &v}) // Er
}
...
}
The iteration value v in your for loop is reused for each iteration. The Pointer &v will be the same for each item appended to the list. Fixed snippet:
for _, v := range *objects {
vcopy := v
deleteObjects = append(deleteObjects, &s3.ObjectIdentifier{Key: &vcopy})
}

Golang slices append

I am having a problem when appending to my slice using Golang.
Here is my code:
func MatchBeaconWithXY(w http.ResponseWriter, r *http.Request) ([]types.BeaconDataXY, error) {
context := appengine.NewContext(r)
returnBeaconData := []types.BeaconDataXY{}
beacondata, err := GetBeaconData(w, r)
if err != nil {
log.Errorf(context, "error getting beacondata %v", err)
w.WriteHeader(http.StatusInternalServerError)
return nil, err
}
for index, element := range beacondata {
q := datastore.NewQuery("physicalbeacondata").Filter("NamespaceID =", element.NamespaceID).Filter("InstanceID =", element.InstanceID)
beacondatastatic := []types.BeaconDataStatic{}
_, err := q.GetAll(context, &beacondatastatic)
if err != nil {
log.Errorf(context, "cant get query %v", err)
w.WriteHeader(http.StatusInternalServerError)
return nil, err
}
var beacondataXY = new(types.BeaconDataXY)
beacondataXY.NamespaceID = element.NamespaceID
beacondataXY.InstanceID = element.InstanceID
beacondataXY.XCoord = beacondatastatic[0].XCoord
beacondataXY.YCoord = beacondatastatic[0].YCoord
beacondataXY.Distance = element.Distance
returnBeaconData = append(returnBeaconData, beacondataXY...)
log.Infof(context, "beaondataXY tot %v", beacondataXY)
}
The beacondataxy.go contains this:
package types
type BeaconDataXY struct {
InstanceID string
NamespaceID string
XCoord float64
YCoord float64
Distance float64
}
The error message is this:
utils.go:139: cannot use beacondataXY (type *types.BeaconDataXY) as
type []types.BeaconDataXY in append
I don't really know how to handle slices in Golang, even after reading some tutorials that makes perfect sense. I'm not sure what I'm doing wrong.
I want to have an array/slice with types inside, return BeaconData is of []types. BeaconDataXY and it should contain single types of BeaconDataXY.
Thanks for all help.
EDIT:
The code now looks like this:
func MatchBeaconWithXY(w http.ResponseWriter, r *http.Request) ([]types.BeaconDataXY, error) {
context := appengine.NewContext(r)
//returnBeaconData := []types.BeaconDataXY{}
returnBeaconData := make([]types.BeaconDataXY, 1)
beacondata, err := GetBeaconData(w, r)
if err != nil {
log.Errorf(context, "error getting beacondata %v", err)
w.WriteHeader(http.StatusInternalServerError)
return nil, err
}
for _, element := range beacondata {
q := datastore.NewQuery("physicalbeacondata").Filter("NamespaceID =", element.NamespaceID).Filter("InstanceID =", element.InstanceID)
beacondatastatic := []types.BeaconDataStatic{}
_, err := q.GetAll(context, &beacondatastatic)
if err != nil {
log.Errorf(context, "cant get query %v", err)
w.WriteHeader(http.StatusInternalServerError)
return nil, err
}
var beacondataXY = types.BeaconDataXY{}
beacondataXY.NamespaceID = element.NamespaceID
beacondataXY.InstanceID = element.InstanceID
beacondataXY.XCoord = beacondatastatic[0].XCoord
beacondataXY.YCoord = beacondatastatic[0].YCoord
beacondataXY.Distance = element.Distance
returnBeaconData = append(returnBeaconData, beacondataXY)
//log.Infof(context, "beaondataXY tot %v", beacondataXY)
}
With this assignment:
var beacondataXY = new(types.BeaconDataXY)
you are creating a variable of type *types.BeaconDataXY. Just create a new BeaconDataXY like this:
var beacondataXY = types.BeaconDataXY{}
When appending to your array do it like this:
returnBeaconData = append(returnBeaconData, beacondataXY)
The "..." would assume that beacondataXY is an array but it isn't, you just want to append beacondataXY to returnBeaconData. See https://golang.org/ref/spec#Appending_and_copying_slices for an explanation of what "..." means in this context.
Try returnBeaconData = append(returnBeaconData, *beacondataXY)
new() built-in function returns a pointer, you can alternatively write:
var beacondataXY = types.BeaconDataXY{}

Generic function to store Go Encoded Data

I need to write a generic function which can store objects as gobjects.
func hash_store(data map[string]string) {
//initialize a *bytes.Buffer
m := new(bytes.Buffer)
//the *bytes.Buffer satisfies the io.Writer interface and can
//be used in gob.NewEncoder()
enc := gob.NewEncoder(m)
//gob.Encoder has method Encode that accepts data items as parameter
enc.Encode(data)
//the bytes.Buffer type has method Bytes() that returns type []byte,
//and can be used as a parameter in ioutil.WriteFile()
err := ioutil.WriteFile("dep_data", m.Bytes(), 0600)
if err != nil {
panic(err)
}
fmt.Printf("just saved all depinfo with %v\n", data)
n,err := ioutil.ReadFile("dep_data")
if err != nil {
fmt.Printf("cannot read file")
panic(err)
}
//create a bytes.Buffer type with n, type []byte
p := bytes.NewBuffer(n)
//bytes.Buffer satisfies the interface for io.Writer and can be used
//in gob.NewDecoder()
dec := gob.NewDecoder(p)
//make a map reference type that we'll populate with the decoded gob
//e := make(map[int]string)
e := make(map[string]string)
//we must decode into a pointer, so we'll take the address of e
err = dec.Decode(&e)
if err != nil {
fmt.Printf("cannot decode")
panic(err)
}
fmt.Println("after reading dep_data printing ",e)
}
In this function I know the data type to be stored in map[string]string . But I need to write a generic function where I don't know data type and still store it as a gobject in a file.
Change your concrete type (map[string]string) to the empty interface type (interface{}).
See this related question why this works.
Encoding:
func store(data interface{}) {
m := new(bytes.Buffer)
enc := gob.NewEncoder(m)
err := enc.Encode(data)
if err != nil { panic(err) }
err = ioutil.WriteFile("dep_data", m.Bytes(), 0600)
if err != nil { panic(err) }
}
Decoding:
func load(e interface{}) {
n,err := ioutil.ReadFile("dep_data")
if err != nil { panic(err) }
p := bytes.NewBuffer(n)
dec := gob.NewDecoder(p)
err = dec.Decode(e)
if err != nil { panic(err) }
}
The value you put in load must be a pointer of the type you stored in the file using gob.
Example for map[string]string:
org := map[string]string{"foo": "bar"}
store(org)
var loadedMap map[string]string
load(&loadedMap)
fmt.Println(loadedMap["foo"]) // bar
When you encode the data, give the Encoder a *interface{}, then you can decoded with a *interface{}
var to_enc interface{} = ...;
god.NewEncoder(...).Encode(&to_enc);
...
var to_dec interface{}
god.NewDecoder(...).Decode(&to_dec);

Resources