Decode JSON into Interface Value - go

As encoding/json needs a non-nil interface to unmarshal into: how can I reliably make a (full) copy of a user-provided pointer type, store that in my User interface, and then JSON decode into that ad-hoc?
Note: the goal here is to do this 'unattended' - that is, pull the bytes from Redis/BoltDB, decode into the interface type, and then check that the GetID() method the interface defines returns a non-empty string, with request middleware.
Playground: http://play.golang.org/p/rYODiNrfWw
package main
import (
"bytes"
"encoding/json"
"fmt"
"log"
"net/http"
"os"
"time"
)
type session struct {
ID string
User User
Expires int64
}
type User interface {
GetID() string
}
type LocalUser struct {
ID string
Name string
Created time.Time
}
func (u *LocalUser) GetID() string {
return u.ID
}
type Auth struct {
key []byte
// We store an instance of userType here so we can unmarshal into it when
// deserializing from JSON (or other non-gob decoders) into *session.User.
// Attempting to unmarshal into a nil session.User would otherwise fail.
// We do this so we can unmarshal from our data-store per-request *without
// the package user having to do so manually* in the HTTP middleware. We can't
// rely on the user passing in an fresh instance of their User satisfying type.
userType User
}
func main() {
// auth is a pointer to avoid copying the struct per-request: although small
// here, it contains a 32-byte key, options fields, etc. outside of this example.
var auth = &Auth{key: []byte("abc")}
local := &LocalUser{"19313", "Matt", time.Now()}
b, _, _, err := auth.SetUser(local)
if err != nil {
log.Fatalf("SetUser: %v", err)
}
user, err := auth.GetUser(b)
if err != nil {
log.Fatalf("GetUser: %#v", err)
}
fmt.Fprintf(os.Stdout, "%v\n", user)
}
func (auth *Auth) SetUser(user User) (buf []byte, id string, expires int64, err error) {
sess := newSession(user)
// Shallow copy the user into our config. struct so we can copy and then unmarshal
// into it in our middleware without requiring the user to provide it themselves
// at the start of every request
auth.userType = user
b := bytes.NewBuffer(make([]byte, 0))
err = json.NewEncoder(b).Encode(sess)
if err != nil {
return nil, id, expires, err
}
return b.Bytes(), sess.ID, sess.Expires, err
}
func (auth *Auth) GetUser(b []byte) (User, error) {
sess := &session{}
// Another shallow copy, which means we're re-using the same auth.userType
// across requests (bad).
// Also note that we need to copy into it session.User so that encoding/json
// can unmarshal into its fields.
sess.User = auth.userType
err := json.NewDecoder(bytes.NewBuffer(b)).Decode(sess)
if err != nil {
return nil, err
}
return sess.User, err
}
func (auth *Auth) RequireAuth(h http.Handler) http.Handler {
fn := func(w http.ResponseWriter, r *http.Request) {
// e.g. user, err := store.Get(r, auth.store, auth.userType)
// This would require us to have a fresh instance of userType to unmarshal into
// on each request.
// Alternative might be to have:
// func (auth *Auth) RequireAuth(userType User) func(h http.Handler) http.Handler
// e.g. called like http.Handle("/monitor", RequireAuth(&LocalUser{})(SomeHandler)
// ... which is clunky and using closures like that is uncommon/non-obvious.
}
return http.HandlerFunc(fn)
}
func newSession(u User) *session {
return &session{
ID: "12345",
User: u,
Expires: time.Now().Unix() + 3600,
}
}

If you need to deep copy an interface, add that method to your interface.
type User interface {
GetID() string
Copy() User
}
type LocalUser struct {
ID string
Name string
Created time.Time
}
// Copy receives a copy of LocalUser and returns a pointer to it.
func (u LocalUser) Copy() User {
return &u
}

Because the application will decode to a User and the argument to the JSON decoder must be a pointer value, we can assume that User values are pointer values. Given this assumption, the following code can be used to create a new zero value for decoding:
uzero := reflect.New(reflect.TypeOf(u).Elem()).Interface().(User)
playground example

Related

Go mockgen, mocked function not called

I am using Go 1.19 on a windows machine with 8 cores, operating system is Windows 10 Pro.
I used the mockgen tool to generate the mock. When I debug my test I see the mocked method is recorded when I execute the EXPECT() function.
The mocked function is called, but the test fails with 'missing call' on the mocked function.
I cannot see what I am doing wrong, can anyone please point it out ?
Directory Structure :
cmd
configure.go
configure_test.go
mocks
mock_validator.go
validator
validator.go
user
user.go
go.mod
main.go
* Contents of main.go
package main
import (
"localdev/mockexample/cmd"
)
func main() {
cmd.Configure()
}
* Contents of configure.go
package cmd
import (
"fmt"
"localdev/mockexample/user"
"os"
"localdev/mockexample/validator"
)
var (
name, password string
)
func Configure() {
name := os.Args[1]
password := os.Args[2]
user, err := validate(validator.NewValidator(name, password))
if err != nil {
fmt.Printf("%v\n", err)
return
}
fmt.Printf("Credentials are valid. Welcome: %s %s\n", user.FirstName, user.LastName)
}
func validate(validator validator.Validator) (*user.Data, error) {
user, err := validator.ValidateUser()
if err != nil {
return nil, fmt.Errorf("some thing went wrong. %v", err)
}
return user, nil
}
* Contents of validator.go
package validator
import (
"fmt"
"localdev/mockexample/user"
)
//go:generate mockgen -destination=../mocks/mock_validator.go -package=mocks localdev/mockexample/validator Validator
type Validator interface {
ValidateUser() (*user.Data, error)
}
type ValidationRequest struct {
Command string
Name string
Password string
}
func (vr ValidationRequest) ValidateUser() (*user.Data, error) {
if vr.Name == "bob" && vr.Password == "1234" {
return &user.Data{UserID: "123", UserName: "bsmith", FirstName: "Bob", LastName: "Smith"}, nil
}
return nil, fmt.Errorf("invalid credentials")
}
func NewValidator(name string, password string) Validator {
return &ValidationRequest{Name: name, Password: password}
}
* Contents of user.go
package user
type Data struct {
UserID string `json:"user_id"`
UserName string `json:"user_name"`
FirstName string `json:"first_name"`
LastName string `json:"last_name"`
}
* Contents of configure_test.go
package cmd
import (
"localdev/mockexample/mocks"
"localdev/mockexample/user"
"os"
"testing"
"github.com/golang/mock/gomock"
)
func TestConfigure(t *testing.T) {
t.Run("ConfigureWithMock", func(t *testing.T) {
os.Args[1] = "bob"
os.Args[2] = "1234"
ctrl := gomock.NewController(t)
mockValidator := mocks.NewMockValidator(ctrl)
//mockValidator.EXPECT().ValidateUser().AnyTimes() // zero more calls, so this will also pass.
userData := user.Data{UserID: "testId"}
mockValidator.EXPECT().ValidateUser().Return(&userData, nil).Times(1) //(gomock.Any(), gomock.Any()) //(&userData, nil)
Configure()
})
}
Contents of generated mock
// Code generated by MockGen. DO NOT EDIT.
// Source: localdev/mockexample/validator (interfaces: Validator)
// Package mocks is a generated GoMock package.
package mocks
import (
user "localdev/mockexample/user"
reflect "reflect"
gomock "github.com/golang/mock/gomock"
)
// MockValidator is a mock of Validator interface.
type MockValidator struct {
ctrl *gomock.Controller
recorder *MockValidatorMockRecorder
}
// MockValidatorMockRecorder is the mock recorder for MockValidator.
type MockValidatorMockRecorder struct {
mock *MockValidator
}
// NewMockValidator creates a new mock instance.
func NewMockValidator(ctrl *gomock.Controller) *MockValidator {
mock := &MockValidator{ctrl: ctrl}
mock.recorder = &MockValidatorMockRecorder{mock}
return mock
}
// EXPECT returns an object that allows the caller to indicate expected use.
func (m *MockValidator) EXPECT() *MockValidatorMockRecorder {
return m.recorder
}
// ValidateUser mocks base method.
func (m *MockValidator) ValidateUser() (*user.Data, error) {
m.ctrl.T.Helper()
ret := m.ctrl.Call(m, "ValidateUser")
ret0, _ := ret[0].(*user.Data)
ret1, _ := ret[1].(error)
return ret0, ret1
}
// ValidateUser indicates an expected call of ValidateUser.
func (mr *MockValidatorMockRecorder) ValidateUser() *gomock.Call {
mr.mock.ctrl.T.Helper()
return mr.mock.ctrl.RecordCallWithMethodType(mr.mock, "ValidateUser", reflect.TypeOf((*MockValidator)(nil).ValidateUser))
}
The root problem is that the function Configure never uses the mock structure, so you get a missing call(s) to *mocks.MockValidator.ValidateUser() error.
In the file configure_test.go, mockValidator is simply not used at all. There must be some kind of injection of that mock in order to be called by the Configure function.
You could make the following changes to fix the test, as an example of what I'm referring to injection. Not saying this is the best approach but I'm trying to make the fewer possible changes to your code.
configure_test.go:
func TestConfigure(t *testing.T) {
t.Run("ConfigureWithMock", func(t *testing.T) {
os.Args[1] = "bob"
os.Args[2] = "1234"
ctrl := gomock.NewController(t)
mockValidator := mocks.NewMockValidator(ctrl)
//mockValidator.EXPECT().ValidateUser().AnyTimes() // zero more calls, so this will also pass.
userData := user.Data{UserID: "testId"}
mockValidator.
EXPECT().
ValidateUser("bob", "1234").
Return(&userData, nil).
Times(1) //(gomock.Any(), gomock.Any()) //(&userData, nil)
Configure(mockValidator)
})
}
configure.go
func Configure(v validator.Validator) {
name := os.Args[1]
password := os.Args[2]
user, err := v.ValidateUser(name, password)
if err != nil {
fmt.Printf("some thing went wrong. %v\n", err)
return
}
fmt.Printf("Credentials are valid. Welcome: %s %s\n", user.FirstName, user.LastName)
}
validator.go
type Validator interface {
ValidateUser(name, password string) (*user.Data, error)
}
type ValidationRequest struct {
Command string
// Name string
// Password string
}
func (vr ValidationRequest) ValidateUser(name, password string) (*user.Data, error) {
if name == "bob" && password == "1234" {
return &user.Data{UserID: "123", UserName: "bsmith", FirstName: "Bob", LastName: "Smith"}, nil
}
return nil, fmt.Errorf("invalid credentials")
}
func NewValidator() Validator {
return &ValidationRequest{}
}
Take into account that you need to generate the mock again. Hope this helps you to understand mock testing.

redis: can't marshal map[string]string (implement encoding.BinaryMarshaler) ----in Refis interface

I want to create a generic Redis interface for storing and getting values.
I am beginner to Golang and redis.
If there are any changes to be done to the code I would request you to help me.
package main
import (
"fmt"
"github.com/go-redis/redis"
)
func main() {
student := map[string]string{
"id": "st01",
"name": "namme1",
}
set("key1", student, 0)
get("key1")
}
func set(key string, value map[string]string, ttl int) bool {
client := redis.NewClient(&redis.Options{
Addr: "localhost:6379",
Password: "",
DB: 0,
})
err := client.Set(key, value, 0).Err()
if err != nil {
fmt.Println(err)
return false
}
return true
}
func get(key string) bool {
client := redis.NewClient(&redis.Options{
Addr: "localhost:6379",
Password: "",
DB: 0,
})
val, err := client.Get(key).Result()
if err != nil {
fmt.Println(err)
return false
}
fmt.Println(val)
return true
}
When i run this code i receive an error of "redis: can't marshal map[string]string (implement encoding.BinaryMarshaler)".
I have tried using marshal but there was no use.
I would request you to help me with this.
The non-scalar type of go cannot be directly converted to the storage structure of redis, so the structure needs to be converted before storage
If you want to implement a general method, then the method should receive a type that can be stored directly, and the caller is responsible for converting the complex structure into a usable type, for example:
// ...
student := map[string]string{
"id": "st01",
"name": "namme1",
}
// Errors should be handled here
bs, _ := json.Marshal(student)
set("key1", bs, 0)
// ...
func set(key string, value interface{}, ttl int) bool {
// ...
}
A specific method can structure a specific structure, but the structure should implement the serializers encoding.MarshalBinary and encoding.UnmarshalBinary, for example:
type Student map[string]string
func (s Student) MarshalBinary() ([]byte, error) {
return json.Marshal(s)
}
// make sure the Student interface here accepts a pointer
func (s *Student) UnmarshalBinary(data []byte) error {
return json.Unmarshal(data, s)
}
// ...
student := Student{
"id": "st01",
"name": "namme1",
}
set("key1", student, 0)
// ...
func set(key string, value Student, ttl int) bool {
// ...
}

Using different levels of interfaces

I'm trying to organize my code using interfaces in Go.
I have 2 sources of data: FTP and API. In each source, I have several structs that make the logic vary depending on the case.
In this question, I will omit API and stick with FTP.
My problem comes from the impossibility to say: FTPAcq is also an Acquisition
If FetchMeters(), when I do ftp.Decrypt(nil) I would like ftp to be "compatible" with FTPAcq
Here is my code:
package main
import (
"github.com/dutchcoders/goftp"
log "github.com/sirupsen/logrus"
"os"
)
type Acquisition interface {
FetchMeters() ([]Meter, error)
Name() string
}
type FTPAcq interface {
Unzip(file string) string
Decrypt(file string) string
}
//type APIAcq interface {
// FetchMeter(meterID string) (Meter, error)
//}
func main() {
var acqs []Acquisition
ftp, err := NewFTPDriver(os.Getenv("FTP_USER"), os.Getenv("FTP_PASSWD"), os.Getenv("FTP_ADDR"), os.Getenv("FTP_PORT"))
if err != nil {
panic(err)
}
ftp1 := NewFTPDriverSGE(*ftp)
ftp2 := NewFTPDriverTA(*ftp)
acqs = append(acqs, ftp1, ftp2)
for _, acq := range acqs {
tmpMeters, err := acq.FetchMeters()
if err != nil {
log.Warn(acq.Name(), " got error :", err)
}
log.Info(tmpMeters)
}
}
type Meter struct {
ID string
OperationID string
Unit string
}
//FtpSGE is a implementation of acquisition Interface (see driver.go)
type FTP struct {
Username string
Password string
Url string
Port string
client *goftp.FTP
}
type FTPSGE struct {
FTP
}
type FTPTA struct {
FTP
}
func (f FTPSGE) Unzip(path string) []string {
return nil
}
func (f FTPTA) Unzip(path string) []string {
return nil
}
func (f FTPSGE) Decrypt(path string) []string {
return nil
}
func (f FTPTA) Decrypt(path string) []string {
return nil
}
func (ftp FTP) FetchMeters() ([]Meter, error) {
log.Info(ftp.Name(), " is running")
files := ftp.Download(nil)
files = ftp.Decrypt("") // I have several implementation of Decrypt
files = ftp.Unzip("") // I have several implementation of Unzip
log.Info(files)
return nil, nil
}
func (ftp FTP) Name() string {
panic("FTP ")
}
func (ftp FTP) Download(files []string) []string {
panic("implement me")
}
func NewFTPDriver(user, password, url, port string) (*FTP, error) {
var err error
ftp := &FTP{
Username: user,
Password: password,
Url: url,
Port: port,
}
if ftp.client, err = goftp.Connect(url + ":" + port); err != nil {
return nil, err
}
if err = ftp.client.Login(user, password); err != nil {
return nil, err
}
return ftp, nil
}
func NewFTPDriverSGE(f FTP) *FTPSGE {
return &FTPSGE{f}
}
func NewFTPDriverTA(f FTP) *FTPTA {
return &FTPTA{f}
}
In my case, I get:
ftp.Decrypt undefined (type FTP has no field or method Decrypt)
How should I do?
FTP does not implement FTPAcq. It implements only Acquisition. It doesn't even have Decrypt() as a method, interface or not.
FTPSGE and FTPTA implement FTPAcq, but they are not the same type as FTP.
I don't know what you're trying to accomplish, but perhaps something to try is embedding FTP in FTPSGE and FTPTA. This gives those 2 types the fields and methods of the embedded type and still allows you to define additional methods on those types (the methods for FTPAcq in your case).
For example
type FTPSGE {
FTP
}
// OR
type FTPSGE {
*FTP
}
Which you then create as so: x := FTPSGE{ftp1}. Keep in mind that this will create a copy of ftp1 inside x. If ftp1 is type FTP (not a pointer), the entire struct gets copied. If ftp1 is type *FTP (a pointer, which seems to be what you're using), the pointer is copied and x.FTP still points to the same data as ftp1.
This means FTPSGE will implement both Acquisition and FTPAcq.
You'll have to be careful of whether or not interfaces are implemented on the value or pointer: func (a A) Something() vs func (a *A) Somthing().
Here is some reading on methods, interfaces, and embedding.
https://www.ardanlabs.com/blog/2014/05/methods-interfaces-and-embedded-types.html
https://travix.io/type-embedding-in-go-ba40dd4264df
https://golang.org/doc/effective_go.html (and lots of other info)

How to use Go's type alias to make own models work with protobufs?

I've got some REST API with my models defined as Go structs.
type User struct {
FirstName string
LastName string
}
Then I've got my database methods for getting data.
GetUserByID(id int) (*User, error)
Now I'd like to replace my REST API with https://github.com/twitchtv/twirp .
Therefore I started defining my models inside .proto files.
message User {
string first_name = 2;
string last_name = 3;
}
Now I've got two User types. Let's call them the native and the proto type.
I've also got a service defined in my .proto file which returns a user to the frontend.
service Users {
rpc GetUser(Id) returns (User);
}
This generates an interface that I have to fill in.
func (s *Server) GetUser(context.Context, id) (*User, error) {
// i'd like to reuse my existing database methods
u, err := db.GetUserByID(id)
// handle error
// do more stuff
return u, nil
}
Unfortunately this does not work. My database returns a native User but the interface requires a proto user.
Is there an easy way to make it work? Maybe using type aliases?
Thanks a lot!
One way you can solve your problem is by doing the conversion manually.
type User struct {
FirstName string
LastName string
}
type protoUser struct {
firstName string
lastName string
}
func main() {
u := db() // Retrieve a user from a mocked db
fmt.Println("Before:")
fmt.Printf("%#v\n", *u) // What db returns (*protoUser)
fmt.Println("After:")
fmt.Printf("%#v\n", u.AsUser()) // What conversion returns (User)
}
// Mocked db that returns pointer to protoUser
func db() *protoUser {
pu := protoUser{"John", "Dough"}
return &pu
}
// Conversion method (converts protoUser into a User)
func (pu *protoUser) AsUser() User {
return User{pu.firstName, pu.lastName}
}
The key part is the AsUser method on the protoUser struct.
There we simply write our custom logic for converting a protoUser into a User type we want to be working with.
Working Example
As #Peter mentioned in the comment section.
I've seen a project which made it with a custom Convert function. It converts the Protobuf to local struct via json.Unmarshal, not sure how's the performance but it's a way to go.
Preview Code PLAYGROUND
// Convert converts the in struct to out struct via `json.Unmarshal`
func Convert(in interface{}, out interface{}) error {
j, err := json.Marshal(in)
if err != nil {
return err
}
err = json.Unmarshal(j, &out)
if err != nil {
return err
}
return nil
}
func main() {
// Converts the protobuf struct to local struct via json.Unmarshal
var localUser User
if err := convert(protoUser, &localUser); err != nil {
panic(err)
}
}
Output
Before:
main.ProtoUser{FirstName:"John", LastName:"Dough"}
After:
main.User{FirstName:"John", LastName:"Dough"}
Program exited.

Querying the database for basic auth using go-http-auth with martini-go

I am attempting to use go-http-auth with martini-go. In the example given here - https://github.com/abbot/go-http-auth
package main
import (
auth "github.com/abbot/go-http-auth"
"fmt"
"net/http"
)
func Secret(user, realm string) string {
if user == "john" {
// password is "hello"
return "$1$dlPL2MqE$oQmn16q49SqdmhenQuNgs1"
}
return ""
}
func handle(w http.ResponseWriter, r *auth.AuthenticatedRequest) {
fmt.Fprintf(w, "<html><body><h1>Hello, %s!</h1></body></html>", r.Username)
}
func main() {
db, err := sql.Open("postgres", "postgres://blabla:blabla#localhost/my_db")
authenticator := auth.NewBasicAuthenticator("example.com", Secret)
m := martini.Classic()
m.Map(db)
m.Get("/users", authenticator.Wrap(MyUserHandler))
m.Run()
}
The Secret function is using a hardcoded user "john".
The authentication is successful when I execute
curl --user john:hello localhost:3000/users
Obviously, this is a trivial example with hardcoded username and password.
I am now changing the Secret function to this
func Secret(user, realm string) string {
fmt.Println("Executing Secret")
var db *sql.DB
var (
username string
password string
)
err := db.QueryRow("select username, password from users where username = ?", user).Scan(&username, &password)
if err == sql.ErrNoRows {
return ""
}
if err != nil {
log.Fatal(err)
}
//if user == "john" {
//// password is "hello"
//return "$1$dlPL2MqE$oQmn16q49SqdmhenQuNgs1"
//}
//return ""
return ""
}
But it fails with PANIC: runtime error: invalid memory address or nil pointer dereference. Which is obviously because I am attempting to instantiate var db *sql.DB in the Secret function. I cannot pass db *sql.DB into the Secret function either because auth.BasicNewAuthentication is expecting a Secret argument that conforms to type func (string, string) string.
How can I implement my database query correctly and return the password for comparison?
You can use a simple closure to pass in a reference to your DB to the authenticator function:
authenticator := auth.NewBasicAuthenticator("example.com", func(user, realm string) string {
return Secret(db, user, realm)
})
…and then change your Secret to accept the database as the first argument:
func Secret(db *sql.DB, user, realm string) string {
// do your db lookup here…
}
Alternative approach to Attilas answer. You can define a struct, define the Secret() handler on it and pass only the referenced function (go keeps the reference to the "owner") into the authhandler.
type SecretDb struct {
db *DB
}
func (db *SecretDb) Secret(user, realm string) string {
// .. use db.db here
}
func main() {
secretdb = SecretDb{db}
...
auth.NewBasicAuthenticator("example.com", secretdb.Secret)
...
}

Resources