mock db connection with testfy golang - go

Context
I have a simple gin based Rest API implementing CRUD over a redis DB. I want to mock the redis client (github.com/go-redis/redis/v9) in my unit tests. Coming from OOP, I feel that I am transposing some patterns wrongly.
Here is the route I want to test (internal/api.go)
func GetRouter() *gin.Engine {
router := gin.Default()
...
router.GET("/session/:sessionid", getSession)
return router
}
Its controller is defined in internal/controlers.go
func getSession(c *gin.Context) {
client := redis_client.GetClient()
sessionid := c.Param("sessionid")
val, err := client.Get(c, sessionid).Result()
if err == redis.Nil {
fmt.Printf("key %s not exist", sessionid)
c.JSON(404, fmt.Sprintf("session %s not found", sessionid))
} else if err != nil {
fmt.Printf("unkown error %s", err)
c.JSON(500, fmt.Sprintf(`{"error":"%s"}`, err))
} else {
c.JSON(200, val)
}
}
redis_client.GetClient comes from one of my pkg. It's a singleton. pkg/redis/redis.go
var redisClient *redis.Client
func GetClient() *redis.Client {
if redisClient == nil {
redisClient = redis.NewClient(&redis.Options{
Addr: os.Getenv("REDIS_HOST"),
Password: os.Getenv("REDIS_PWD"),
DB: 0,
})
}
return redisClient
}
Testing
Spontaneously, I tried to mock either redis.Client.Get or redis.StringCmd.Result, for it is the place where the actual call to Redis occurs. In both cases, I am using github.com/stretchr/testify/mock
Moking redis.Client.Get
import (
....
"testing"
"github.com/stretchr/testify/mock"
)
type MockedRedisConn struct {
mock.Mock
}
func (m *MockedRedisConn) Get(ctx context.Context, key string) *redis.StringCmd {
ret := m.Called(ctx, key)
var r0 *redis.StringCmd
if ret.Get(0) != nil {
r0 = ret.Get(0).(*redis.StringCmd)
}
return r0
}
func Test_Sessions(t *testing.T) {
r := api.GetRouter()
redisConn := new(MockedRedisConn)
var ctx = context.Background()
t.Run("Should return the session", func(t *testing.T) {
//mocks
strCmd := redis.NewStringCmd(ctx)
strCmd.SetVal(`{"session":""}`)
redisConn.On("Get", mock.AnythingOfType("*gin.Context"), "00000000").Return(strCmd)
//call
req, _ := http.NewRequest("GET", "/session/00000000", nil)
w := httptest.NewRecorder()
r.ServeHTTP(w, req)
responseData, _ := ioutil.ReadAll(w.Body)
assert.Equal(t, `{"session":""}`, string(responseData))
assert.Equal(t, http.StatusOK, w.Code)
})
}
Same logic with redis.StringCmd.Result: I end up on a call to localhost:6379. What puzzles me is that, in most examples I could find, the mocked object is an argument of the function to be tested (I do not quite understand the point of it, by the way). In my case, it is not.
I typically miss a link to set the client in getSession to be my MockedRedisConn rather than the client given by GetClient() (i.e. some kind of monkey patching, if I am not mistaken). Is my singleton based setting of the client compatible with this approach? Should I rather load the client from *gin.Context so I could easily replace it when calling GetRouter in my tests?

Related

Go create a mock for gcp compute sdk

I use the following function, and I need to raise the coverage of it (if possible to 100%), the problem is that typically I use interface to handle such cases in Go and for this specific case not sure how to do it, as this is a bit more tricky, any idea?
The package https://pkg.go.dev/google.golang.org/genproto/googleapis/cloud/compute/v1
Which I use doesn't have interface so not sure how can I mock it?
import (
"context"
"errors"
"fmt"
"os"
compute "cloud.google.com/go/compute/apiv1"
"google.golang.org/api/iterator"
"google.golang.org/api/option"
computev1 "google.golang.org/genproto/googleapis/cloud/compute/v1"
)
func Res(ctx context.Context, project string, region string,vpc string,secret string) error {
c, err := compute.NewAddressesRESTClient(ctx, option.WithCredentialsFile(secret))
if err != nil {
return err
}
defer c.Close()
addrReq := &computev1.ListAddressesRequest{
Project: project,
Region: region,
}
it := c.List(ctx, addrReq)
for {
resp, err := it.Next()
if err == iterator.Done {
break
}
if err != nil {
return err
}
if *(resp.Status) != "IN_USE" {
return ipConverter(*resp.Name, vpc)
}
}
return nil
}
Whenever I find myself in this scenario, I found that the easiest solution is to create missing interfaces myself. I limit these interfaces to the types and functions that I am using, instead of writing interfaces for the entire library. Then, in my code, instead of accepting third-party concrete types, I accept my interfaces for those types. Then I use gomock to generate mocks for these interfaces as usual.
The following is a descriptive example inspired by your code.
type RestClient interface {
List(context.Context, *computev1.ListAddressesRequest) (ListResult, error) // assuming List returns ListResult type.
Close() error
}
func newRestClient(ctx context.Context, secret string) (RestClient, error) {
return compute.NewAddressesRESTClient(ctx, option.WithCredentialsFile(secret))
}
func Res(ctx context.Context, project string, region string, vpc string, secret string) error {
c, err := newRestClient(ctx, secret)
if err != nil {
return err
}
defer c.Close()
return res(ctx, project, region, vpc, c)
}
func res(ctx context.Context, project string, region string, vpc string, c RestClient) error {
addrReq := &computev1.ListAddressesRequest{
Project: project,
Region: region,
}
it, err := c.List(ctx, addrReq)
if err != nil {
return err
}
for {
resp, err := it.Next()
if err == iterator.Done {
break
}
if err != nil {
return err
}
if *(resp.Status) != "IN_USE" {
return ipConverter(*resp.Name, vpc)
}
}
return nil
}
Now you can test the important bits of the Res function by injecting a mock RestClient to the internal res function.
One obstacle to testability here is that you instantiate a client inside your Res function rather than injecting it. Because
the secret doesn't change during the lifetime of the programme,
the methods of *compute.AddressesClient (other than Close) are concurrency-safe,
you could create one client and reuse it for each invocation or Res. To inject it into Res, you can declare some Compute type and turn Res into a method on that type:
type Compute struct {
Lister Lister // some appropriate interface type
}
func (cp *Compute) Res(ctx context.Context, project, region, vpc string) error {
addrReq := &computev1.ListAddressesRequest{
Project: project,
Region: region,
}
it := cp.Lister.List(ctx, addrReq)
for {
resp, err := it.Next()
if err == iterator.Done {
break
}
if err != nil {
return err
}
if *(resp.Status) != "IN_USE" {
return ipConverter(*resp.Name, vpc)
}
}
return nil
}
One question remains: how should you declare Lister? One possibility is
type Lister interface {
List(ctx context.Context, req *computev1.ListAddressesRequest, opts ...gax.CallOption) *compute.AddressIterator
}
However, because compute.AddressIterator is a struct type with some unexported fields and for which package compute provides no factory function, you can't easily control how the iterator returned from List behaves in your tests. One way out is to declare an additional interface,
type Iterator interface {
Next() (*computev1.Address, error)
}
and change the result type of List from *compute.AddressIterator to Iterator:
type Lister interface {
List(ctx context.Context, req *computev1.ListAddressesRequest, opts ...gax.CallOption) Iterator
}
Then you can declare another struct type for the real Lister and use that on the production side:
type RealLister struct {
Client *compute.AddressesClient
}
func (rl *RealLister) List(ctx context.Context, req *computev1.ListAddressesRequest, opts ...gax.CallOption) Iterator {
return rl.Client.List(ctx, req, opts...)
}
func main() {
secret := "don't hardcode me"
ctx, cancel := context.WithCancel(context.Background()) // for instance
defer cancel()
c, err := compute.NewAddressesRESTClient(ctx, option.WithCredentialsFile(secret))
if err != nil {
log.Fatal(err) // or deal with the error in some way
}
defer c.Close()
cp := Compute{Lister: &RealLister{Client: c}}
if err := cp.Res(ctx, "my-project", "us-east-1", "my-vpc"); err != nil {
log.Fatal(err) // or deal with the error in some way
}
}
For your tests, you can declare another struct type that will act as a configurable test double:
type FakeLister func(ctx context.Context, req *computev1.ListAddressesRequest, opts ...gax.CallOption) Iterator
func (fl FakeLister) List(ctx context.Context, req *computev1.ListAddressesRequest, opts ...gax.CallOption) Iterator {
return fl(ctx, req, opts...)
}
To control the behaviour of the Iterator in your test, you can declare another configurable concrete type:
type FakeIterator struct{
Err error
Status string
}
func (fi *FakeIterator) Next() (*computev1.Address, error) {
addr := computev1.Address{Status: &fi.Status}
return &addr, fi.Err
}
A test function may look like this:
func TestResStatusInUse(t *testing.T) {
// Arrange
l := func(_ context.Context, _ *computev1.ListAddressesRequest, _ ...gax.CallOption) Iterator {
return &FakeIterator{
Status: "IN_USE",
Err: nil,
}
}
cp := Compute{Lister: FakeLister(l)}
dummyCtx := context.Background()
// Act
err := cp.Res(dummyCtx, "my-project", "us-east-1", "my-vpc")
// Assert
if err != nil {
// ...
}
}

How to create generic or global context in golang for new relic (Golang New relic integration)?

I am creating new relic transaction in main.go and have to pass it along to handler and then to controller and so on. Is there a way i can define this globally and then can be accessed in any handler, controller or and db transaction?
import(
"github.com/gin-gonic/gin"
newrelic "github.com/newrelic/go-agent"
)
// main.go
func newrelicHandler() (gin.HandlerFunc, newrelic.Application) {
newrelicConfig := newrelic.NewConfig("NEW_RELIC_APP_NAME", "NEW_RELIC_APP_KEY")
app, err := newrelic.NewApplication(newrelicConfig)
if err != nil {
return func(c *gin.Context) {
c.Next()
}, app
}
return func(c *gin.Context) {
txn := app.StartTransaction(c.Request.URL.Path, c.Writer, c.Request)
c.Set("newRelicTransaction", txn)
apm, ok := c.Get("newRelicTransaction")
defer txn.End()
c.Next()
}, app
}
func main() {
r := gin.New()
x, app := newrelicHandler()
r.Use(x)
}
// test/handler.go
func (t *TestHandler) Display(c *gin.Context) {
apmTxn, ok := c.Get("newRelicTransaction")
test, err := t.testCtrl.Display(apmTxn)
if err != nil {
return err
}
c.JSON(http.StatusOK, gin.H{"message": "success"})
}
// test/controller.go
func (t *TestCtrl) Display(txn interface{}) {
apmTxn, ok := txn.(newrelic.Transaction)
if !ok {
fmt.Println("ERR")
}
segment := newrelic.StartSegment(apmTxn, "SomeSegmentName")
// get data
segment.End()
return
}
Avoid using a global context, rather create one at the entrypoint and then just pass it as an argument to any function that needs it.
You can make use of the nrgin package provided by the Gin framework.
And in the main() function
Create an instance of newrelic - newrelic.NewApplication(cfg)
Call the - nrgin.Middleware(app) function passing in the newrelic instance. This will add the Gin transaction context key - newRelicTransaction to the context.
Register the function in step 2 as a middleware for all your routes - router.Use(nrgin.Middleware(app))
You can then pass this same context object to your other functions that can accept a parameter of type context.Context since gin.Context is simply implementing the context interface of Go.
Example code
import "github.com/newrelic/go-agent/_integrations/nrgin/v1"
func main() {
cfg := newrelic.NewConfig("Gin App", mustGetEnv("NEW_RELIC_LICENSE_KEY"))
app, err := newrelic.NewApplication(cfg)
if nil != err {
fmt.Println(err)
}
router := gin.Default()
router.Use(nrgin.Middleware(app))
router.GET("/example-get", GetController)
router.Run(":80")
}
func GetController(c *gin.Context) {
if txn := nrgin.Transaction(c); nil != txn {
txn.SetName("custom-name")
}
databaseGet(c)
c.Writer.WriteString("example response...")
}
func databaseGet(c context.Context) {
if txn := nrgin.Transaction(c); nil != txn {
txn.SetName("custom-name")
}
c.Writer.WriteString("example response...")
}

How to use mongo-driver connection into other packages

I am using Mongo-driver with gin framework. I have written code to connect mongodb in DB package and if I write query inside db/connect.go, it works but when I use same dbcon in other package it doesn't.
db/connect.go:
var dbcon *mongo.Database
func ConfigDB() (*mongo.Database) {
ctx := context.Background()
client, err := mongo.Connect(
ctx,
options.Client().ApplyURI("mongodb://localhost:27017/todo"),
)
if err != nil {
log.Fatal(err)
}
dbcon = client.Database("todo")
}
if I use the code below in same db/connect.go, then it works but when I use the same code in handler/task.go, then it won't.
func CreateTask() () {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
res, err := dbcon.Collection("ttest").InsertOne(ctx, bson.D{
{"task", "test4"},
{"createdAt", "test"},
{"modifiedAt","test3"},
})
if err != nil {
fmt.Println( err))
}
}
I have to implement a mongo-driver in my project, but due to above issue I am facing problem to implement.
You'll have to import to import the db/connect.go file into the handler/task.go. This is not working because they are in different packages.
In my opinion you could refactor your code like this
func ConfigDB() (*mongo.Database) {
ctx := context.Background()
client, err := mongo.Connect(
ctx,
options.Client().ApplyURI("mongodb://localhost:27017/todo"),
)
if err != nil {
log.Fatal(err)
}
return client.Database("todo")
}
import (
"db/connect"
)
func CreateTask() () {
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
res, err := ConfigDB().Collection("test").InsertOne(ctx, bson.D{
{"task", "test4"},
{"createdAt", "test"},
{"modifiedAt","test3"},
})
if err != nil {
fmt.Println( err))
}
}
Here I post a complete working example.
I catch the mongo session connection in a global variable. So that, anywhere in the project you can access the active connection.
package main
import (
"fmt"
"net/http"
"os"
"github.com/gin-gonic/gin"
"gopkg.in/mgo.v2"
"gopkg.in/mgo.v2/bson"
)
// SESSION ensure global mongodb connection
var SESSION *mgo.Session
func init() {
// mongodb manual connection using host ip. Replace your host IP address there
session, err := mgo.Dial("172.17.0.2")
// session, err := mgo.Dial("<HostIP>")
Must(err)
fmt.Println(err)
SESSION = session
}
func main() {
port := os.Getenv("PORT")
gin.SetMode(gin.ReleaseMode)
// gin.SetMode(gin.DebugMode)
r := gin.Default()
r.Use(mapMongo)
if port == "" {
port = "8000"
}
r.POST("/api/v1/task", CreateTask)
http.ListenAndServe(":"+port, r)
}
// close connection
func mapMongo(c *gin.Context) {
s := SESSION.Clone()
defer s.Close()
c.Set("mongo", s.DB("mongotask"))
c.Next()
}
// Must to catch the mongo panic issues
func Must(err error) {
if err != nil {
panic(err.Error())
}
}
// NewTask Struct/model
type NewTask struct {
Id bson.ObjectId `json:"_id,omitempty" bson:"_id,omitempty"`
Task string
}
// Mongo bson generate New unique Id each request
func (self *NewTask) Init() {
self.Id = bson.NewObjectId()
}
const (
// CollectionTask is the collection name
CollectionTask = "taskCollection"
)
// CreateTask to create new Task message
func CreateTask(c *gin.Context) {
var newTask NewTask
err := c.BindJSON(&newTask)
if err != nil {
c.Error(err)
return
}
mongodb := c.MustGet("mongo").(*mgo.Database)
con := mongodb.C(CollectionTask)
// fmt.Println(newTask)
con.Insert(newTask)
if err != nil {
c.Error(err)
}
}

Test wrap function with dependency injection

I have this function which I need to mock in test,
I was able to mock it as expected with http mock package , but now I’ve function that are calling
To the HttpReq method and here I cannot use http mock package
I read about dependency injection and tried something but I wasn’t able to fully do it,
This is the function
type params struct {
cs string
ci string
method string
url string
}
// I added this struct but not sure if it's needed ... probably for test purpose but not sure how to use it.
type Impl struct {
client *http.Client
}
func (i *Impl) HttpReq(p *params) ([]byte, error) {
httpClient := i.client
req, err := http.NewRequest(p.method, p.url, nil)
if err != nil {
fmt.Sprintf(err)
}
req.SetBasicAuth(p.cs, p.ci)
res, err := httpClient.Do(req)
if err != nil {
fmt.Sprintf(err)
}
t, err := ioutil.ReadAll(res.Body)
if err != nil {
fmt.Sprintf(err)
}
defer res.Body.Close()
return t, nil
}
This is what I tried
I’ve created interface
type Req interface {
HttpReq(params) ([]byte, error)
}
Now I’ve created a struct which contain the interface
type Service struct {
req Req
}
This is the function which I need to tests
func (c *Service) execute(cli Connection , args []string) (error, []byte) {
sk, err := c.doSomthing(cli, args)
sc, err := c.doSometing2(serviceK, []string{"url", "cl", "ct"})
cc := strings.Fields(serviceCredentials)
// ----------Here is what I need to mock ----------
t, err := c.req.HttpReq(params{cs: cc[1],
ci: cc[2],
method: http.MethodPost,
url: cc[0],})
return err, t
}
Any idea how I can run test for this function ??? Im struggling with it a lot.
Independent of the original question, you should not create new HTTP clients for each request. Client's maintain a connection pool and should be reused as much as possible.
You can fix that, and continue using your existing mock server by injecting the HTTP client.
Note also that the interface definition in the question doesn't match the implementation. These two method signatures are not the same:
HttpReq(params) ([]byte, error) // Req
HttpReq(*params) ([]byte, error) // Impl
Pick one. I would probably go with the non-pointer type here. And upper case initials are idiomatic in Go (HTTPReq, not HttpReq).
Add the client to the Impl type and use it in HTTPReq:
type Impl struct {
client *http.Client
}
func (i *Impl) HTTPReq(p params) ([]byte, error) {
req, err := http.NewRequest(p.method, p.url, nil)
if err != nil {
return nil, err
}
req.SetBasicAuth(p.cs, p.ci)
res, err := i.client.Do(req)
if err != nil {
return nil, err
}
defer res.Body.Close()
return ioutil.ReadAll(res.Body)
}
The Service type doesn't have to change.
In the tests, simply inject a test client into the Impl value:
import (
"context"
"net"
"net/http"
"net/http/httptest"
"testing"
)
func TestService_execute(t *testing.T) {
var testHandler http.Handler // TODO
srv := httptest.NewServer(testHandler)
defer srv.Close()
client := srv.Client()
tp := client.Transport.(*http.Transport)
// Direct all requests to the test server, no matter what the request URL is.
tp.DialContext = func(ctx context.Context, network, addr string) (net.Conn, error) {
// Note that we ignore the network and addr parameters, since these are
// derived from the request URL and are unrelated to the test server.
srvAddr := srv.Listener.Addr()
return (&net.Dialer{}).DialContext(ctx, srvAddr.Network(), srvAddr.String())
}
svc := &Service{
req: &Impl{
client: client,
},
}
svc.execute(/* ... */)
// assert request, response, etc.
}
Since Service struct already has an req interface, during tests initialise service object with mock that satisfies req interface.
Something like this
https://stackoverflow.com/a/53805535/3968921

rpc.ServerCodec Still Serving?

I was performing some RPC tests, and stumbled across a problem I can't seem to solve. In my testing I create three separate RPC servers, all of which I try to close and shutdown. However upon performing my last test (TestRpcCodecServerClientComm), it seems my client connection is connecting to the first RPC server I started (I know this because I at some point attached IDs to the RPCHandlers), even though I attempted everything I could to make sure it was shutdown. Though the code is not there I have attempted to inspect every single error I could, but that did not bring about anything.
rpc.go
package rbot
import (
"io"
"net"
"net/rpc"
"net/rpc/jsonrpc"
)
func RpcCodecClientWithPort(port string) (rpc.ClientCodec, error) {
conn, err := net.Dial("tcp", "localhost:"+port)
if err != nil {
return nil, err
}
return jsonrpc.NewClientCodec(conn), nil
}
func RpcCodecServer(conn io.ReadWriteCloser) rpc.ServerCodec {
return jsonrpc.NewServerCodec(conn)
}
rpc_test.go
package rbot
import (
"errors"
"fmt"
"net"
"net/rpc"
"testing"
)
type RPCHandler struct {
RPCServer net.Listener
conn rpc.ServerCodec
done chan bool
TestPort string
stop bool
GotRPC bool
}
func (r *RPCHandler) SetupTest() {
r.stop = false
r.GotRPC = false
r.done = make(chan bool)
r.TestPort = "5556"
}
// TODO: Create separate function to handle erroring
func (r *RPCHandler) CreateRPCServer() error {
rpc.RegisterName("TestMaster", TestAPI{r})
var err error
r.RPCServer, err = net.Listen("tcp", ":"+r.TestPort)
if err != nil {
return err
}
go func() {
for {
conn, err := r.RPCServer.Accept()
if err != nil || r.stop {
r.done <- true
return
}
r.conn = RpcCodecServer(conn)
rpc.ServeCodec(r.conn)
}
}()
return nil
}
func (r *RPCHandler) CloseRPCServer() error {
r.stop = true
if r.conn != nil {
err := r.conn.Close()
if err != nil {
fmt.Println(err)
}
}
err := r.RPCServer.Close()
<-r.done
return err
}
type TestAPI struct {
t *RPCHandler
}
func (tapi TestAPI) Send(msg string, result *string) error {
if msg == "Got RPC?" {
tapi.t.GotRPC = true
return nil
}
return errors.New("Didn't receive right message")
}
// Check if we can create and close an RPC server successfully using the RPC server codec.
func TestRpcCodecServer(t *testing.T) {
r := RPCHandler{}
r.SetupTest()
err := r.CreateRPCServer()
if err != nil {
t.Fatalf("Could not create rpc server! %s:", err.Error())
}
err = r.CloseRPCServer()
if err != nil {
t.Fatalf("Could not close RPC server! %s:", err.Error())
}
}
// Check if we can create a client without erroring.
func TestRpcCodecClientWithPortt(t *testing.T) {
r := RPCHandler{}
r.SetupTest()
r.CreateRPCServer()
defer r.CloseRPCServer()
RPCClient, err := RpcCodecClientWithPort(r.TestPort)
defer RPCClient.Close()
if err != nil {
t.Fatalf("Could not create an RPC client! %s:", err.Error())
}
}
// Let's double check and make sure our server and client can speak to each other
func TestRpcCodecServerClientComm(t *testing.T) {
r := RPCHandler{}
r.SetupTest()
r.CreateRPCServer()
defer r.CloseRPCServer()
RPCCodec, _ := RpcCodecClientWithPort(r.TestPort)
RPCClient := rpc.NewClientWithCodec(RPCCodec)
defer RPCClient.Close()
var result string
err := RPCClient.Call("TestMaster.Send", "Got RPC?", &result)
if err != nil {
t.Fatalf("Error while trying to send RPC message: %s", err.Error())
}
if !r.GotRPC {
t.Fatalf("Could not send correct message over RPC")
}
}
Not sure if I'm just mishandling the connection or something of the like, any help would be much appreciated.
For the Record The RPC api does receive the correct string message
While not the source of your problems, your test configuration has a few race conditions which you should take care of before they cause problems. Always check for issues with the -race option. You should also let the OS allocate the port so you don't run into conflicts. See for example how httptest.Server works.
Your failure here is that you're not creating a new rpc.Server for each test, you're reusing the rpc.DefaultServer. The first call to CreateRPCServer registers a TestAPI under the name TestMaster. Each subsequent call uses the already registered instance.
If you create a new rpc.Server each time you setup the test and register a new TestAPI, the final test will pass.
srv := rpc.NewServer()
srv.RegisterName("TestMaster", testAPI)
...
// and then use srv to handle the new connection
srv.ServeCodec(RpcCodecServer(conn))

Resources