How to mock GCP's storage in golang? - go

I'm really new to mocking third-party library in go, I'm mocking cloud.google.com/go/storage right now
I'm using mockery. This is my current interface:
//Client storage client
type Client interface {
Bucket(name string) BucketHandle
Buckets(ctx context.Context, projectID string) BucketIterator
}
//BucketHandle storage's BucketHandle
type BucketHandle interface {
Attrs(context.Context) (*storage.BucketAttrs, error)
Objects(context.Context, *storage.Query) ObjectIterator
}
//ObjectIterator storage's ObjectIterator
type ObjectIterator interface {
Next() (*storage.ObjectAttrs, error)
}
//BucketIterator storage's BucketIterator
type BucketIterator interface {
Next() (*storage.BucketAttrs, error)
}
and this is how I use it in my function
//Runner runner for this module
type Runner struct {
StorageClient stiface.Client
}
.... function
//get storage client
client, err := storage.NewClient(ctx)
if err != nil {
return err
}
runner := Runner{
StorageClient: client,
}
.... rest of functions
However, I got this error:
cannot use client (type *"cloud.google.com/go/storage".Client) as type stiface.Client in field value:
*"cloud.google.com/go/storage".Client does not implement stiface.Client (wrong type for Bucket method)
have Bucket(string) *"cloud.google.com/go/storage".BucketHandle
want Bucket(string) stiface.BucketHandle
What have I done wrong here? Thanks!
Edit
here's one example of the code that I want to mock. I'd like to mock on bucketIterator.Next():
//GetBuckets get list of buckets
func GetBuckets(ctx context.Context, client *storage.Client, projectName string) []checker.Resource {
//Get bucket iterator based on a project
bucketIterator := client.Buckets(ctx, projectName)
//iterate over the buckets and store bucket details
buckets := make([]checker.Resource, 0)
for bucket, done := bucketIterator.Next(); done == nil; bucket, done = bucketIterator.Next() {
buckets = append(buckets, checker.Resource{
Name: bucket.Name,
Type: "Bucket",
})
}
return buckets
}

The error message is basically saying your stiface.Client defines an interface that *storage.Client does not implement. On first glance your code looks valid however the problem lies in your interface method signatures and because they have outputs as interfaces.
Go makes a difference between the statements:
This function returns a BucketHandle
and this function returns a *storage.BucketHandle that is a BucketHandle
Try changing your interface to return the *storage.BucketHandle. You can see a more complex example of similar behaviour in the mockery S3API example where the functions return the s3 types, not their own interfaces.

After some trial and error, the way you'd use stiface is as below
If you need to mock stiface.BucketIterator, you can create a mock as
type mockBucketIterator struct {
stiface.BucketIterator
}
and mock the Next accordingly
func (m mockBucketIterator) Next() (*storage.BucketAttrs, error) {
// mocks that you need this to return
return
}
You could use the same method to mock all the way up to satiface.Client and feed the mock client to your test.
For reference, a full example in my tests:
type clientMock struct {
stiface.Client
}
type bucketMock struct {
stiface.BucketHandle
}
type objectItMock struct {
stiface.ObjectIterator
i int
next []storage.ObjectAttrs
}
func (m clientMock) Bucket(name string) stiface.BucketHandle {
return bucketMock{}
}
and then the object iterator to return mocked iterator as well
func (it *objectItMock) Next() (a *storage.ObjectAttrs, err error) {
if it.i == len(it.next) {
err = iterator.Done
return
}
a = &it.next[it.i]
it.i += 1
return
}
func (m bucketMock) Objects(ctx context.Context, q *storage.Query) (it stiface.ObjectIterator) {
it = &objectItMock{
i: 0,
next: []storage.ObjectAttrs{
{Name: "abc"},
{Name: "def"},
{Name: "ghi"},
},
}
return
}

Related

Generic function for fetching different models in REST API

I am writing a library in Go for using the Strava API. It's a simple API to expose the various objects (athlete, activity, and so on) that make up Strava's data. I am struggling to come up with a way that separates the mechanics of making a request so it can be reused to fetch the various different objects in the API. What I have so far:
type Model interface {
Url() *url.URL
Data() interface{} // pointer to location of unmarshaled response
}
// an activity (run, bike ride, etc)
type Activity struct {
Id int64 `json:"id"`
Name string `json:"name"`
Distance float64 `json:"distance"`
// ...
}
func (a *Activity) Url() *url.URL {
return url.Parse(fmt.Sprintf("https://www.strava.com/api/v3/activities/%d", a.Id))
}
func (a *Activity) Data() interface{} {
return a
}
// gear (shoes, bike, etc)
type Gear struct {
Id string `json:"id"`
Name string `json:"name"`
}
func (g *Gear) Url() *url.URL {
return url.Parse(fmt.Sprintf("https://www.strava.com/api/v3/gear/%s", g.Id))
}
func (g *Gear) Data() interface{} {
return g
}
// a page of activities
type ActivityPage struct {
AthleteId int64
PageNum int
Activities []Activity
}
func (p *ActivityPage) Url() *url.URL {
return url.Parse(fmt.Sprintf("https://www.strava.com/api/v3/athletes/%d/activities?page=%d&per_page=%d", p.AthleteId, p.PageNum, perPage))
}
func (p *ActivityPage) Data() interface{} {
return &p.Activities
}
type Client struct {
hc *http.Client
}
// error handling omitted
func (c *Client) fetch(m Model) error {
data, _ := c.fetchUrl(m.Url())
json.Unmarshal(data, m.Data())
return nil
}
func (c *Client) fetchUrl(u *url.URL) ([]byte, error) {
req := &http.Request{
// omit access token
Method: "GET",
URL: u,
}
resp, _ := c.hc.Do(req)
defer resp.Body.Close()
return ioutil.ReadAll(resp.Body)
}
The Data() pointer is needed because the GET /athlete/activities endpoint returns a list of Activitys, rather than a specific model. ActivityPage is therefore a bit of a hack — an object that carries the data needed to build the URL along with a place to put the results. For cases where a GET returns a model the Data() pointer is just the object.
With this code, I can do:
client := Client{}
activity := Activity{Id: 1234}
client.fetch(activity)
fmt.Print(activity.Name)
page := ActivityPage{AthleteId: 1, PageNum: 1}
client.fetch(page)
fmt.Print(len(page.Activities))
But this feels.. icky. I don't like partially constructing the object and passing it to fetch() to be finished off, or that fetch doesn't actually return anything except an error on failure. The Data() pointer is a hack.
AIUI, interfaces are a way to write code that can work with objects of different types, but I feel like I want the inverse — to have some code (a Fetch() method or something) that is inherited by all objects with a certain trait.
How can I make this cleaner? I realise this is kind of open-ended so I'm more than happy to refine what the exact question is as appropriate. Are there canonical examples of building a REST client in Go? (I haven't found anything compelling so far)
this is a standard workaround typical for go, you pass a pointer to data you want to be modified by reflection, standard lib is built like that, you can at most make method to accept pointer to data and url directly to be more verbose and not interface. It will make it at least match cleaner what will get modified just from looking at api calls for user like:
func (c *Client) fetch(url string, responceBuffer interface{}) error {
data, err := c.fetchUrl(url)
if err != nil {
return err
}
return json.Unmarshal(data, responceBuffer)
}

Is this dependecy injection pattern thread safe?

I'm having a hard time coming up with a clean pattern to inject dependencies in a REST server that allows me to write isolated unit tests. The below structure seems to work but I'm not sure if it's thread safe.
store:
package store
type InterfaceStore interface {
Connect()
Disconnect()
User() interfaceUser
}
// Wiring up
type store struct {
db *mongo.Database
}
func (s *store) Connect() {
client, err := mongo.Connect()
if err != nil {
log.Fatal(err.Error())
}
s.db = client.Database()
}
func (s *store) Disconnect() {
s.db.Client().Disconnect(context.TODO())
}
func (s *store) User() interfaceUser {
return &user{s.db}
}
// Exposed from the package to create a store instance
func GetStore() InterfaceStore {
return &store{}
}
// User related
type interfaceUser interface {
InsertOne(models.User) (string, error)
}
type user struct {
db *mongo.Database
}
func (u *user) InsertOne(user models.User) (primitive.ObjectID, error) {
collection := u.db.Collection(collectionUsers)
// persisting user in DB
}
server:
package server
type server struct{}
func (s *server) Start() {
storeInstance := store.GetStore()
storeInstance.Connect()
defer storeInstance.Disconnect()
r := gin.Default()
keys := keys.GetKeys()
routes.InitRoutes(r, storeInstance)
port := fmt.Sprintf(":%s", keys.PORT)
r.Run(port)
}
func CreateInstance() *server {
return &server{}
}
routes:
package routes
func InitRoutes(router *gin.Engine, store store.InterfaceStore) {
router.Use(middlewares.Cors)
// createSubrouter creates a Gin routerGroup with the prefix "/user"
userRoutes(createSubrouter("/user", router), store)
}
func userRoutes(router *gin.RouterGroup, store store.InterfaceStore) {
controller := controllers.GetUserController(store)
router.GET("/", controller.Get)
}
controllers:
package controllers
type userControllers struct {
UserService services.InterfaceUser
}
func (u *userControllers) Get(c *gin.Context) {
userDetails, _ := u.UserService.FetchAllInformation(bson.M{"_id": userData.(models.User).ID})
utils.RespondWithJSON(c, userDetails)
}
func GetUserController(store store.InterfaceStore) userControllers {
userService := services.GetUserService(store)
return userControllers{
UserService: &userService,
}
}
services:
package services
type InterfaceUser interface {
FetchAllInformation(bson.M) (*models.User, error)
}
type user struct {
store store.InterfaceStore
}
func (u *user) FetchAllInformation(filter bson.M) (*models.User, error) {
user, err := u.store.User().FindOne(filter)
if err != nil {
return nil, err
}
return user, nil
}
func GetUserService(store store.InterfaceStore) user {
return user{
store: store,
}
}
By using interfaces I'm able to mock the entire service when writing tests for the controller and I can mock the entire store to test the service component without hitting the DB.
I'm wondering if the store instance is safely shared across the code because the interfaces are no pointers. Does that mean a copy of the store is created every time I pass it down the tree?
The type user struct {} definition states store is anything that implements the store.InterfaceStore interface.
If you look carefully, you're implementing it with pointer receivers. That means the (instance pointed by the) receiver will be shared.
If your mock implements them over the value-type it will be copied on method call and you'll be safe, but it will also mean this mock won't be holding new state after the method calls, which I guess is not what you want.
Bottom line, it's not really about how you defined it in the struct, by value or by reference, but what the methods accept as receiver.

Return polymorphic type with data member

I'm trying to write a function getTargetServer() to return a polymorphic type that has both a data member URL and a method Close(). This would be a generalization of the *Server returned from httptest.NewServer() but I want to alternatively be able to return a custom type for which Close() is a NOP.
type externalTestServer struct {
URL string
}
func (externalTestServer) Close() {}
func getTargetServer() *externalTestServer {
if urlbase, ok := optionals["urlbase"].(string); ok {
return &externalTestServer{URL: urlbase}
} else {
testServer := httptest.NewServer(newMyServerHandler())
// return testServer // ### Error ###
return &externalTestServer{URL: testServer.URL}
}
}
func Test_health_check(t *testing.T) {
testServer := getTargetServer()
defer testServer.Close()
response, err := http.Get(testServer.URL + "/health")
assert.NilError(t, err)
assert.Assert(t, cmp.Equal(response.StatusCode, http.StatusOK))
}
This works like a charm except that Close() is always a NOP. When I uncomment the indicated ### Error ### line in order to return a closable *Server, I get the following error message:
cannot use testServer (type *httptest.Server) as type *externalTestServer in return argument
I understand the error, but haven't discovered a solution that lets me return a polymorphic type that generalizes *Server
Note: "A Tour of Go" defines an interface type as follows:
An interface type is defined as a set of method signatures.
Therefore, returning a simple interface will not allow for directly-accessible data members.
You could create a struct that has a field which is a string URL and a field that is a Close func. The Close func can be implemented by either externalTestServer or httptest.Server:
type server struct {
URL string
Close func()
}
if urlbase, ok := optionals["urlbase"].(string); ok {
extServer := &externalTestServer{URL: urlbase}
return &server{
URL: urlbase,
Close: extServer.Close,
}
}
testServer := httptest.NewServer(newMyServerHandler())
return &server{
URL: testServer.URL,
Close: testServer.Close,
}
http.Server is a struct, so you cannot return a polymorphic object that generalizes that. You can do something else though:
type Server interface {
GetURL() string
Close() error
}
type testServer struct {
URL string
}
func (t testServer) Close() error {}
func (t testServer) GetURL() string {return t.URL}
type httpServer struct {
*http.Server
}
func (t httpServer) GetURL() string { return the url }
You can then return Server from your function.
Chris Drew's approach is ideal for this specific case, because it requires minimal code overhead. Put another way, it is the simplest thing that will solve today's problem. The solution uses the magic of an implicit closure (for the receiver) to reference the implementation of Close() polymorphically.
My slightly simplified version of that is here...
type genericServer struct {
URL string // snapshot of internal data
Close func() // single-function polymorphism without 'interface'
}
func getTargetServer() genericServer {
if urlbase, ok := optionals["urlbase"].(string); ok {
return genericServer{URL: urlbase, Close: func() {}}
}
testServer := httptest.NewServer(newMyServerHandler())
return genericServer{URL: testServer.URL, Close: testServer.Close}
}
By embedding an actual interface type into to return struct, this concept can be seamlessly extended to better support non-trivial polymorphic interfaces. In this case, the Polymorphism is explicit based on the internal use of an interface type. Still, the returned type is a wrapper that includes a copy of the (constant) member data, so the usage is identical -- effectively generalizing that of *Server for this use-case.
type Server interface {
Close()
}
type nopServer struct {
}
func (nopServer) Close() {}
type genericServer struct {
URL string // snapshot of specific data
Server // embedded interface type for explicit polymorphism
}
func getTargetServer() genericServer {
if urlbase, ok := optionals["urlbase"].(string); ok {
return genericServer{URL: urlbase, Server: nopServer{}}
}
testServer := httptest.NewServer(newMyServerHandler())
return genericServer{URL: testServer.URL, Server: testServer}
}
Note that the URL value is not a live member of the implementation, so these solutions, as presented, are only meaningful when the data value will not change, although perhaps that limitation could be overcome by using a pointer.

How to use Go's type alias to make own models work with protobufs?

I've got some REST API with my models defined as Go structs.
type User struct {
FirstName string
LastName string
}
Then I've got my database methods for getting data.
GetUserByID(id int) (*User, error)
Now I'd like to replace my REST API with https://github.com/twitchtv/twirp .
Therefore I started defining my models inside .proto files.
message User {
string first_name = 2;
string last_name = 3;
}
Now I've got two User types. Let's call them the native and the proto type.
I've also got a service defined in my .proto file which returns a user to the frontend.
service Users {
rpc GetUser(Id) returns (User);
}
This generates an interface that I have to fill in.
func (s *Server) GetUser(context.Context, id) (*User, error) {
// i'd like to reuse my existing database methods
u, err := db.GetUserByID(id)
// handle error
// do more stuff
return u, nil
}
Unfortunately this does not work. My database returns a native User but the interface requires a proto user.
Is there an easy way to make it work? Maybe using type aliases?
Thanks a lot!
One way you can solve your problem is by doing the conversion manually.
type User struct {
FirstName string
LastName string
}
type protoUser struct {
firstName string
lastName string
}
func main() {
u := db() // Retrieve a user from a mocked db
fmt.Println("Before:")
fmt.Printf("%#v\n", *u) // What db returns (*protoUser)
fmt.Println("After:")
fmt.Printf("%#v\n", u.AsUser()) // What conversion returns (User)
}
// Mocked db that returns pointer to protoUser
func db() *protoUser {
pu := protoUser{"John", "Dough"}
return &pu
}
// Conversion method (converts protoUser into a User)
func (pu *protoUser) AsUser() User {
return User{pu.firstName, pu.lastName}
}
The key part is the AsUser method on the protoUser struct.
There we simply write our custom logic for converting a protoUser into a User type we want to be working with.
Working Example
As #Peter mentioned in the comment section.
I've seen a project which made it with a custom Convert function. It converts the Protobuf to local struct via json.Unmarshal, not sure how's the performance but it's a way to go.
Preview Code PLAYGROUND
// Convert converts the in struct to out struct via `json.Unmarshal`
func Convert(in interface{}, out interface{}) error {
j, err := json.Marshal(in)
if err != nil {
return err
}
err = json.Unmarshal(j, &out)
if err != nil {
return err
}
return nil
}
func main() {
// Converts the protobuf struct to local struct via json.Unmarshal
var localUser User
if err := convert(protoUser, &localUser); err != nil {
panic(err)
}
}
Output
Before:
main.ProtoUser{FirstName:"John", LastName:"Dough"}
After:
main.User{FirstName:"John", LastName:"Dough"}
Program exited.

Typecasting one function into another in golang

I have the following code:
package vault
type Client interface {
GetHealth() error
}
func (c DefaultClient) GetHealth () error {
resp := &VaultHealthResponse{}
err := c.get(resp, "/v1/sys/health")
if err != nil {
return err
}
return nil;
}
Now, I want to use this function as part of this struct:
type DependencyHealthFunction func() error
type Dependency struct {
Name string `json:"name"`
Required bool `json:"required"`
Healthy bool `json:"healthy"`
Error error `json:"error,omitempty"`
HealthFunction DependencyHealthFunction
}
Basically, set the value of HealthFunction to GetHealth. Now, when I do the following:
func (config *Config) GetDependencies() *health.Dependency {
vaultDependency := health.Dependency{
Name: "Vault",
Required: true,
Healthy: true,
HealthFunction: vault.Client.GetHealth,
}
temp1 := &vaultDependency
return temp1;
}
This gives me an error and it says cannot use vault.Client.GetHealth (type func(vault.Client) error) as type health.DependencyHealthFunction in field value. How can I do this?
Edit: How DependencyHealthFunction is used?
As its part of Dependency struct, it's simply used as following: d.HealthFunction() where d is a variable of type *Dependency.
This is abstract:
HealthFunction: vault.Client.GetHealth,
If we were to call HealthFunction(), what code do you expect to run? vault.Client.GetHealth is just a promise that such a function exists; it isn't a function itself. Client is just an interface.
You need to create something that conforms to Client and pass its GetHealth. For example, if you had a existing DefaultClient such as:
defaultClient := DefaultClient{}
Then you could pass its function:
HealthFunction: defaultClient.GetHealth,
Now when you later call HealthFunction() it will be the same as calling defaultClient.GetHealth().
https://play.golang.org/p/9Lw7uc0GaE
I believe the issue is related to understanding how interfaces are treated in Go.
An interface simply defines a method or set of methods that a particular type must satisfy to be considered as "implementing" the interface.
For example:
import "fmt"
type Greeter interface {
SayHello() string
}
type EnglishGreeter struct{}
// Satisfaction of SayHello method
func (eg *EnglishGreeter) SayHello() string {
return "Hello"
}
type SpanishGreeter struct{}
func (sg *SpanishGreeter) SayHello() string {
return "Ola"
}
func GreetPerson(g Greeter) {
fmt.Println(g.SayHello())
}
func main() {
eg := &EnglishGreeter{}
sg := &SpanishGreeter{}
// greet person in english
GreetPerson(eg)
// greet person in spanish
GreetPerson(sg)
}
You can add this behavior into a custom struct by simply having a Greeter field inside the struct. ie
type FrontEntrance struct {
EntranceGreeter Greeter
}
fe := &FrontEntrance { EntranceGreeter: &EnglishGreeter{} }
// then call the SayHello() method like this
fe.EntranceGreeter.SayHello()
Interfaces in golang are useful for composing common expected behavior for types based on the methods that they satisfy.
Hope this helps.

Resources