I am writing a library in Go for using the Strava API. It's a simple API to expose the various objects (athlete, activity, and so on) that make up Strava's data. I am struggling to come up with a way that separates the mechanics of making a request so it can be reused to fetch the various different objects in the API. What I have so far:
type Model interface {
Url() *url.URL
Data() interface{} // pointer to location of unmarshaled response
}
// an activity (run, bike ride, etc)
type Activity struct {
Id int64 `json:"id"`
Name string `json:"name"`
Distance float64 `json:"distance"`
// ...
}
func (a *Activity) Url() *url.URL {
return url.Parse(fmt.Sprintf("https://www.strava.com/api/v3/activities/%d", a.Id))
}
func (a *Activity) Data() interface{} {
return a
}
// gear (shoes, bike, etc)
type Gear struct {
Id string `json:"id"`
Name string `json:"name"`
}
func (g *Gear) Url() *url.URL {
return url.Parse(fmt.Sprintf("https://www.strava.com/api/v3/gear/%s", g.Id))
}
func (g *Gear) Data() interface{} {
return g
}
// a page of activities
type ActivityPage struct {
AthleteId int64
PageNum int
Activities []Activity
}
func (p *ActivityPage) Url() *url.URL {
return url.Parse(fmt.Sprintf("https://www.strava.com/api/v3/athletes/%d/activities?page=%d&per_page=%d", p.AthleteId, p.PageNum, perPage))
}
func (p *ActivityPage) Data() interface{} {
return &p.Activities
}
type Client struct {
hc *http.Client
}
// error handling omitted
func (c *Client) fetch(m Model) error {
data, _ := c.fetchUrl(m.Url())
json.Unmarshal(data, m.Data())
return nil
}
func (c *Client) fetchUrl(u *url.URL) ([]byte, error) {
req := &http.Request{
// omit access token
Method: "GET",
URL: u,
}
resp, _ := c.hc.Do(req)
defer resp.Body.Close()
return ioutil.ReadAll(resp.Body)
}
The Data() pointer is needed because the GET /athlete/activities endpoint returns a list of Activitys, rather than a specific model. ActivityPage is therefore a bit of a hack — an object that carries the data needed to build the URL along with a place to put the results. For cases where a GET returns a model the Data() pointer is just the object.
With this code, I can do:
client := Client{}
activity := Activity{Id: 1234}
client.fetch(activity)
fmt.Print(activity.Name)
page := ActivityPage{AthleteId: 1, PageNum: 1}
client.fetch(page)
fmt.Print(len(page.Activities))
But this feels.. icky. I don't like partially constructing the object and passing it to fetch() to be finished off, or that fetch doesn't actually return anything except an error on failure. The Data() pointer is a hack.
AIUI, interfaces are a way to write code that can work with objects of different types, but I feel like I want the inverse — to have some code (a Fetch() method or something) that is inherited by all objects with a certain trait.
How can I make this cleaner? I realise this is kind of open-ended so I'm more than happy to refine what the exact question is as appropriate. Are there canonical examples of building a REST client in Go? (I haven't found anything compelling so far)
this is a standard workaround typical for go, you pass a pointer to data you want to be modified by reflection, standard lib is built like that, you can at most make method to accept pointer to data and url directly to be more verbose and not interface. It will make it at least match cleaner what will get modified just from looking at api calls for user like:
func (c *Client) fetch(url string, responceBuffer interface{}) error {
data, err := c.fetchUrl(url)
if err != nil {
return err
}
return json.Unmarshal(data, responceBuffer)
}
Related
I'm trying to write a "Binder" middleware that will validate any request query using a struct type with gin bindings/validators
So for example, let's say I have an endpoint group called /api/subject which requires the query string to have a subject code and an ID that will be validated using the following struct (called entity.Subject):
type Subject struct {
Code string `binding:"required,alphanum"`
ID string `binding:"required,alphanum,len=4"`
}
That's just one example, but I'd like to be able to pass any struct type to this middleware, because I'd like to access the query data on future handlers without worrying about query validation.
So I tried something like this:
func Binder(t reflect.Type) gin.HandlerFunc {
return func(c *gin.Context) {
obj := reflect.New(t).Elem().Interface()
if err := c.BindQuery(&obj); err != nil {
c.AbortWithStatus(http.StatusBadRequest)
return
}
c.Set(t.Name(), obj)
}
}
And added this middleware like so:
apiGroup := router.Group("/api")
{
// other subgroups/endpoints
// ...
subjectGroup := apiGroup.Group("/subject", middleware.Binder(reflect.TypeOf(entity.Subject{})))
}
And later on, in another handler function, let's say GetSubject, I want to access the subject data passed by doing c.MustGet("Subject").(entity.Subject)
But this isn't working =(, when I print obj, it's just an empty interface, how would I do this?
I managed to do something similar!
I created the following middleware
var allowedTypes = []binding.Binding{
binding.Query,
binding.Form,
binding.FormPost,
binding.FormMultipart,
}
func Bind(name string, data interface{}, bindingType binding.Binding) gin.HandlerFunc {
return func(ctx *gin.Context) {
ok := false
for _, b := range allowedTypes {
if b == bindingType {
ok = true
}
}
if !ok {
ctx.AbortWithError(
http.StatusInternalServerError,
fmt.Errorf("Bind function only allows %v\n", allowedTypes),
)
}
_ = ctx.MustBindWith(data, bindingType)
ctx.Set(name, data)
}
}
Remember to pass a pointer to your desired type in the call, like so:
router.GET("/something", Bind("Object", &myObject, binding.Query))
I restricted only to a few binding types because they allow ShouldBind to be called multiple times, whereas JSON, XML and others consume the Request body.
This way you can pass multiple Bind middlewares and if the validation fails it automatically aborts with http.StatusBadRequest
I would like to pass a function pointer to a function to "anything".
It's easy to print something that gets passed in from just about anything (as in https://play.golang.org/p/gmOy6JWxGm0):
func printStuff(stuff interface{}) {
fmt.Printf("Testing : %v", stuff)
}
Let's say, though, that I want to do this:
Have multiple structs
Have data loaded from various functions
Have a generic print that calls the function for me
I tried this in a Play (https://play.golang.org/p/l3-OkL6tsMW) and I get the following errors:
./prog.go:35:12: cannot use getStuff1 (type func() SomeObject) as type FuncType in argument to printStuff
./prog.go:36:12: cannot use getStuff2 (type func() SomeOtherObject) as type FuncType in argument to printStuff
In case the Play stuff gets deleted, here's the code I'm trying to figure out how to get to work:
package main
import (
"fmt"
)
type SomeObject struct {
Value string
}
type SomeOtherObject struct {
Value string
}
type FuncType func() interface{}
func getStuff1() SomeObject {
return SomeObject{
Value: "Hello, world!",
}
}
func getStuff2() SomeOtherObject {
return SomeOtherObject{
Value: "Another, hello!",
}
}
func printStuff(toCall FuncType) {
stuff := toCall()
fmt.Printf("Testing : %v", stuff)
}
func main() {
printStuff(getStuff1)
printStuff(getStuff2)
}
What is the secret sauce to get this stuff passed in properly?
Larger Goal Explanation
So what I am trying to accomplish here is reduction of boilerplate code that lives inside a gigantic file. Unfortunately I cannot refactor it further at this point due to other restrictions and I was wondering if this were possible at all considering the error messages and what I had read seemed to dictate otherwise.
There's a large amount of copy-and-paste code that looks like this:
func resendContraDevice(trap *TrapLapse, operation *TrapOperation) {
loaded := contra.Load()
err := trap.SnapBack(operation).send(loaded);
// default error handling
// logging
// boilerplate post-process
}
func resendPolicyDevice(trap *TrapLapse, operation *TrapOperation) {
loaded := policy.Load()
err := trap.SnapBack(operation).send(loaded);
// default error handling
// logging
// boilerplate post-process
}
// etc.
In these, the Load() functions all return a different struct type and they are used elsewhere throughout the application.
I want hoping to get something where I could have:
loaded := fn()
err := trap.SnapBack(operation).send(loaded);
// default error handling
// logging
// boilerplate post-process
Signature for send is, which accepts an interface{} argument:
func (s SnapBack) send(data interface{}) error
I don't know if you have control over the return values of contra.Load() and policy.Load(), for instance, so there may be a better approach, but assuming those cannot be modified, this would allow you to eliminate a lot of boilerplate, without any fancy manipulation:
func boilerplate(tram *TrapLapse, operation *TrapOperation, loader func() interface{}) {
loaded := loader()
err := trap.SnapBack(operation).send(loaded);
// default error handling
// logging
// boilerplate post-process
}
func resendContraDevice(trap *TrapLapse, operation *TrapOperation) {
boilerplate(trap, operation, func() interface{} { return contra.Load() })
}
func resendPolicyDevice(trap *TrapLapse, operation *TrapOperation) {
boilerplate(trap, operation, func() interface{} { return policy.Load() })
}
If there's nothing more complex, you can also simplify this even further:
func boilerplate(tram *TrapLapse, operation *TrapOperation, loaded interface{}) {
err := trap.SnapBack(operation).send(loaded);
// default error handling
// logging
// boilerplate post-process
}
func resendContraDevice(trap *TrapLapse, operation *TrapOperation) {
boilerplate(trap, operation, contra.Load())
}
func resendPolicyDevice(trap *TrapLapse, operation *TrapOperation) {
boilerplate(trap, operation, policy.Load())
}
I'm working on my first Go application, and I'm trying to keep my files well organized, DRY, etc. I have a struct that represents the data I expect to receive from my SQL queries, defined in a closure environment that returns my request handlerFunc. Pared down version:
func (s *server) getComponents() http.HandlerFunc {
type component struct {
ID int `json:"_id,omitempty"`
Name string `json:"name"`
// ... many more fields
}
return func(res http.ResponseWriter, req *http.Request) {
rows, err := s.db.GetComponents()
defer rows.Close()
if err != nil {
s.testQueryError(err, "etc, etc")
return
}
var components []component
for rows.Next() {
var c component
rows.Scan(&c)
components = append(components, c)
}
data, _ := json.Marshal(components)
res.Write(data)
}
}
This implementation responds with the expected data. However, I expected that I may need this struct in other routes, and thought that it would be good practice to define this struct elsewhere and export it from a "models" directory/package. New attempt:
// app/models/models.go
package "models"
type Component struct {
ID int `json:"_id,omitempty"`
// ...
}
// app/routes.go
import "app/models"
func (s *server) getComponents() http.HandlerFunc {
return func(res http.ResponseWriter, req *http.Request) {
rows, err := s.db.GetComponents()
// ...
var components []models.Component
for rows.Next() {
var c models.Component
rows.Scan(&c)
components = append(components, c)
}
data, _ := json.Marshal(components)
res.Write(data)
}
}
This version does not properly Scan, and Postman receives all empty fields. I'm new to strongly typed languages like Go, so could anybody point me in the right direction on where I am going wrong? The package is being properly exported/imported, so why should it work any differently?
I'm really new to mocking third-party library in go, I'm mocking cloud.google.com/go/storage right now
I'm using mockery. This is my current interface:
//Client storage client
type Client interface {
Bucket(name string) BucketHandle
Buckets(ctx context.Context, projectID string) BucketIterator
}
//BucketHandle storage's BucketHandle
type BucketHandle interface {
Attrs(context.Context) (*storage.BucketAttrs, error)
Objects(context.Context, *storage.Query) ObjectIterator
}
//ObjectIterator storage's ObjectIterator
type ObjectIterator interface {
Next() (*storage.ObjectAttrs, error)
}
//BucketIterator storage's BucketIterator
type BucketIterator interface {
Next() (*storage.BucketAttrs, error)
}
and this is how I use it in my function
//Runner runner for this module
type Runner struct {
StorageClient stiface.Client
}
.... function
//get storage client
client, err := storage.NewClient(ctx)
if err != nil {
return err
}
runner := Runner{
StorageClient: client,
}
.... rest of functions
However, I got this error:
cannot use client (type *"cloud.google.com/go/storage".Client) as type stiface.Client in field value:
*"cloud.google.com/go/storage".Client does not implement stiface.Client (wrong type for Bucket method)
have Bucket(string) *"cloud.google.com/go/storage".BucketHandle
want Bucket(string) stiface.BucketHandle
What have I done wrong here? Thanks!
Edit
here's one example of the code that I want to mock. I'd like to mock on bucketIterator.Next():
//GetBuckets get list of buckets
func GetBuckets(ctx context.Context, client *storage.Client, projectName string) []checker.Resource {
//Get bucket iterator based on a project
bucketIterator := client.Buckets(ctx, projectName)
//iterate over the buckets and store bucket details
buckets := make([]checker.Resource, 0)
for bucket, done := bucketIterator.Next(); done == nil; bucket, done = bucketIterator.Next() {
buckets = append(buckets, checker.Resource{
Name: bucket.Name,
Type: "Bucket",
})
}
return buckets
}
The error message is basically saying your stiface.Client defines an interface that *storage.Client does not implement. On first glance your code looks valid however the problem lies in your interface method signatures and because they have outputs as interfaces.
Go makes a difference between the statements:
This function returns a BucketHandle
and this function returns a *storage.BucketHandle that is a BucketHandle
Try changing your interface to return the *storage.BucketHandle. You can see a more complex example of similar behaviour in the mockery S3API example where the functions return the s3 types, not their own interfaces.
After some trial and error, the way you'd use stiface is as below
If you need to mock stiface.BucketIterator, you can create a mock as
type mockBucketIterator struct {
stiface.BucketIterator
}
and mock the Next accordingly
func (m mockBucketIterator) Next() (*storage.BucketAttrs, error) {
// mocks that you need this to return
return
}
You could use the same method to mock all the way up to satiface.Client and feed the mock client to your test.
For reference, a full example in my tests:
type clientMock struct {
stiface.Client
}
type bucketMock struct {
stiface.BucketHandle
}
type objectItMock struct {
stiface.ObjectIterator
i int
next []storage.ObjectAttrs
}
func (m clientMock) Bucket(name string) stiface.BucketHandle {
return bucketMock{}
}
and then the object iterator to return mocked iterator as well
func (it *objectItMock) Next() (a *storage.ObjectAttrs, err error) {
if it.i == len(it.next) {
err = iterator.Done
return
}
a = &it.next[it.i]
it.i += 1
return
}
func (m bucketMock) Objects(ctx context.Context, q *storage.Query) (it stiface.ObjectIterator) {
it = &objectItMock{
i: 0,
next: []storage.ObjectAttrs{
{Name: "abc"},
{Name: "def"},
{Name: "ghi"},
},
}
return
}
I've got some REST API with my models defined as Go structs.
type User struct {
FirstName string
LastName string
}
Then I've got my database methods for getting data.
GetUserByID(id int) (*User, error)
Now I'd like to replace my REST API with https://github.com/twitchtv/twirp .
Therefore I started defining my models inside .proto files.
message User {
string first_name = 2;
string last_name = 3;
}
Now I've got two User types. Let's call them the native and the proto type.
I've also got a service defined in my .proto file which returns a user to the frontend.
service Users {
rpc GetUser(Id) returns (User);
}
This generates an interface that I have to fill in.
func (s *Server) GetUser(context.Context, id) (*User, error) {
// i'd like to reuse my existing database methods
u, err := db.GetUserByID(id)
// handle error
// do more stuff
return u, nil
}
Unfortunately this does not work. My database returns a native User but the interface requires a proto user.
Is there an easy way to make it work? Maybe using type aliases?
Thanks a lot!
One way you can solve your problem is by doing the conversion manually.
type User struct {
FirstName string
LastName string
}
type protoUser struct {
firstName string
lastName string
}
func main() {
u := db() // Retrieve a user from a mocked db
fmt.Println("Before:")
fmt.Printf("%#v\n", *u) // What db returns (*protoUser)
fmt.Println("After:")
fmt.Printf("%#v\n", u.AsUser()) // What conversion returns (User)
}
// Mocked db that returns pointer to protoUser
func db() *protoUser {
pu := protoUser{"John", "Dough"}
return &pu
}
// Conversion method (converts protoUser into a User)
func (pu *protoUser) AsUser() User {
return User{pu.firstName, pu.lastName}
}
The key part is the AsUser method on the protoUser struct.
There we simply write our custom logic for converting a protoUser into a User type we want to be working with.
Working Example
As #Peter mentioned in the comment section.
I've seen a project which made it with a custom Convert function. It converts the Protobuf to local struct via json.Unmarshal, not sure how's the performance but it's a way to go.
Preview Code PLAYGROUND
// Convert converts the in struct to out struct via `json.Unmarshal`
func Convert(in interface{}, out interface{}) error {
j, err := json.Marshal(in)
if err != nil {
return err
}
err = json.Unmarshal(j, &out)
if err != nil {
return err
}
return nil
}
func main() {
// Converts the protobuf struct to local struct via json.Unmarshal
var localUser User
if err := convert(protoUser, &localUser); err != nil {
panic(err)
}
}
Output
Before:
main.ProtoUser{FirstName:"John", LastName:"Dough"}
After:
main.User{FirstName:"John", LastName:"Dough"}
Program exited.