I'm working on my first Go application, and I'm trying to keep my files well organized, DRY, etc. I have a struct that represents the data I expect to receive from my SQL queries, defined in a closure environment that returns my request handlerFunc. Pared down version:
func (s *server) getComponents() http.HandlerFunc {
type component struct {
ID int `json:"_id,omitempty"`
Name string `json:"name"`
// ... many more fields
}
return func(res http.ResponseWriter, req *http.Request) {
rows, err := s.db.GetComponents()
defer rows.Close()
if err != nil {
s.testQueryError(err, "etc, etc")
return
}
var components []component
for rows.Next() {
var c component
rows.Scan(&c)
components = append(components, c)
}
data, _ := json.Marshal(components)
res.Write(data)
}
}
This implementation responds with the expected data. However, I expected that I may need this struct in other routes, and thought that it would be good practice to define this struct elsewhere and export it from a "models" directory/package. New attempt:
// app/models/models.go
package "models"
type Component struct {
ID int `json:"_id,omitempty"`
// ...
}
// app/routes.go
import "app/models"
func (s *server) getComponents() http.HandlerFunc {
return func(res http.ResponseWriter, req *http.Request) {
rows, err := s.db.GetComponents()
// ...
var components []models.Component
for rows.Next() {
var c models.Component
rows.Scan(&c)
components = append(components, c)
}
data, _ := json.Marshal(components)
res.Write(data)
}
}
This version does not properly Scan, and Postman receives all empty fields. I'm new to strongly typed languages like Go, so could anybody point me in the right direction on where I am going wrong? The package is being properly exported/imported, so why should it work any differently?
Related
I am writing a library in Go for using the Strava API. It's a simple API to expose the various objects (athlete, activity, and so on) that make up Strava's data. I am struggling to come up with a way that separates the mechanics of making a request so it can be reused to fetch the various different objects in the API. What I have so far:
type Model interface {
Url() *url.URL
Data() interface{} // pointer to location of unmarshaled response
}
// an activity (run, bike ride, etc)
type Activity struct {
Id int64 `json:"id"`
Name string `json:"name"`
Distance float64 `json:"distance"`
// ...
}
func (a *Activity) Url() *url.URL {
return url.Parse(fmt.Sprintf("https://www.strava.com/api/v3/activities/%d", a.Id))
}
func (a *Activity) Data() interface{} {
return a
}
// gear (shoes, bike, etc)
type Gear struct {
Id string `json:"id"`
Name string `json:"name"`
}
func (g *Gear) Url() *url.URL {
return url.Parse(fmt.Sprintf("https://www.strava.com/api/v3/gear/%s", g.Id))
}
func (g *Gear) Data() interface{} {
return g
}
// a page of activities
type ActivityPage struct {
AthleteId int64
PageNum int
Activities []Activity
}
func (p *ActivityPage) Url() *url.URL {
return url.Parse(fmt.Sprintf("https://www.strava.com/api/v3/athletes/%d/activities?page=%d&per_page=%d", p.AthleteId, p.PageNum, perPage))
}
func (p *ActivityPage) Data() interface{} {
return &p.Activities
}
type Client struct {
hc *http.Client
}
// error handling omitted
func (c *Client) fetch(m Model) error {
data, _ := c.fetchUrl(m.Url())
json.Unmarshal(data, m.Data())
return nil
}
func (c *Client) fetchUrl(u *url.URL) ([]byte, error) {
req := &http.Request{
// omit access token
Method: "GET",
URL: u,
}
resp, _ := c.hc.Do(req)
defer resp.Body.Close()
return ioutil.ReadAll(resp.Body)
}
The Data() pointer is needed because the GET /athlete/activities endpoint returns a list of Activitys, rather than a specific model. ActivityPage is therefore a bit of a hack — an object that carries the data needed to build the URL along with a place to put the results. For cases where a GET returns a model the Data() pointer is just the object.
With this code, I can do:
client := Client{}
activity := Activity{Id: 1234}
client.fetch(activity)
fmt.Print(activity.Name)
page := ActivityPage{AthleteId: 1, PageNum: 1}
client.fetch(page)
fmt.Print(len(page.Activities))
But this feels.. icky. I don't like partially constructing the object and passing it to fetch() to be finished off, or that fetch doesn't actually return anything except an error on failure. The Data() pointer is a hack.
AIUI, interfaces are a way to write code that can work with objects of different types, but I feel like I want the inverse — to have some code (a Fetch() method or something) that is inherited by all objects with a certain trait.
How can I make this cleaner? I realise this is kind of open-ended so I'm more than happy to refine what the exact question is as appropriate. Are there canonical examples of building a REST client in Go? (I haven't found anything compelling so far)
this is a standard workaround typical for go, you pass a pointer to data you want to be modified by reflection, standard lib is built like that, you can at most make method to accept pointer to data and url directly to be more verbose and not interface. It will make it at least match cleaner what will get modified just from looking at api calls for user like:
func (c *Client) fetch(url string, responceBuffer interface{}) error {
data, err := c.fetchUrl(url)
if err != nil {
return err
}
return json.Unmarshal(data, responceBuffer)
}
Go here. Trying to get the chi renderer to return a list of Order struct instances and getting a compiler error that I don't understand:
package myapp
import (
"net/http"
"github.com/go-chi/render"
)
type Order struct {
OrderId string
Status string
}
func (*Order) Bind(r *http.Request) error {
return nil
}
func GetAllOrderByCustomerId(dbClient DbClient, customerId string) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
// fetch all customer orders from the DB
orders,err := dbClient.FetchAllOrdersByCustomerId(customerId)
if err != nil {
log.Error("unable to fetch orders for customer", err)
render.Render(w, r, NewInternalServerError(err))
return
}
render.Bind(r, &orders)
return
}
}
When I go to compile this code I get:
fizz/buzz/myapp/order_fetcher.go:136:20: cannot use &orders (type *[]Order) as type render.Binder in argument to render.Bind:
*[]Order does not implement render.Binder (missing Bind method)
So even though I've defined a Bind for Order, it doesn't seem to automatically apply that Bind to a collection/list of Orders.
Can anyone see what I'm missing? Some endpoints will only be returning a single Order, whereas others (like this one) need to be able to return a collection/list of Orders.
As in the example in chi repository, you have to create a helper method to render the list of something, in your case, list of orders.
First, you have to implement the render.Renderer method then create a helper method to build a list of render.Renderer.
I've adapted your code from the example here:
type Order struct {
OrderId string
Status string
}
// Render implement render.Renderer
func (*Order) Render(w http.ResponseWriter, r *http.Request) error {
// do something
return nil
}
// newOrderList is a helper method to make list of render.Renderer
func newOrderList(orders []*Order) []render.Renderer {
list := []render.Renderer{}
for _, order := range orders {
list = append(list, order)
}
return list
}
func GetAllOrderByCustomerId(dbClient DbClient, customerId string) http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
// fetch all customer orders from the DB
orders, err := dbClient.FetchAllOrdersByCustomerId(customerId)
if err != nil {
log.Error("unable to fetch orders for customer", err)
render.Render(w, r, NewInternalServerError(err))
return
}
// render list of orders
render.RenderList(w, r, newOrderList(orders))
return
}
}
I'm using GORM to retrieve data from a Postgresql database. Inside the postgresql database I'm storing times as the default UTC. When I load them through gorm/golang I would like to automatically convert them to 'Europe/London' location.
Currently, all times are returned as my local timezone (CEST). I'm struggling to find a way to manually override this?
Here is the relevant code:
type Booking struct {
gorm.Model
Service Service
ServiceID uint `json:serviceId`
Start time.Time
Finish time.Time
}
func getBookings() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
bookings := &[]Booking{}
GetDB().Find(bookings)
render.JSON(w, r, bookings)
}
}
I've been looking around and I can't seem to find any information from gorm or golang docs. The two closest things to mentions of this problem that I've found is:
https://github.com/jinzhu/gorm/wiki/How-To-Do-Time
Setting timezone globally in golang
I thought a work around could be to manually change the query results with a loop, but I'm not sure if this is the most efficient solution? - code below:
func getBookings() http.HandlerFunc {
return func(w http.ResponseWriter, r *http.Request) {
bookings := &[]Booking{}
timeZoneBookings := *&[]Booking{}
GetDB().Find(bookings)
for _, booking := range *bookings {
booking.Start = parseToUkTime(booking.Start)
booking.Finish = parseToUkTime(booking.Finish)
timeZoneBookings = append(timeZoneBookings, booking)
}
render.JSON(w, r, timeZoneBookings)
}
}
func parseToUkTime(timeToParse time.Time) time.Time {
loc, _ := time.LoadLocation("Europe/London")
t := timeToParse.In(loc)
return t
}
Here is an image of the DB entry:
I assumed it would be easy to either state inside the type I would like the location to be set to Europe/London so the struct would automatically be populated this way, however this doesn't seem to be the case? It's the first time I have worked with timezones, so everything is quite confusing.
Loop through the slice and update the time values in place. Lookup the location once outside of the handler.
func getBookings() http.HandlerFunc {
loc, _ := time.LoadLocation("Europe/London")
return func(w http.ResponseWriter, r *http.Request) {
var bookings []Booking
GetDB().Find(&bookings)
for i := range bookings {
booking[i].Start = bookings[i].Start.In(loc)
booking[i].Finish = bookings[i].Finish.In(loc)
}
render.JSON(w, r, bookings)
}
}
It would be awesome to have a straight forward mapping from the standard library URL.Query() to an struct.
Query() returns a map like:
map[a:[aaaa] b:[bbbb] c:[cccc]]
The struct looks like:
type Thing struct {
A string
B string
C string
}
I've no idea why URL.Query returns a map with array elements inside tough. (well.. I know why but a GET is not likely to have duplicated params)
Please find below the complete example of parsing get query params directly in a golang struct and then sending the struct back as response
package main
import (
"log"
"net/http"
"encoding/json"
"github.com/gorilla/schema"
)
var decoder = schema.NewDecoder()
type EmployeeStruct struct {
MemberId string `schema:"memberId"`
ActivityType string `schema:"activityType"`
BusinessUnitCode int `schema:"businessUnitCode"`
}
func GetEmployee(w http.ResponseWriter, r *http.Request) {
var employeeStruct EmployeeStruct
err := decoder.Decode(&employeeStruct, r.URL.Query())
if err != nil {
log.Println("Error in GET parameters : ", err)
} else {
log.Println("GET parameters : ", employeeStruct)
}
w.Header().Set("Content-Type", "application/json")
json.NewEncoder(w).Encode(employeeStruct)
}
func main() {
mux := http.NewServeMux()
mux.HandleFunc("/GetEmployee", GetEmployee)
log.Fatal(http.ListenAndServe(":8080", mux))
}
Steps to execute & Test (Assuming you are saving above code in employee.go) :
Step 1 : go run employee.go
Step 2 : Open in browser http://localhost:8080/GetEmployee?memberId=123&activityType=Call&businessUnitCode=56
Step 3 : You should get below response in browser window
{
"MemberId": "123",
"ActivityType": "Call",
"BusinessUnitCode": 56
}
Step 4 : On console you should see below
GET parameters : {123 Call 56}
example:
filters={"reference":["docker.io/library/alpine:latest"]}
need url encode to:
filters=%7B%22reference%22%3A%5B%22docker.io%2Flibrary%2Falpine%3Alatest%22%5D%7D
and could use "github.com/gorilla/schema"
query := struct {
All bool
Filters map[string][]string `schema:"filters"`
Digests bool
Filter string
}{}
decoder := schema.NewDecoder()
decoder.Decode(&query, r.URL.Query())
As pointed out by #mh-cbon gorilla schema is the ultimate solution here.
Instead for obtaining the queryParams from the URL attribute.
func handleRequest(w http.ResponseWriter, r *http.Request) {
queryString := r.URL.Query()
//...parsing the Values -> map[string][]string
}
The approach of gorilla schema is to ship r.PostForm to the decode function.
func handleRequest(w http.ResponseWriter, r *http.Request) {
err := decoder.Decode(person, r.PostForm)
//...using reflect each struct's property can be called using
// the PostForm(url string, data url.Values) signature
fmt.Print(person.GoodJobGorilla)
}
Just parse the string to URL and after you can use the lib github.com/gorilla/schema to parse it :)
// Example to parse querystring to struct
package main
import (
"log"
"net/url"
"github.com/gorilla/schema"
)
type URLParams struct {
Code string `schema:"code"`
State string `schema:"state"`
}
func main() {
var (
params URLParams
decoder = schema.NewDecoder()
)
p := "https://www.redirect-url.com?code=CODE&state=RANDOM_ID"
u, _ := url.Parse(p)
err := decoder.Decode(¶ms, u.Query())
if err != nil {
log.Println("Error in Decode parameters : ", err)
} else {
log.Printf("Decoded parameters : %#v\n", params)
}
}
https://go.dev/play/p/CmuPhdKh6Yg
Using ggicci/httpin
Disclaimer: I'm the creator and maintainer of this package.
httpin helps you easily decoding HTTP request data from
Query parameters, e.g. ?name=john&is_member=true
Headers, e.g. Authorization: xxx
Form data, e.g. username=john&password=******
JSON/XML Body, e.g. POST {"name":"john"}
Path variables, e.g. /users/{username}
File uploads
How to use?
type ListUsersInput struct {
Page int `in:"query=page"`
PerPage int `in:"query=per_page"`
IsMember bool `in:"query=is_member"`
}
func ListUsers(rw http.ResponseWriter, r *http.Request) {
input := r.Context().Value(httpin.Input).(*ListUsersInput)
if input.IsMember {
// Do sth.
}
// Do sth.
}
httpin is:
well documented: at https://ggicci.github.io/httpin/
well tested: coverage over 98%
open integrated: with net/http, go-chi/chi, gorilla/mux, gin-gonic/gin, etc.
extensible (advanced feature): by adding your custom directives. Read httpin - custom directives for more details.
awesome mentioned: https://github.com/avelino/awesome-go#forms
You can use the graceful package of Echo.
I write some codes as an example, with self-explanatory comments
package main
import (
"log"
"github.com/labstacks/echo"
)
// Declare your struct with form: "" tag
type Employee struct {
MemberId string `form:"memberId"`
ActivityType string `form:"activityType"`
BusinessUnitCode int `form:"businessUnitCode"`
}
// Your handlers should look like this method
// Which takes an echo.Context and returns an error
func GetEmployee(ctx echo.Context) error{
var employee Employee
// With Bind, you can get the Post Body or query params from http.Request
// that is wrapped by echo.Context here
if err := ctx.Bind(&employee);err != nil {
return err
}
// now you can use your struct , e.g
return ctx.json(200, employee.MemberId)
}
// now use the handler in your main function or anywhere you need
func main() {
e := echo.New()
e.Get("/employee", GetEmployee)
log.Fatal(e.Start(":8080"))
}
If anyone is using Echo, query struct tag will be useful for this case.
Example Struct
type struct Name {
FirstName string `query:"first_name"`
LastName string `query:"last_name"`
}
Example Query Param
?first_name="shahriar"&last_name="sazid"
Code
var name Name
err := c.Bind(&name); if err != nil {
return c.String(http.StatusBadRequest, "bad request")
}
I want to be able to unmarshal yaml files less rigidly. That is, my library has a predefined number of options the yaml file must have. Then, the user should be able to extend this to include any custom options.
Here is what I have
package main
import (
"net/http"
"yamlcms"
"github.com/julienschmidt/httprouter"
)
type Page struct {
*yamlcms.Page
Title string
Date string
}
func getBlogRoutes() {
pages := []*Page{}
yamlcms.ReadDir("html", pages)
}
// This section is a work in progress, I only include it for loose context
func main() {
router := httprouter.New()
//blogRoutes := getBlogRoutes()
//for _, blogRoute := range *blogRoutes {
// router.Handle(blogRoute.Method, blogRoute.Pattern,
// func(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {})
//}
http.ListenAndServe(":8080", router)
}
Here is the yamlcms package:
package yamlcms
import (
"io/ioutil"
"os"
"strings"
"gopkg.in/yaml.v2"
)
type Page struct {
Slug string `yaml:"slug"`
File string `yaml:"file"`
}
func (page *Page) ReadFile(file string) (err error) {
fileContents, err := ioutil.ReadFile(file)
if err != nil {
return
}
err = yaml.Unmarshal(fileContents, &page)
return
}
func isYamlFile(fileInfo os.FileInfo) bool {
return !fileInfo.IsDir() && strings.HasSuffix(fileInfo.Name(), ".yaml")
}
func ReadDir(dir string, pages []*Page) (err error) {
filesInfo, err := ioutil.ReadDir(dir)
if err != nil {
return
}
for i, fileInfo := range filesInfo {
if isYamlFile(fileInfo) {
pages[i].ReadFile(fileInfo.Name())
}
}
return
}
There is a compiler issue here:
src/main.go:19: cannot use pages (type []*Page) as type []*yamlcms.Page in argument to yamlcms.ReadDir
My main intent in this question is to learn the idiomatic way of doing this kind of thing in Go. Other 3rd-party solutions may exist but I am not immediately interested in them because I have problems like this frequently in Go having to do with inheritance, etc. So along the lines of what I've presented, how can I best (idiomatically) accomplish what I am going for?
EDIT:
So I've made some changes as suggested. Now I have this:
type FileReader interface {
ReadFile(file string) error
}
func ReadDir(dir string, pages []*FileReader) (err error) {
filesInfo, err := ioutil.ReadDir(dir)
if err != nil {
return
}
for i, fileInfo := range filesInfo {
if isYamlFile(fileInfo) {
(*pages[i]).ReadFile(fileInfo.Name())
}
}
return
}
However, I still get a similar compiler error:
src/main.go:19: cannot use pages (type []*Page) as type []*yamlcms.FileReader in argument to yamlcms.ReadDir
Even though main.Page should be a FileReader because it embeds yamlcms.Page.
EDIT: I forgot that slices of interfaces don't work like that. You'd need to allocate a new slice, convert all pages to FileReaders, call the function, and convert them back.
Another possible solution is refactoring yamlcms.ReadDir to return the contents of the files, so that they could be unmarshaled later:
// In yamlcms.
func ReadYAMLFilesInDir(dir string) ([][]byte, error) { ... }
// In client code.
files := yamlcms.ReadYAMLFilesInDir("dir")
for i := range pages {
if err := yaml.Unmarshal(files[i], &pages[i]); err != nil { return err }
}
The original answer:
There are no such things as inheritance or casting in Go. Prefer composition and interfaces in your designs. In your case, you can redefine your yamlcms.ReadDir to accept an interface, FileReader.
type FileReader interface {
ReadFile(file string) error
}
Both yamlcms.Page and main.Page will implement this, as the latter embeds the former.