I have two questions on this go swagger generated codes, firstly I made my first api with go swagger but my employer asked me to implement the unit (go test) but trying to perform the usual http test does not work,
here is my test code bellow
// handlers_test.go
package handlers
import (
"net/http"
"net/http/httptest"
"testing"
"StocksApp/restapi/operations"
)
func TestStockHandler(t *testing.T) {
// Create a request to pass to our handler. We don't have any query parameters for now, so we'll
// pass 'nil' as the third parameter.
req, err := http.NewRequest("GET", "/api/v1/stocks", nil)
if err != nil {
t.Fatal(err)
}
// We create a ResponseRecorder (which satisfies http.ResponseWriter) to record the response.
rr := httptest.NewRecorder()
handler := http.HandlerFunc(operations.StocksHandler)
// Our handlers satisfy http.Handler, so we can call their ServeHTTP method
// directly and pass in our Request and ResponseRecorder.
handler.ServeHTTP(rr, req)
// Check the status code is what we expect.
if status := rr.Code; status != http.StatusOK {
t.Errorf("handler returned wrong status code: got %v want %v",
status, http.StatusOK)
}
// Check the response body is what we expect.
expected := `{"alive": true}`
if rr.Body.String() != expected {
t.Errorf("handler returned unexpected body: got %v want %v",
rr.Body.String(), expected)
}
}
but I am getting this error and I don't know how to solve it
# StocksApp/handlers [StocksApp/handlers.test]
.\handlers_test.go:21:32: type operations.StocksHandler is not an expression
FAIL StocksApp/handlers [build failed]
secondly, when ever i run the go run main.go command, the server runs in a different port, i will like to know how I can hardcode a permanent port number that the server will always run on
You can use HandlerFor method to get valid http.Handler for your operation.
See an example on: https://github.com/go-swagger/go-swagger/blob/master/examples/generated/restapi/operations/petstore_api.go#L436
Related
Update 1: it seems that using a context tied to the HTTP request may lead to the 'context canceled' error. However, using the context.Background() as the parent seems to work fine.
// This works, no 'context canceled' errors
ctx, cancel := context.WithTimeout(context.Background(), 100*time.Second)
// However, this creates 'context canceled' errors under mild load
// ctx, cancel := context.WithTimeout(r.Context(), 100*time.Second)
defer cancel()
app.Insert(ctx, record)
(updated code sample below to produce a self-contained example for repro)
In go, I have an http handler like the following code. On the first HTTP request to this endpoint I get a context cancelled error. However, the data is actually inserted into the database. On subsequent requests to this endpoint, no such error is given and data is also successfully inserted into the database.
Question: Am I setting up and passing the context correctly between the http handler and pgx QueryRow method? (if not is there a better way?)
If you copy this code into main.go and run go run main.go, go to localhost:4444/create and hold ctrl-R to produce a mild load, you should see some context canceled errors produced.
package main
import (
"context"
"fmt"
"log"
"math/rand"
"net/http"
"time"
"github.com/jackc/pgx/v4/pgxpool"
)
type application struct {
DB *pgxpool.Pool
}
type Task struct {
ID string
Name string
Status string
}
//HTTP GET /create
func (app *application) create(w http.ResponseWriter, r *http.Request) {
fmt.Println(r.URL.Path, time.Now())
task := &Task{Name: fmt.Sprintf("Task #%d", rand.Int()%1000), Status: "pending"}
// -------- problem code here ----
// This line works and does not generate any 'context canceled' errors
//ctx, cancel := context.WithTimeout(context.Background(), 100*time.Second)
// However, this linegenerates 'context canceled' errors under mild load
ctx, cancel := context.WithTimeout(r.Context(), 100*time.Second)
// -------- end -------
defer cancel()
err := app.insertTask(ctx, task)
if err != nil {
fmt.Println("insert error:", err)
return
}
fmt.Fprintf(w, "%+v", task)
}
func (app *application) insertTask(ctx context.Context, t *Task) error {
stmt := `INSERT INTO task (name, status) VALUES ($1, $2) RETURNING ID`
row := app.DB.QueryRow(ctx, stmt, t.Name, t.Status)
err := row.Scan(&t.ID)
if err != nil {
return err
}
return nil
}
func main() {
rand.Seed(time.Now().UnixNano())
db, err := pgxpool.Connect(context.Background(), "postgres://test:test123#localhost:5432/test")
if err != nil {
log.Fatal(err)
}
log.Println("db conn pool created")
stmt := `CREATE TABLE IF NOT EXISTS public.task (
id uuid NOT NULL DEFAULT gen_random_uuid(),
name text NULL,
status text NULL,
PRIMARY KEY (id)
); `
_, err = db.Exec(context.Background(), stmt)
if err != nil {
log.Fatal(err)
}
log.Println("task table created")
defer db.Close()
app := &application{
DB: db,
}
mux := http.NewServeMux()
mux.HandleFunc("/create", app.create)
log.Println("http server up at localhost:4444")
err = http.ListenAndServe(":4444", mux)
if err != nil {
log.Fatal(err)
}
}
TLDR: Using r.Context() works fine in production, testing using Browser is a problem.
An HTTP request gets its own context that is cancelled when the request is finished. That is a feature, not a bug. Developers are expected to use it and gracefully shutdown execution when the request is interrupted by client or timeout. For example, a cancelled request can mean that client never see the response (transaction result) and developer can decide to roll back that transaction.
In production, request cancelation does not happen very often for normally design/build APIs. Typically, flow is controlled by the server and the server returns the result before the request is cancelled.
Multiple Client requests does not affect each other because they get independent go-routine and context. Again, we are talking about happy path for normally designed/build applications. Your sample app looks good and should work fine.
The problem is how we test the app. Instead of creating multiple independent requests, we use Browser and refresh a single browser session. I did not check what exactly is going on, but assume that the Browser terminates the existing request in order to run a new one when you click ctrl-R. The server sees that request termination and communicates it to your code as context cancelation.
Try to test your code using curl or some other script/utility that creates independent requests. I am sure you will not see cancelations in that case.
I am using go-kit to create an RPC endpoint. I am creating an endpoint like this
httptransport.NewServer(
endPoint.MakeGetBlogEndPoint(blogService),
transport.DecodeGetBlogRequest,
transport.EncodeGetBlogResponse
Below is my DecodeGetBlogRequest function
func DecodeGetBlogRequest(c context.Context, r *http.Request) (interface{}, error) {
vars := mux.Vars(r)
id, err := strconv.Atoi(vars["id"])
if err != nil {
return nil, err
}
req := endPoint.GetBlogRequest{
ID: id,
}
return req, nil
}
What I want to do is validate the HTTP request in this function and if found invalid, send a response with a valid error code from here only, without passing it to the service layer. i.e. If ID is not a valid number, return 400 Bad Request response from here.
But as I don't have a ResponseWriter reference in this function, I am not sure how to do it.
I am following this example from go-kit docs
https://gokit.io/examples/stringsvc.html
Is it a valid assumption that request/payload should be validated in the transport layer only and the service layer should only be called if the request/payload is valid? If yes, how to do so in this example?
You could use ServerErrorEncoder which returns Server options (can be found in github.com/go-kit/kit/transport/server.go).
Basically in your transport layer, apart from the Decode and Encode functions, you can define an YourErrorEncoderFunc() function which could look like the following. This will catch any error thrown in the transport layer.
YourErrorEncoderFunc(_ context.Context, err error, w http.ResponseWriter).
You will need to attach this function as an option in your endpoint registration like:
ABCOpts := []httptransport.ServerOption{
httptransport.ServerErrorEncoder(YourErrorEncoderFunc),
}
r.Methods("GET").Path("/api/v1/abc/def").Handler(httptransport.NewServer(
endpoints.GetDataEndpoint,
DecodeGetRequest,
EncodeGetResponse,
ABCOpts...,
))
This will stop at transport layer if your request validation is invalid and throw and error in the http response based of whatever format you've written in YourErrorEncoderFunc().
Not 100% sure if this applies to go-kit grpc as well:
You have an error return variable. Use that to indicate there was a problem. In the go grpc module there is a status package to return errors with status codes. If you return an error with a status code, the grpc layer will take the code from the error and send it back.
For example:
func DecodeGetBlogRequest(c context.Context, r *http.Request) (interface{}, error) {
vars := mux.Vars(r)
id, err := strconv.Atoi(vars["id"])
if err != nil {
return nil, status.Error(codes.InvalidArgument, err.Error())
}
req := endPoint.GetBlogRequest{
ID: id,
}
return req, nil
}
Note also that grpc uses different status codes. In Go they are located in the codes package.
I have a simple service created using infinite loop to call a certain HTTP API periodically, implemented in a package aservice. I created a Service struct there. Typically, to run that service, I expose a StartService method which is used to run that service synchronously. Users of the package can then run it using a goroutine. My question is, how do you write the tests for this kind of scenario?
Do you run the whole system and "mock" the API? I have heard that code that uses 3rd party services don't need to be tested, but the whole aservice package may only contain StartService and Shutdown methods. The rest of them are unexported functions/methods which are then cannot be tested individually. If that is the case then I can't write any tests at all?
With Go you will have awesome experience while mocking external http requests. Long story short just substitute base url with server url from net/http/httptest package.
You can mimic the way Google mocks their external requests for example exploring tests in google maps here.
server := mockServer(200, response)
defer server.Close()
c, _ := NewClient(WithAPIKey(apiKey), WithBaseURL(server.URL))
r := &DirectionsRequest{
Origin: "Google Sydney",
Destination: "Glebe Pt Rd, Glebe",
Mode: TravelModeTransit,
}
resp, _, err := c.Directions(context.Background(), r)
// your assertions goes here
// Create a mock HTTP Server that will return a response with HTTP code and body.
func mockServer(code int, body string) *httptest.Server {
server := mockServerForQuery("", code, body)
return server.s
}
func mockServerForQuery(query string, code int, body string) *countingServer {
server := &countingServer{}
server.s = httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if query != "" && r.URL.RawQuery != query {
dmp := diffmatchpatch.New()
diffs := dmp.DiffMain(query, r.URL.RawQuery, false)
log.Printf("Query != Expected Query: %s", dmp.DiffPrettyText(diffs))
server.failed = append(server.failed, r.URL.RawQuery)
http.Error(w, "fail", 999)
return
}
server.successful++
w.WriteHeader(code)
w.Header().Set("Content-Type", "application/json; charset=UTF-8")
fmt.Fprintln(w, body)
}))
return server
}
I want to mock memcache cache data in go lang to avoid authhorization
i tried with gomock but couldn't work out as i dont have any interface for it.
func getAccessTokenFromCache(accessToken string)
func TestSendData(t *testing.T) {
mockCtrl := gomock.NewController(t)
defer mockCtrl.Finish()
mockObj := mock_utils.NewMockCacheInterface(mockCtrl)
mockObj.EXPECT().GetAccessToken("abcd")
var jsonStr = []byte(`{
"devices": [
{"id": "avccc",
"data":"abcd/"
}
]
}`)
req, err := http.NewRequest("POST", "/send/v1/data",
bytes.NewBuffer(jsonStr))
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Authorization", "d958372f5039e28")
rr := httptest.NewRecorder()
handler := http.HandlerFunc(SendData)
handler.ServeHTTP(rr, req)
if status := rr.Code; status != 200 {
t.Errorf("handler returned wrong status code: got %v want %v",
status, http.StatusOK)
}
expected := `{"error":"Invalid access token"}`
body, _ := ioutil.ReadAll(rr.Body)
if string(body) != expected {
t.Errorf("handler returned unexpected body: got %v want %v",
string(body), expected)
}
func SendData(w http.ResponseWriter, r *http.Request) {
accessToken := r.Header.Get(constants.AUTHORIZATION_HEADER_KEY)
t := utils.CacheType{At1: accessToken}
a := utils.CacheInterface(t)
isAccessTokenValid := utils.CacheInterface.GetAccessToken(a, accessToken)
if !isAccessTokenValid {
RespondError(w, http.StatusUnauthorized, "Invalid access token")
return
}
response := make(map[string]string, 1)
response["message"] = "success"
RespondJSON(w, http.StatusOK, response)
}
tried to mock using gomock
package mock_utils
gen mock for utils for get access controler
(1) Define an interface that you wish to mock.
(2) Use mockgen to generate a mock from the interface.
(3) Use the mock in a test:
You need to architect your code such that every such access to a service happens via an interface implementation. In your case, you should ideally create an interface like
type CacheInterface interface {
Set(key string, val interface{}) error
Get(key string) (interface{},error)
}
Your MemcacheStruct should implement this interface and all your memcache related calls should happen from there. Like in your case GetAccessToken should call cacheInterface.get(key) wherein your cacheInterface should refer to memcache implementation of this interface. This is a way better way to design your go programs and this would not only help you in writing tests but would also help in case let's say you want to use a different in memory database to help with caching. Like for ex., let's say in future if you want to use redis as your cache storage, then all you need to change is create a new implementation of this interface.
I have some code (see below) written in Go which is supposed to "fan-out" HTTP requests, and collate/aggregate the details back.
I'm new to golang and so expect me to be a nOOb and my knowledge to be limited
The output of the program is currently something like:
{
"Status":"success",
"Components":[
{"Id":"foo","Status":200,"Body":"..."},
{"Id":"bar","Status":200,"Body":"..."},
{"Id":"baz","Status":404,"Body":"..."},
...
]
}
There is a local server running that is purposely slow (sleeps for 5 seconds and then returns a response). But I have other sites listed (see code below) that sometime trigger an error as well (if they error, then that's fine).
The problem I have at the moment is how best to handle these errors, and specifically the "timeout" related errors; in that I'm not sure how to recognise if a failure is a timeout or some other error?
At the moment I get a blanket error back all the time:
Get http://localhost:8080/pugs: read tcp 127.0.0.1:8080: use of closed network connection
Where http://localhost:8080/pugs will generally be the url that failed (hopefully by timeout!). But as you can see from the code (below), I'm not sure how to determine the error code is related to a timeout nor how to access the status code of the response (I'm currently just blanket setting it to 404 but obviously that's not right - if the server was to error I'd expect something like a 500 status code and obviously I'd like to reflect that in the aggregated response I send back).
The full code can be seen below. Any help appreciated.
package main
import (
"encoding/json"
"fmt"
"io/ioutil"
"net/http"
"sync"
"time"
)
type Component struct {
Id string `json:"id"`
Url string `json:"url"`
}
type ComponentsList struct {
Components []Component `json:"components"`
}
type ComponentResponse struct {
Id string
Status int
Body string
}
type Result struct {
Status string
Components []ComponentResponse
}
var overallStatus string = "success"
func main() {
var cr []ComponentResponse
var c ComponentsList
b := []byte(`{"components":[{"id":"local","url":"http://localhost:8080/pugs"},{"id":"google","url":"http://google.com/"},{"id":"integralist","url":"http://integralist.co.uk/"},{"id":"sloooow","url":"http://stevesouders.com/cuzillion/?c0=hj1hfff30_5_f&t=1439194716962"}]}`)
json.Unmarshal(b, &c)
var wg sync.WaitGroup
timeout := time.Duration(1 * time.Second)
client := http.Client{
Timeout: timeout,
}
for i, v := range c.Components {
wg.Add(1)
go func(i int, v Component) {
defer wg.Done()
resp, err := client.Get(v.Url)
if err != nil {
fmt.Printf("Problem getting the response: %s\n", err)
cr = append(cr, ComponentResponse{
v.Id,
404,
err.Error(),
})
} else {
defer resp.Body.Close()
contents, err := ioutil.ReadAll(resp.Body)
if err != nil {
fmt.Printf("Problem reading the body: %s\n", err)
}
cr = append(cr, ComponentResponse{
v.Id,
resp.StatusCode,
string(contents),
})
}
}(i, v)
}
wg.Wait()
j, err := json.Marshal(Result{overallStatus, cr})
if err != nil {
fmt.Printf("Problem converting to JSON: %s\n", err)
return
}
fmt.Println(string(j))
}
If you want to fan out then aggregate results and you want specific timeout behavior the net/http package isn't giving you, then you may want to use goroutines and channels.
I just watched this video today and it will walk you through exactly those scenarios using the concurrency features of Go. Plus, the speaker Rob Pike is quite the authority -- he explains it much better than I could.
https://www.youtube.com/watch?v=f6kdp27TYZs
I am adding this for completes, as the correct answer was provided by Dave C in the comments of the accepted answer.
We can try to cast the error to a net.Error and check if it is a timeout.
resp, err := client.Get(url)
if err != nil {
// if there is an error check if its a timeout error
if e, ok := err.(net.Error); ok && e.Timeout() {
// handle timeout
return
}
// otherwise handle other types of error
}
The Go 1.5 release solved this issue by being more specific about the type of error it has handled.
So if you see this example https://github.com/Integralist/Go-Requester/blob/master/requester.go#L38 you'll see that I'm able to apply a regex pattern to the error message to decipher if the error was indeed a timeout or not
status := checkError(err.Error())
func checkError(msg string) int {
timeout, _ := regexp.MatchString("Timeout", msg)
if timeout {
return 408
}
return 500
}