I wrote a functions that save data into redis database server. The challenge is that I want to test these functions and do not know how to test it.
I just start somehow with
Functions
package sessrage
/*
* Save data into redis database. In the common case,
* the data will be only valid during a request. Use
* hash datatype in redis.
*/
import (
"../context"
"github.com/garyburd/redigo/redis"
"net/http"
)
const (
protocol string = "tcp"
port string = ":6379"
)
func connectAndCloseRedis(connectCall func(con redis.Conn)) {
c, err := redis.Dial("tcp", ":6379")
defer c.Close()
if err != nil {
panic(err.Error())
}
connectCall(c)
}
func PostSessionData(r *http.Request, key, value string) {
go connectAndCloseRedis(func(con redis.Conn) {
sessionId := context.Get(r, context.JwtId).(string)
con.Do("HMSET", sessionId, key, value)
})
}
func GetSessionData(r *http.Request, key string) interface{} {
var result interface{}
sessionId := context.Get(r, context.JwtId).(string)
reply, _ := redis.Values(c.Do("HMGET", sessionId, key))
redis.Scan(reply, &result)
return result
}
and the test file
package sessrage
import (
//"fmt"
"../context"
. "github.com/smartystreets/goconvey/convey"
"github.com/stretchr/testify/assert"
"net/http"
"net/http/httptest"
"testing"
"time"
)
var server *httptest.Server
var glrw http.ResponseWriter
var glr *http.Request
func init() {
server = httptest.NewServer(http.HandlerFunc(func(rw http.ResponseWriter, r *http.Request) {
glrw = rw
glr = r
context.Set(glr, context.JwtId, "TestId")
}))
}
func TestPostAndGetSession(t *testing.T) {
Convey("POST and GET data on redis.", t, func() {
PostSessionData(glr, "key1", "value1")
time.Sleep(time.Second * 10)
v := GetSessionData(glr, "key1")
assert.Equal(t, "value1", v)
})
}
when I try to run the test I've got
an't load package: ......./sessrage.go:10:2: local import "../context" in non-local package
and the context package looks like
package context
import (
"github.com/gorilla/context"
"net/http"
)
type contextKey int
const (
LanguageId contextKey = iota
JwtId
)
func Get(r *http.Request, key interface{}) interface{} {
return context.Get(r, key)
}
func Set(r *http.Request, key, val interface{}) {
context.Set(r, key, val)
}
What do I wrong?
That is the first time, I am testing code in conjunction with http. It seems to be very hard to test.
There are a few issues:
Don't use relative import paths.
Use a pool instead of dialing redis on every action.
The call to sessionId := context.Get(r, context.JwtId).(string) in the PostSessionData anonymous function can fail if the mux or something higher in the call chain clears the Gorilla context before the goroutine runs. Do this instead:
func PostSessionData(r *http.Request, key, value string) {
c := pool.Get()
defer c.Close()
sessionId := context.Get(r, context.JwtId).(string)
if err := c.Do("HMSET", sessionId, key, value); err != nil {
// handle error
}
}
Related
So I currently have a function that will take in a string APIKey to check it against my Mongo collection. If nothing is found (not authenticated), it returns false - if a user is found, it returns true. My problem, however, is I'm unsure how to integrate this with a Martini POST route. Here is my code:
package main
import (
"context"
"fmt"
"log"
"os"
"github.com/go-martini/martini"
_ "github.com/joho/godotenv/autoload"
"go.mongodb.org/mongo-driver/bson"
"go.mongodb.org/mongo-driver/mongo"
"go.mongodb.org/mongo-driver/mongo/options"
)
type User struct {
Name string
APIKey string
}
func validateAPIKey(users *mongo.Collection, APIKey string) bool {
var user User
filter := bson.D{{"APIKey", APIKey}}
if err := users.FindOne(context.TODO(), filter).Decode(&user); err != nil {
fmt.Printf("Found 0 results for API Key: %s\n", APIKey)
return false
}
fmt.Printf("Found: %s\n", user.Name)
return true
}
func uploadHandler() {
}
func main() {
mongoURI := os.Getenv("MONGO_URI")
mongoOptions := options.Client().ApplyURI(mongoURI)
client, _ := mongo.Connect(context.TODO(), mongoOptions)
defer client.Disconnect(context.TODO())
if err := client.Ping(context.TODO(), nil); err != nil {
log.Fatal(err, "Unable to access MongoDB server, exiting...")
}
// users := client.Database("sharex_api").Collection("authorized_users") // commented out when testing to ignore unused warnings
m := martini.Classic()
m.Post("/api/v1/upload", uploadHandler)
m.RunOnAddr(":8085")
}
The validateAPIKey function works exactly as intended if tested alone, I am just unsure how I would run this function for a specific endpoint (in this case, /api/v1/upload).
I've some web-application server using go http and I want that each request will have context with uuid, for this I can use http request context https://golang.org/pkg/net/http/#Request.Context
we are using logrus and we initiate it in one file and use the logger instance in other files.
what I need is to print request ID in all the logs but not to add new paremeters to each log print, I want do to it once in each http request (pass the req-id) and all the logs print will have it without doing anything with it
e.g. if the id=123 so log.info("foo") will print
// id=123 foo
I've tried with the following but not sure it's the right way, please advice.
package main
import (
"context"
"errors"
log "github.com/sirupsen/logrus"
)
type someContextKey string
var (
keyA = someContextKey("a")
keyB = someContextKey("b")
)
func main() {
ctx := context.Background()
ctx = context.WithValue(ctx, keyA, "foo")
ctx = context.WithValue(ctx, keyB, "bar")
logger(ctx, nil).Info("did nothing")
err := errors.New("an error")
logger(ctx, err).Fatal("unrecoverable error")
}
func logger(ctx context.Context, err error) *log.Entry {
entry := log.WithField("component", "main")
entry = entry.WithField("ReqID", "myRequestID")
return entry
}
https://play.golang.org/p/oCW09UhTjZ5
Every time you call the logger function you are creating a new *log.Entry and writing the request ID to it again. From your question it sounded like you do not want that.
func main() {
ctx := context.Background()
ctx = context.WithValue(ctx, keyA, "foo")
ctx = context.WithValue(ctx, keyB, "bar")
lg := logger(ctx)
lg.Info("did nothing")
err := errors.New("an error")
lg.WithError(err).Fatal("unrecoverable error")
}
func logger(ctx context.Context) *log.Entry {
entry := log.WithField("component", "main")
entry = entry.WithField("ReqID", "myRequestID")
return entry
}
The downside of this is that you will have to pass the lg variable to every function this request calls and which should also log the request ID.
What we did at our company is create a thin layer around logrus that has an additional method WithRequestCtx so we could pass in the request context and it would extract the request ID itself (which we had written to the context in a middleware). If no request ID was present nothing was added to the log entry. This however did add the request ID to every log entry again as your sample code also did.
Note: our thin layer around logrus had a lot more functionality and default settings to justify the extra effort. In the long run this turned out very helpful to have one place to be able to adjust logging for all our services.
Note2: meanwhile we are in the process of replacing logrus with zerolog to be more lightweight.
Late answer but all you need to do is just call logrus.WithContext(/* your *http.Request.Context() goes here*/).... in your application and logrus will automatically add "id":"SOME-UUID" to each logs. Design is flexible for extracting more key-value from request context if you wanted to.
initialise logger
package main
import (
"path/to/logger"
"path/to/request"
)
func main() {
err := logger.Setup(logger.Config{
ContextFields: map[string]interface{}{
string(request.CtxIDKey): request.CtxIDKey,
}
})
if err != nil {
// ...
}
}
logger
package logger
import (
"github.com/sirupsen/logrus"
)
type Config struct {
Level string
ContextFields map[string]interface{}
}
func Setup(config Config) error {
lev, err := logrus.ParseLevel(config.Level)
if err != nil {
return err
}
logrus.SetLevel(lev)
return nil
}
func (c Config) Fire(e *logrus.Entry) error {
for k, v := range c.StaticFields {
e.Data[k] = v
}
if e.Context != nil {
for k, v := range c.ContextFields {
if e.Context.Value(v) != nil {
e.Data[k] = e.Context.Value(v).(string)
}
}
}
return nil
}
request
package request
import (
"context"
"net/http"
"github.com/google/uuid"
)
type ctxID string
const CtxIDKey = ctxID("id")
func ID(h http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
h.ServeHTTP(w, r.WithContext(context.WithValue(r.Context(), CtxIDKey, uuid.New().String())))
})
}
I'm trying to write a tiny application in Go that can send an HTTP request to all IP addresses in hopes to find a specific content. The issue is that the application seems to crash in a very peculiar way when the call is executed asynchronously.
ip/validator.go
package ip
import (
"io/ioutil"
"net/http"
"regexp"
"time"
)
type ipValidator struct {
httpClient http.Client
path string
exp *regexp.Regexp
confirmationChannel *chan string
}
func (this *ipValidator) validateUrl(url string) bool {
response, err := this.httpClient.Get(url)
if err != nil {
return false
}
defer response.Body.Close()
if response.StatusCode != http.StatusOK {
return false
}
bodyBytes, _ := ioutil.ReadAll(response.Body)
result := this.exp.Match(bodyBytes)
if result && this.confirmationChannel != nil {
*this.confirmationChannel <- url
}
return result
}
func (this *ipValidator) ValidateIp(addr ip) bool {
httpResult := this.validateUrl("http://" + addr.ToString() + this.path)
httpsResult := this.validateUrl("https://" + addr.ToString() + this.path)
return httpResult || httpsResult
}
func (this *ipValidator) GetSuccessChannel() *chan string {
return this.confirmationChannel
}
func NewIpValidadtor(path string, exp *regexp.Regexp) ipValidator {
return newValidator(path, exp, nil)
}
func NewAsyncIpValidator(path string, exp *regexp.Regexp) ipValidator {
c := make(chan string)
return newValidator(path, exp, &c)
}
func newValidator(path string, exp *regexp.Regexp, c *chan string) ipValidator {
httpClient := http.Client{
Timeout: time.Second * 2,
}
return ipValidator{httpClient, path, exp, c}
}
main.go
package main
import (
"./ip"
"fmt"
"os"
"regexp"
)
func processOutput(c *chan string) {
for true {
url := <- *c
fmt.Println(url)
}
}
func main() {
args := os.Args[1:]
fmt.Printf("path: %s regex: %s", args[0], args[1])
regexp, regexpError := regexp.Compile(args[1])
if regexpError != nil {
fmt.Println("The provided regexp is not valid")
return
}
currentIp, _ := ip.NewIp("172.217.22.174")
validator := ip.NewAsyncIpValidator(args[0], regexp)
successChannel := validator.GetSuccessChannel()
go processOutput(successChannel)
for currentIp.HasMore() {
go validator.ValidateIp(currentIp)
currentIp = currentIp.Increment()
}
}
Note the line that says go validator.ValidateIp(currentIp) in main.go. Should I remove the word "go" to execute everything within the main routine, the code works as expected -> it sends requests to IP addresses starting 172.217.22.174 and should one of them return a legitimate result that matches the regexp that the ipValidator was initialized with, the URL is passed to the channel and the value is printed out by processOutput function from main.go. The issue is that simply adding go in front of validator.ValidateIp(currentIp) breaks that functionality. In fact, according to the debugger, I never seem to go past the line that says response, err := this.httpClient.Get(url) in validator.go.
The struggle is real. Should I decide to scan the whole internet, there's 256^4 IP addresses to go through. It will take years, unless I find a way to split the process into multiple routines.
I need to get property value:
telegram_token: "telegramtoken"
other_token: "othertoken"
But if I do Init() of import api and initialize function in func main() I don't get property value.
Why?
Thanks!
This is works:
package main
import (
"fmt"
"github.com/go-yaml/yaml"
)
var (
cfg Config
configData = []byte(`api:
telegram_token: "telegramtoken"
other_token: "othertoken"`)
)
type Config struct {
API ConfigAPI `yaml:"api"`
}
type ConfigAPI struct {
TelegramToken string `yaml:"telegram_token"`
OtherToken string `yaml:"other_token"`
}
func (c *Config) parse() {
err := yaml.Unmarshal(configData, c)
if err != nil {
fmt.Println(err.Error())
}
}
func Init() {
cfg.parse()
fmt.Printf("%+v\n", cfg)
}
func main() {
Init()
}
Console:
{API:{TelegramToken:telegramtoken OtherToken:othertoken}}
Doesn't work:
package api
import (
"fmt"
"github.com/go-yaml/yaml"
)
var (
cfg Config
configData = []byte(`api:
telegram_token: "telegramtoken"
waves_token: "wavestoken"`)
)
type Config struct {
API ConfigAPI `yaml:"api"`
}
type ConfigAPI struct {
TelegramToken string `yaml:"telegram_token"`
WavesToken string `yaml:"waves_token"`
}
func (c *Config) parse() {
err := yaml.Unmarshal(configData, c)
if err != nil {
fmt.Println(err.Error())
}
}
func Init() {
cfg.parse()
fmt.Printf("%+v\n", cfg)
}
package main, see on api.Init()
// The daemon that starts the API in background process.
package main
import (
"fmt"
"log"
"net/http"
"time"
"github.com/julienschmidt/httprouter"
api "github.com/krypton-code/waves-bot/pkg/api"
)
var (
host = "https://api.telegram.org/bot"
method = "/getMe"
)
// Index - home page.
func Index(w http.ResponseWriter, r *http.Request, _ httprouter.Params) {
fmt.Fprint(w, "<h1>Server of TelegaBot for Waves Platform is running!</h1>\n")
}
func main() {
api.Init()
router := httprouter.New()
router.GET("/", Index)
s := &http.Server{
Addr: ":8089",
Handler: router,
ReadTimeout: 10 * time.Second,
WriteTimeout: 10 * time.Second,
MaxHeaderBytes: 1 << 20,
}
log.Printf("\nApp listening on port%s. Go to http://localhost:8089/", s.Addr)
log.Fatal(s.ListenAndServe())
}
Console:
{API:{TelegramToken: WavesToken:}}
Modify the YAML to match the expected structure by indenting the token fields:
configData = []byte(`api:
telegram_token: "telegramtoken"
waves_token: "wavestoken"`)
)
I wrote a small test function in go. I'm having hard time in making a request to actual endpoint and test it. I tried importing the file which has the handler function (I think I'm trying to import whole directory : import (".")). Both my project.go and handler_test.go are in the same directory (I don't think this matters). Could someone give me heads up so that I can write more tests.
Here is my project.go:
package main
import (
"encoding/json"
"net/http"
"github.com/gorilla/mux"
"github.com/rs/cors"
)
type Person struct {
ID string `json:"id,omitempty"`
Firstname string `json:"firstname,omitempty"`
Lastname string `json:"lastname,omitempty"`
Address *Address `json:"address,omitempty"`
}
type Address struct {
City string `json:"city,omitempty"`
State string `json:"state,omitempty"`
}
var people []Person;
func GetPersonEndpoint(w http.ResponseWriter, req *http.Request) {
params := mux.Vars(req)
for _, item := range people {
if item.ID == params["id"] {
json.NewEncoder(w).Encode(item)
return
}
}
json.NewEncoder(w).Encode(&Person{})
}
func GetPeopleEndpoint(w http.ResponseWriter, req *http.Request) {
json.NewEncoder(w).Encode(people)
}
func CreatePersonEndpoint(w http.ResponseWriter, req *http.Request) {
params := mux.Vars(req)
var person Person
_ = json.NewDecoder(req.Body).Decode(&person)
person.ID = params["id"]
people = append(people, person)
json.NewEncoder(w).Encode(people)
}
func DeletePersonEndpoint(w http.ResponseWriter, req *http.Request) {
params := mux.Vars(req)
for index, item := range people {
if item.ID == params["id"] {
people = append(people[:index], people[index+1:]...)
break
}
}
json.NewEncoder(w).Encode(people)
}
func main() {
Router := mux.NewRouter()
people = append(people, Person{ID: "1", Firstname: "sarath", Lastname: "v", Address: &Address{City: "sunnyvale", State: "CA"}})
people = append(people, Person{ID: "2", Firstname: "dead", Lastname: "pool"})
// router.PathPrefix("/tmpfiles/").Handler(http.StripPrefix("/tmpfiles/", http.FileServer(http.Dir("."))))
Router.HandleFunc("/people", GetPeopleEndpoint).Methods("GET")
Router.HandleFunc("/people/{id}", GetPersonEndpoint).Methods("GET")
Router.HandleFunc("/people/{id}", CreatePersonEndpoint).Methods("POST")
c := cors.New(cors.Options{
AllowedOrigins: []string{"http://localhost:3000"},
AllowCredentials: true,
})
// Insert the middleware
handler := c.Handler(Router)
http.ListenAndServe(":12345", handler)
}
Here is my handler_test.go. In this code I'm testing GetPersonEndPoint.
package main
import (
"."
"net/http"
"net/http/httptest"
"testing"
"encoding/json"
)
func checkResponseCode(t *testing.T, expected, actual int) {
if expected != actual {
t.Errorf("Expected response code %d. Got %d\n", expected, actual)
}
}
func executeRequest(req *http.Request) *httptest.ResponseRecorder {
rr := httptest.NewRecorder()
handler := http.HandlerFunc(GetPersonEndpoint)
handler.ServeHTTP(rr, req)
if status := rr.Code; status != http.StatusOK {
fmt.Printf("Handler returned wrong status code: got %v want %v" , status, http.statusOk);
}
return rr
}
func TestGetPersonEndPoint(t *testing.T){
req, _ := http.NewRequest("GET", "/people/5", nil)
response := executeRequest(req)
checkResponseCode(t, http.StatusNotFound, response.Code)
var m map[string]string
json.Unmarshal(response.Body.Bytes(), &m)
if m["error"] != "Product not found" {
t.Errorf("Expected the 'error' key of the response to be set to 'Product not found'. Got '%s'", m["error"])
}
}
And finally this is the error:
./new.go:14: main redeclared in this block
previous declaration at ./myproject.go:62
./new.go:20: not enough arguments in call to server.ListenAndServeTLS
have ()
want (string, string)
Have a look at some http tests I've written: https://github.com/eamonnmcevoy/go_web_server/blob/master/pkg/server/user_router_test.go
// Arrange
us := mock.UserService{}
testUserRouter := NewUserRouter(&us, mux.NewRouter())
...
w := httptest.NewRecorder()
r, _ := http.NewRequest("PUT", "/", payload)
r.Header.Set("Content-Type", "application/json")
testUserRouter.ServeHTTP(w, r)
Simply create an instance of your router and call the endpoints using go's httptest. This snippet will perform a PUT request at the default endpoint /