Set up database for integration tests with TestMain across multiple packages - go

I am trying to write database integration tests in my go application for my repositories files.
My idea was to leverage the TestMain function to do the database bootstrap before the tests are run.
Example:
test/integration/integration_test.go
// +build integrationdb
package main
func TestMain(m *testing.M) {
flag.Parse()
// setup database
setupDB()
// run tests
os.Exit(m.Run())
}
Because this is global to all my integration tests I have placed this code in the test/integration package.
Then in each module/package of my application, together with my repository.go code, I have a repository_test.go with my test code:
// +build integrationdb
package todo_test
import (
"github.com/brpaz/go-api-sample/internal/todo"
"github.com/brpaz/go-api-sample/test/testutil"
"github.com/stretchr/testify/assert"
"testing"
)
func TestPgRepository_CreateTodo(t *testing.T) {
db, err := testutil.GetTestDBConnection()
if err != nil {
t.Fatal(err)
}
repo := todo.NewPgRepository(db)
err = repo.CreateTodo(todo.CreateTodo{
Description: "some-todo",
})
assert.Nil(t, err)
}
The setup works fine, the issue is that when running the tests, it says "testing: warning: no tests to run" and the tests are still executed, without running the TestMain function.
It seems that the TestMain function is per package? Is that true? Any workarounds?
I could put all the test files into a unique and separate package (I do that for higher-level tests like acceptance and smoke tests), but since I am testing each repository individually, It doesn't differ that much from a unit test, so I think it makes sense to keep then together with the code.
I guess I could create a TestMain per package and have some global variable in my global testutils package that kept if the database was initialized and run the setup if not set. Not really a fan of globals.
The other alternative I see is to move the setup code outside of the test logic. I prefer it to keep tightly integrated with Go as it makes it easier to defer cleanup functions for example, after the tests run.
Maybe I can create my own "integration test runner" command, that would run the setup code and then call "go test" programmatically?
Any other ideas?

The tests of each package can be run independently as they should. The only missing link was the bootstrap and teardown of the test database.
I decided to create a command in my application that will bootstrap the tests database and then running the "go test".
I could have this bootstrap logic separated, let´s say in a bash script, but I feel this way makes it easier.
Here is the code I ended up with for reference:
test/integration/db/main.go
func main() {
flag.Parse()
log.Println("Setup DB")
// setupDB creates a new test database and runs the application migrations before the tests start.
setupDB()
log.Println("Running Tests")
// calls go test to execute the tests.
runTests()
// delete the test database.
teardownDB()
}
func runTests() {
// TODO this flags can be passed from the original command instead.
cmd := exec.Command("go", "test", "-v", "--tags", "integrationdb", "-p", "1", "./...")
cmd.Env = os.Environ()
cmd.Stdout = os.Stdout
if err := cmd.Start(); err != nil {
log.Fatal(err)
}
if err := cmd.Wait(); err != nil {
log.Fatal(err)
}
}

Related

TestMain for all tests?

I have a fairly large project with many integration tests sprinkled throughout different packages. I'm using build tags to separate unit, integration and e2e tests.
I need to do some setup before running my integration and e2e tests, so I put a TestMain function in a main_test.go file in the root directory. It's pretty simple:
//go:build integration || e2e
// +build integration e2e
package test
import (
...
)
func TestMain(m *testing.M) {
if err := setup(); err != nil {
os.Exit(1)
}
exitCode := m.Run()
if err := tearDown(); err != nil {
os.Exit(1)
}
os.Exit(exitCode)
}
func setup() error {
// setup stuff here...
return nil
}
func tearDown() error {
// tear down stuff here...
return nil
}
However, when I run test:
$ go test -v --tags=integration ./...
testing: warning: no tests to run
PASS
# all of my subdirectory tests now run and fail...
I really don't want to write a TestMain in each package that requires it and was hoping I could just get away with one in the root. Is there any solution that you could suggest? Thanks.
The only alternative I can think of is setting up everything outside of code and then running the integration tests. Maybe some shell script that does setup and then calls $ go test?
The go test ./... command compiles a test binary for each package in the background and runs them one by one. This is also the reason you get a cannot use -o flag with multiple packages error if you attempt to specify an output. This is the reason code in your main package doesn't effect your sub packages.
So the only way to get this to work is to put all your setup logic in sort of "setup" package and call the shared code from all of your sub-packages(still a lot of work, I know).
Trying to avoid code repetition, I used a function that makes the setup/teardown and evaluates a function as a test.
The function should look like this
func WithTestSetup(t *testing.T, testFunction func()) {
// setup code
testFunction()
// teardown code
}
I use the t *testing.T argument to report errors in setup or teardown, but it can be omitted.
Then in your tests you can make:
func TestFoo(t *testing.T) {
WithTestSetup(
t, func() {
if err := Foo(); err != nil {
t.Fatal(err)
}
},
)
}
Just call WithTestSetup if needed, looks easier for me than add a bunch of TestMains on the project.

how to create a context object in google cloud run for firebase

I'm trying to create a context object, so that I can connect to firestore from cloud run, however, all the examples I find on the net basically say I need a context objects, examples I find online usually look like this:
ctx := context.Background()
client, err := firestore.NewClient(ctx, "projectID")
if err != nil {
fail(w, err.Error())
// TODO: Handle error.
}
You can find these examples in places like this one:
https://godoc.org/cloud.google.com/go/firestore#example-NewClient
Nowhere in this example is there an explanation of where to find the context object.
so I just get this error:
undefined: context
I think the documentation is just too confusing.
You should use r.Context() of http.Request object in the request handler instead of initializing a new detached context like context.Background().
The main purpose of contexts is to propagate them, and in Cloud Run you always process requests, so if you pass the request’s context, it’s the right thing to do.
I think in your case, “context” package is not imported. Make sure to use go 1.11+ in your Dockerfile and say:
import “context”
In Go, you need to import packages. For this statement ctx := context.Background() add to the top of your source file import "context" or merge with your existing import set.
Like most languages, the more experience you have the more the language makes sense and you just know what to do. Most languages are the same. In C/C++ you have the include statement, C# the using statement, in Python the import statement, etc.
Google has a large package of examples for using Go and Google Cloud Platform:
Google Cloud Examples in Go
I wrote an article that documents my 30-day journey to learn Go and Google Cloud Platform.
Google Cloud and Go – My Journey to Learn a new Language in 30 days
Given the plethora of 3 lines of code examples that are hard for beginners, and the lack of complete working examples online, like myself, here is a full working example, which is kinda what I needed when I started this task, I hope this helps anybody in the future.
package main
import (
"cloud.google.com/go/firestore" // https://godoc.org/cloud.google.com/go/firestore"
"context" // https://blog.golang.org/context
firebase "firebase.google.com/go"
"fmt"
"log"
"net/http"
"os"
)
func fail(w http.ResponseWriter, msg string) {
fmt.Fprintln(w, "fail:"+msg)
log.Println("fail:" + msg)
}
// State example code
type State struct {
Capital string `firestore:"capital"`
Population float64 `firestore:"pop"` // in millions
}
func main() {
http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
override := make(map[string]interface{})
ctx := context.Background()
client, err := firestore.NewClient(ctx, "YOURPID")// set GOOGLE_APPLICATION_CREDENTIALS env var
if err != nil {
fail(w, err.Error())
return
}
states := client.Collection("States")
ny := states.Doc("NewYork")
wr, err := ny.Create(ctx, State{
Capital: "Albany",
Population: 19.8,
})
fmt.Println(wr)
})
log.Fatal(http.ListenAndServe("0.0.0.0:8082", nil))
}

Using testing.T in TestMain

There are a few test cases I want to run for which there is a need to start a GRPC mock server. I am using gomock library for this. To start the server, I have to pass a variable of type testing.T to this function - gomock.NewController(). Since this is a kind of initialization for all the test cases, I want to do this in TestMain. But TestMain has access to only testing.M So how do I handle this case? Create a new testing.T structure in TestMain? Will it work?
It sounds like you are looking for a BeforeEach pattern. You don't have access to a testing.T object in TestMain because this is something more of a place to do initialization before and after the test suite runs.
There are a few frameworks out there that can give you a BeforeEach cheap:
Ginkgo
Onpar
to name a few.
You could also hand roll your own:
type test struct{
ctrl *gomock.Controller
mockFoo *MockFoo
// ...
}
func beforeEach(t *testing.T) test {
ctrl := gomock.NewController(t)
return test {
ctrl:ctrl,
mockFoo: NewMockFoo(ctrl),
}
}
func TestBar(t *testing.T) {
test := beforeEach(t)
// ...
}
func TestBaz(t *testing.T) {
test := beforeEach(t)
// ...
}

How do I change the import file?

I have the following function in golang:
import (
"github.com/aws/aws-sdk-go/service/iam"
"github.com/aws/aws-sdk-go/aws/session"
"fmt"
)
func NewIAM() *SphinxIAM {
// awsConfig := aws.NewConfig()
sess, err := session.NewSession()
if err != nil {
fmt.Println("Failed to create session,", err)
return nil
}
session := &SphinxIAM{iam: iam.New(sess)}
return session
}
Now, I am getting the following error when I run this:
cannot use sess (type *session.Session) as type "github.com/aws/aws-sdk-go/aws/client".ConfigProvider in argument to iam.New:
*session.Session does not implement "github.com/aws/aws-sdk-go/aws/client".ConfigProvider (wrong type for ClientConfig method)
have ClientConfig(string, ...*"stash/cloud/sphinx/vendor/github.com/aws/aws-sdk-go/aws".Config) "stash/cloud/sphinx/vendor/github.com/aws/aws-sdk-go/aws/client".Config
want ClientConfig(string, ...*"github.com/aws/aws-sdk-go/aws".Config) "github.com/aws/aws-sdk-go/aws/client".Config
I have to change the method getting imported but how do I exactly do that?
Thanks!
The problem here is that your github.com/aws/aws-sdk-go/aws/session package is vendored, it is loaded from the folder stash/cloud/sphinx/vendor/github.com/aws/aws-sdk-go/aws.
But the function you want to pass it: iam.New() is not vendored, it does not come from the same vendor folder (stash/cloud/sphinx/vendor/xxx) but it comes directly from github.com/aws/aws-sdk-go/service/iam.
Either put both packages under the same vendor folder, or none. It is possible that one of your dependency tool does this, (e.g. glide), in which case you should instruct your tool to handle both as vendored.

How to make templates work with gin framework?

I am newbie to golang.
To learn it I have started with a simple web app using gin framework.
I have followed the gin doc & configured template file but not able to make it work. I am getting an error -
panic: html/template: pattern matches no files: `templates/*`
goroutine 1 [running]:
html/template.Must
/usr/local/Cellar/go/1.5.2/libexec/src/html/template/template.go:330
github.com/gin-gonic/gin.(*Engine).LoadHTMLGlob
/Users/ameypatil/deployment/go/src/github.com/gin-gonic/gin/gin.go:126
main.main()
/Users/ameypatil/deployment/go/src/github.com/ameykpatil/gospike/main.go:17
Below is my code -
package main
import (
"net/http"
"github.com/gin-gonic/gin"
)
func main() {
//os.Setenv("GIN_MODE", "release")
//gin.SetMode(gin.ReleaseMode)
// Creates a gin router with default middleware:
// logger and recovery (crash-free) middleware
router := gin.Default()
router.LoadHTMLGlob("templates/*")
//router.LoadHTMLFiles("templates/index.tmpl")
router.GET("/", func(c *gin.Context) {
c.HTML(http.StatusOK, "index.tmpl", gin.H{
"title": "GoSpike",
})
})
// By default it serves on :8080 unless a
// PORT environment variable was defined.
router.Run(":4848")
}
My directory structure is
- gospike
--- templates
------index.tmpl
--- main.go
go install command does not give any error
but on actually running, it gives the above error. I searched & there were similar issues logged on gin's github repo but they are closed now.
I have tried various things but I guess I am missing something obvious. What am I missing?
I'm guessing the issue is that you're using a relative filepath to access your templates.
If I compile and run your code from the gospike directory, it works fine. But if I run gospike from any other directory, I get the same error you were seeing.
So either you need to always run gospike in the parent directory of templates, or you need to use the absolute path. You could either hard code it:
router.LoadHTMLGlob("/go/src/github.com/ameykpatil/gospike/templates/*")
or you could do something like
router.LoadHTMLGlob(filepath.Join(os.Getenv("GOPATH"),
"src/github.com/ameykpatil/gospike/templates/*"))
but that will fail if you have multiple paths set in your GOPATH. A better long-term solution might be setting a special environment variable like TMPL_DIR, and then just using that:
router.LoadHTMLGlob(filepath.Join(os.Getenv("TMPL_DIR"), "*"))
use relative path glob will working, you can try to code
router.LoadHTMLGlob("./templates/*")
notice the . dot sign which meaning current directory, gin.Engine will load template
base on templates subdirectory of current directory .
Here is how I do it. This walks through the directory and collects the files marked with my template suffix which is .html & then I just include all of those. I haven't seen this answer anywhere so I thought Id post it.
// START UP THE ROUTER
router := gin.Default()
var files []string
filepath.Walk("./views", func(path string, info os.FileInfo, err error) error {
if strings.HasSuffix(path, ".html") {
files = append(files, path)
}
return nil
})
router.LoadHTMLFiles(files...)
// SERVE STATICS
router.Use(static.Serve("/css", static.LocalFile("./css", true)))
router.Use(static.Serve("/js", static.LocalFile("./js", true)))
router.Use(static.Serve("/images", static.LocalFile("./images", true)))
routers.LoadBaseRoutes(router)
routers.LoadBlog(router)
router.Run(":8080")
There is a multitemplate HTML render to support multi tempaltes.
You can use AddFromFiles to add multi files:
r.AddFromFiles("index", "templates/base.html", "templates/index.html")
r.AddFromFiles("article", "templates/base.html", "templates/index.html", "templates/article.html")
Or load multi files under a directory:
package main
import (
"path/filepath"
"github.com/gin-contrib/multitemplate"
"github.com/gin-gonic/gin"
)
func main() {
r := gin.New()
r.HTMLRender = loadTemplates()
// ...
}
func loadTemplates() multitemplate.Render {
files, err := filepath.Glob("template/*.tmpl")
if err != nil {
panic("failed to load html templates: " + err.Error())
}
r := multitemplate.New()
// Generate our templates map from our template/ directories
for _, file := range files {
r.AddFromFiles(filepath.Base(file), file)
}
// add other html templates directly
r.Add("test.tmpl", someTemplate)
return r
}
You could see more APIs in this repo you want, hope this post help :)

Resources