How to speed up Google App Engine Go unit tests? - performance

I am currently writing a lot of unit tests for my package that runs on GAE Go. The package in question is focused on data saving and loading to and from appengine/datastore. As such, I have about 20 unit test files that look a bit like this:
package Data
import (
"appengine"
"appengine/aetest"
. "gopkg.in/check.v1"
"testing"
)
func TestUsers(t *testing.T) { TestingT(t) }
type UsersSuite struct{}
var _ = Suite(&UsersSuite{})
const UserID string = "UserID"
func (s *UsersSuite) TestSaveLoad(cc *C) {
c, err := aetest.NewContext(nil)
cc.Assert(err, IsNil)
defer c.Close()
...
As a result, each individual test file appears to be starting its own version of devappserver:
Repeat this 20 times and my unit tests run for over 10 minutes.
I am wondering, how can I speed up the execution of my testing suite? Should I have just one file that creates aetest.NewContext and passes that onwards, or is it due to me using separate Suites for each unit test? How can I speed this thing up?

You can use a custom TestMain function:
var ctx aetest.Context
var c aetest.Context
func TestMain(m *testing.M) {
var err error
ctx, err = aetest.NewContext(nil)
if err != nil {
panic(err)
}
code := m.Run() // this runs the tests
ctx.Close()
os.Exit(code)
}
func TestUsers(t *testing.T) {
// use ctx here
}
This way the dev server is started once for all the tests. More details on TestMain are available here: http://golang.org/pkg/testing/#hdr-Main.

Related

Automated integration testing in lambda support (golang, serverless)

So I have a web app running in serverless. It spins up a bunch of lambdas and then a 'test' lambda is invoked at a later stage of our pipeline to run some api tests against the other lambbas. It passes or fails code pipeline depending on the result of the tests.
My concern is the implementation of the tests portion itself.
We're using golang, and I wasn't able to successfully find a way to have a bunch of go test files and run them, record the results and determine pass/fail or not.. but I wanted to use the test suite and library in order to run my assertions. So I came up with this solution which was to break about the go test library and run it myself using MainStart(), ex:
// T is used to manage our test cases manually using MainStart
type T struct{}
func (*T) ImportPath() string { return "" }
func (*T) MatchString(pat, str string) (bool, error) { return true, nil }
func (*T) SetPanicOnExit0(bool) {}
func (*T) StartCPUProfile(io.Writer) error { return nil }
func (*T) StopCPUProfile() {}
func (*T) StartTestLog(io.Writer) {}
func (*T) StopTestLog() error { return nil }
func (*T) WriteHeapProfile(io.Writer) error { return nil }
func (*T) WriteProfileTo(string, io.Writer, int) error { return nil }
func (h *Handler) Handle(ctx context.Context, event events.CodePipelineEvent) (interface{}, error) {
job := event.CodePipelineJob
ok := runTest()
if ok {
logger.Info("Tests Passed!")
input := &codepipeline.PutJobSuccessResultInput{
JobId: &job.ID,
success, err := h.service.PutJobSuccessResult(input)
return success, err
} else {
logger.Info("Tests Failed :(")
input := &codepipeline.PutJobFailureResultInput{
JobId: &job.ID,
FailureDetails: &codepipeline.FailureDetails{
Message: aws.String("tests failed"),
Type: aws.String("JobFailed"),
},
failure, err := h.service.PutJobFailureResult(input)
return failure, err
}
}
}
// NewHandler returns a pointer to a Handler struct
func NewHandler(service *codepipeline.CodePipeline) *Handler {
return &Handler{
service,
}
}
// runTest returns the test results
func runTest() bool {
testSuite := []testing.InternalTest{
{Name: "Test Service", F: TestService},
}
logger.Info("Running Tests")
errors := testing.MainStart(&T{}, testSuite, nil, nil).Run()
if errors == 0 {
return true
} else {
return false
}
}
So the test looks like this:
func TestServices(t *testing.T) {
logger.Info("blah blah do tests")
}
And for reference, main.go contains:
// Initial code pipeline session
cpSession := session.Must(session.NewSession())
// Create the credentials from AssumeRoleProvider to assume the role
// referenced by the "putJobResultRoleArn" ARN.
creds := stscreds.NewCredentials(cpSession, "putJobResultRoleArn", func(p *stscreds.AssumeRoleProvider) {
p.RoleARN = roleArn
p.RoleSessionName = "put_job_result_session"
})
// Create a new instance of the CodePipeline client with a session
svc := codepipeline.New(cpSession, &aws.Config{Credentials: creds})
intTesthandler := inttest.NewHandler(
svc,
)
lambda.Start(func(ctx context.Context, event events.CodePipelineEvent) (interface{}, error) {
return intTesthandler.Handle(ctx, event)
})
This works quite well, but what I don't like is:
It's using an unsupported approach from golang that could break in the future
I have to handhold any failures, meaning if one fails, I need to write code to stop the rest of the suite (if there were more tests) from running.
Bit of a learning curve
To note the tests I have mainly do an http request to an api endpoint and validate the response + parse/store the data.
Anyone have any other creative solution? Is there a repo out there that makes it simple to run integration tests in golang?
Cheers

Google cloud function Golang unit test with init()

I am having problems creating a Cloud Function in Golang and creating unit tests for it. Specifically, the environment variables that I want to inject during the test phase are not recognized, since the init() function always runs before the test. This init() function uses the environment variables to create a client. Is there any way around this problem?
Google recommends putting global variables like clients in the init() function, to reuse those objects in future invocations. This reduces startup time (source).
Directory structure
/function
do_something.go
do_something_test.go
do_something.go
package main
import (
"context"
"log"
"os"
"cloud.google.com/go/firestore"
)
var (
projectID = os.Getenv("GCLOUD_PROJECT")
client *firestore.Client
)
func init() {
ctx := context.Background()
var err error
client, err = firestore.NewClient(ctx, projectID)
if err != nil {
log.Fatalf("Firestore: %v", err)
}
}
func Main(w http.ResponseWriter, r *http.Request) {
// Do something
}
do_something_test.go
package main
import (
"os"
"testing"
)
func init() {
os.Setenv("GCLOUD_PROJECT", "my-test-project") // This does not work because of lexigraphical order, init() in do_something.go before this init().
}
func TestMain(t *testing.T) {
os.Setenv("GCLOUD_PROJECT", "my-test-project") // This does not work because init() is run before TestMain.
}
Possible solutions
Do not use init() in do_something.go. However, this gives a penalty in cloud function performance, latency is increased due to extended startup time.
Inject variables through a configuration file. However, I think this introduces the same problem.
Set environment variables before running go test. This is feasible, however I find this a suboptimal solution, since I don't want to run extra arguments before doing the actual test.
Always default to the development environment unless an environment variable specifies the code runs in production:
projectID = GetEnv("GCLOUD_PROJECT", "my-test-project")
func GetEnv(key string, defaultVal string) string {
if value, exists := os.LookupEnv(key); exists {
return value
}
return defaultVal
}
You should be exposing an environment variable at project level stating if the code is being executed in test settings or prod settings. Based on this, your init function will set the values of the variables.

Mocking a function return value called from main() in unit test

In main.go, I have some code that makes a network call to AWS Secrets manager.
func main() {
secretName := os.Getenv("DYNAMO_SECRET")
credentials, err := getSecret(secretName)
if err != nil {
logger.Errorf("Failed to retrieve secret from AWS Secrets manager %+v\n", err)
panic(err)
}
router, err := setupRouter(credentials)
The getSecret(secretName) function makes a network call to AWS Secrets manager underneath the hood. In my unit test for main, I have the code below.
func TestMainProgram(t *testing.T) {
defer mockStartServer(nil)()
defer mockSetupRouter(mux.NewRouter(), nil)()
main()
t.Log("Everything is perfect.")
}
When running my unit test, I want to mock the network call to AWS Secrets Manager. Is it possible to mock the return value of getSecret(secretName)? In a Java context, I'm trying to do something similar using Mockito and the when(functionIsCalled).thenReturn(mockValue) syntax.
You can use a function variable to set it to some other value for testing:
func defaultGetSecret(secretName string) (Credentials, error) {...}
var getSecret=defaultGetSecret
func main() {
...
credentials, err:=getSecret(...)
}
In your tests, you can change getSecret to point to something else:
func TestMain(t *testing.T) {
getSecret=func(secretName string) (Credentials,error) {
return mockCredentials,nil
}
main()
}

Gikngo Tests hang during goapp tests

I'm trying to use Gikngo to write some tests for appengine.
My setup for the tests is as follows:
suite_test.go:
BeforeSuite() {
inst, err = aetest.NewInstance(options)
if err != nil {
Fail(fmt.Sprintf("%s", err))
}
}
var(
req *http.Request
ctx context.Context
)
BeforeEach() {
req = inst.NewRequest()
ctx = appengine.NewContext(req)
// Clean up local datastore using the context.
}
validation_test.go
Describe("Some Test", func() {
It("ValidateFoo", func() {
// Access ctx here
})
...
It("ValidateBar", func() {
// Access ctx here.
})
})
I see our tests consistently hanging with the error of the type:
Expected success, but got an error:
<*url.Error | 0xc8210570b0>: {
Op: "Post",
URL: "http://localhost:59072",
Err: {s: "EOF"},
}
Post http://localhost:59072: EOF
This seems to indicate that the API server has become inaccessible. However, the test output does not seem to indicate this.
What are the ways in which we can debug a goapp test?
It turns out that Ginkgo or Golang had nothing to do with this. There seems to be some limitation on the number of fields one can read per second from dev_appserver.py. (I suspect that it might be related to SQLite which is the DB that dev_appserver uses internally).
The following code points out the problem:
package preorder
import (
"fmt"
"testing"
"google.golang.org/appengine"
"google.golang.org/appengine/aetest"
"google.golang.org/appengine/datastore"
)
func TestLoad(t *testing.T) {
opts := aetest.Options{StronglyConsistentDatastore: true}
inst, _ := aetest.NewInstance(&opts)
defer inst.Close()
for i := 0; i < 10000; i++ {
req, _ := inst.NewRequest("GET", "/", nil)
ctx := appengine.NewContext(req)
k := datastore.NewKey(ctx, ENTITY_NAME, "", 12345, nil)
var entity Entity
datastore.Get(ctx, k, &entity)
fmt.Println("Iteration Count: ", i)
ctx.Done()
}
}
Any help on figuring out how to work around the limit of 240 operations would be appreciated. One technique I can think of is to artificially inject delays.

Cobra + Viper Golang How to test subcommands?

I am developing an web app with Go. So far so good, but now I am integrating Wercker as a CI tool and started caring about testing. But my app relies heavily on Cobra/Viper configuration/flags/environment_variables scheme, and I do not know how to properly init Viper values before running my test suite. Any help would be much appreciated.
When I use Cobra/Viper or any other combination of CLI helpers, my way of doing this is to have the CLI tool run a function whose sole purpose will be to get arguments and pass them to another method who will do the actual work.
Here is a short (and dumb) example using Cobra :
package main
import (
"fmt"
"os"
"github.com/spf13/cobra"
)
func main() {
var Cmd = &cobra.Command{
Use: "boom",
Short: "Explode all the things!",
Run: Boom,
}
if err := Cmd.Execute(); err != nil {
fmt.Println(err)
os.Exit(-1)
}
}
func Boom(cmd *cobra.Command, args []string) {
boom(args...)
}
func boom(args ...string) {
for _, arg := range args {
println("boom " + arg)
}
}
Here, the Boom function is hard to test, but the boom one is easy.
You can see another (non-dumb) example of this here (and the correspond test here).
i have found an easy way to test commands with multiple level sub commands, it is not professional but it worked well.
assume we have a command like this
RootCmd = &cobra.Command{
Use: "cliName",
Short: "Desc",
}
SubCmd = &cobra.Command{
Use: "subName",
Short: "Desc",
}
subOfSubCmd = &cobra.Command{
Use: "subOfSub",
Short: "Desc",
Run: Exec
}
//commands relationship
RootCmd.AddCommand(SubCmd)
SubCmd.AddCommand(subOfSubCmd)
When testing the subOfSubCmd we can do this way:
func TestCmd(t *testing.T) {
convey.Convey("test cmd", t, func() {
args := []string{"subName", "subOfSub"}
RootCmd.SetArgs(args)
RootCmd.Execute()
})
}

Resources