I have a GO project with this project structure (multiple couples of this kind of files in each package).
- api
- userHandler.go
- userHandler_test.go
- database
- user.go
- user_test.go
Inside user.go I have the User struct and the functions to Create/Get/Update a User (I'm using GORM but this is not the issue). In the user_test.go.
I'd like to have the DB cleaned (with all the data removed or in a certain state) for each different file, so I've tried to create 1 suite (using Testify) for each file, then use the SetupSuite function but the behaviour seems not deterministic, and probably I'm doing something wrong.
So my questions are:
Which is the best way to have a DB connection shared? Using a global variable is the best option?
Which is the best way to create the tables in the DB once and then init the DB with custom data before each file_test.go is run?
Right now I'm also having a strange bug: running
go test path/package1
go test path/package2
Everything works fine, but if I run (for testing all the packages)
cd path && go test ./...
I have errors that seems not to be deterministic, that's why I'm guessing that the DB connection is not handled properly
If your api package depends on your database package (which it appears to) then your api package should have a way to provide a database connection pool (e.g. a *sql.DB) to it.
In your tests for the api package, you should just pass in an initialised pool (perhaps with the test schema/fixtures pre-populated) that you can use. This can either be a global you initialise in init() for the api package or a setup() and defer teardown() pattern in each test function.
Here's the former (simplest) approach where you just create a shared database and schema for your tests to use.
package database
import testing
var testDB *sql.DB
// This gets run before your actual test functions do.
func init() {
var err error
db, err = sql.Open(...)
if err != nil {
log.Fatalf("test init failed: %s", err)
}
_, err := db.Exec(`CREATE TABLE ....`)
if err != nil {
log.Fatalf("test schema creation failed: %s", err)
}
}
Some tips:
You can also call a setup() function can create a table with a random suffix and insert your test data so that your tests don't use the same test table (and therefore risk conflicting or relying on each other). Capture that table name and dump it in your defer teardown() function.
https://medium.com/#benbjohnson/structuring-applications-in-go-3b04be4ff091 is worth reading for some additional perspective.
Related
I'm setting up some integration testing, which I'm doing in a separate test package to my src code. This is done to prevent circular dependencies. Unit tests are not stored here, they are stored alongside the files they are testing.
My golang project hierarchy looks like:
cmd
public
...
testing/
main_test.go
database_test.go
in main_test.go, I plan to initialise the connections to external dependencies, such as my test database.
package tests
type Database struct {
...
}
var DB Database
func TestMain(m *testing.M){
SetUpDatabase()
exitCode := m.Run()
os.Exit(exitCode)
}
database_integration_test.go
func Test(t *testing.T) {
tests := []struct {
title string
run func(t *testing.T)
}{
{"should make query", testQuery},
}
for _, test := range tests {
t.Run(test.title, func(t *testing.T) {
test.run(t)
})
}
}
func testQuery(t *testing.T) {
var r result.Result
err := DB.database.DoQuery("").Decode(&r)
if err != nil {
t.Errorf(err.Error())
t.Fatal()
}
}
This setup works when I run it, however, I would like to add build tags to these files, or the type: // +build integration
However, as soon as I use the build tag, the database_integration_test.go file cannot see the initalised Database type. How can I stop this? Also as a side note, should I change the name of main_test.go. I only called it that due to main being the standrd entry point.
Firstly, Regarding this:
Also as a side note, should I change the name of main_test.go. I only
called it that due to main being the standard entry point.
I think it is confusing to name it as main_test.go as it might indicate that you are testing the main function in this file (according to golang convention)
Secondly, Regarding this:
However, as soon as I use the build tag, the
database_integration_test.go file cannot see the initialised Database
type. How can I stop this?
A Build Constraint or also known as Build Tag is used to include or exclude files in a package during a build process. With this, we can build different types of builds from the same source code.
So if you are not seeing the Database Type initialized then most probably the definition of the Database Type and the integration test are tagged with different build tags. Make sure they are present in the same build tags. Also, I think you can use more than one build tag to label a file. So you can try that as well.
For more details on the Build Tags check out the following article by Dave Cheney here
You could simply add a flag to your TestMain:
var isIntegration bool
func init() {
flag.StringVar(&isIntegration, "mytest.integration", "Set flag to set up DB for integration tests")
}
func TestMain(t *testing.M) {
SetUpDatabase(isIntegration)
//etc...
}
Then just simply have the SetUpDatabase call different, unexported, functions based on whether or not the argument is true or false. That'd be a quick way to get the behaviour, without having to much about with custom build constraints. Especially considering you're running tests, not building the application as-such.
As far as renaming main_test.go is concerned: I don't see why you'd need to change it. It does what it says on the tin. When someone else wants to see how the tests are structured/run, or what possible flags have been added, it's a lot easier to just check the directory and look for a main_test.go file (along with init.go, that'd be first file I'd look for). Any other name like setup_integration_test.go, integration_setup_test.go, integration_start_test.go, ... is just going to muddy the waters.
I'm using https://github.com/kubernetes/client-go and all works well.
I have a manifest (YAML) for the official Kubernetes Dashboard: https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-beta4/aio/deploy/recommended.yaml
I want to mimic kubectl apply of this manifest in Go code, using client-go.
I understand that I need to do some (un)marshalling of the YAML bytes into the correct API types defined in package: https://github.com/kubernetes/api
I have successfully Createed single API types to my cluster, but how do I do this for a manifest that contains a list of types that are not the same? Is there a resource kind: List* that supports these different types?
My current workaround is to split the YAML file using csplit with --- as the delimiter
csplit /path/to/recommended.yaml /---/ '{*}' --prefix='dashboard.' --suffix-format='%03d.yaml'
Next, I loop over the new (14) parts that were created, read their bytes, switch on the type of the object returned by the UniversalDeserializer's decoder and call the correct API methods using my k8s clientset.
I would like to do this to programmatically to make updates to any new versions of the dashboard into my cluster. I will also need to do this for the Metrics Server and many other resources. The alternative (maybe simpler) method is to ship my code with kubectl installed to the container image and directly call kubectl apply -f -; but that means I also need to write the kube config to disk or maybe pass it inline so that kubectl can use it.
I found this issue to be helpful: https://github.com/kubernetes/client-go/issues/193
The decoder lives here: https://github.com/kubernetes/apimachinery/tree/master/pkg/runtime/serializer
It's exposed in client-go here: https://github.com/kubernetes/client-go/blob/master/kubernetes/scheme/register.go#L69
I've also taken a look at the RunConvert method that is used by kubectl: https://github.com/kubernetes/kubernetes/blob/master/pkg/kubectl/cmd/convert/convert.go#L139 and assume that I can provide my own genericclioptions.IOStreams to get the output?
It looks like RunConvert is on a deprecation path
I've also looked at other questions tagged [client-go] but most use old examples or use a YAML file with a single kind defined, and the API has changed since.
Edit: Because I need to do this for more than one cluster and am creating clusters programmatically (AWS EKS API + CloudFormation/eksctl), I would like to minimize the overhead of creating ServiceAccounts across many cluster contexts, across many AWS accounts. Ideally, the only authentication step involved in creating my clientset is using aws-iam-authenticator to get a token using cluster data (name, region, CA cert, etc). There hasn't been a release of aws-iam-authenticator for a while, but the contents of master allow for the use of a third-party role cross-account role and external ID to be passed. IMO, this is cleaner than using a ServiceAccount (and IRSA) because there are other AWS services the application (the backend API which creates and applies add-ons to these clusters) needs to interact with.
Edit: I have recently found https://github.com/ericchiang/k8s. It's definitely simpler to use than client-go, at a high-level, but doesn't support this behavior.
It sounds like you've figured out how to deserialize YAML files into Kubernetes runtime.Objects, but the problem is dynamically deploying a runtime.Object without writing special code for each Kind.
kubectl achieves this by interacting with the REST API directly. Specifically, via resource.Helper.
In my code, I have something like:
import (
meta "k8s.io/apimachinery/pkg/api/meta"
"k8s.io/cli-runtime/pkg/resource"
"k8s.io/client-go/kubernetes"
"k8s.io/client-go/rest"
"k8s.io/client-go/restmapper"
"k8s.io/apimachinery/pkg/runtime"
)
func createObject(kubeClientset kubernetes.Interface, restConfig rest.Config, obj runtime.Object) error {
// Create a REST mapper that tracks information about the available resources in the cluster.
groupResources, err := restmapper.GetAPIGroupResources(kubeClientset.Discovery())
if err != nil {
return err
}
rm := restmapper.NewDiscoveryRESTMapper(groupResources)
// Get some metadata needed to make the REST request.
gvk := obj.GetObjectKind().GroupVersionKind()
gk := schema.GroupKind{Group: gvk.Group, Kind: gvk.Kind}
mapping, err := rm.RESTMapping(gk, gvk.Version)
if err != nil {
return err
}
name, err := meta.NewAccessor().Name(obj)
if err != nil {
return err
}
// Create a client specifically for creating the object.
restClient, err := newRestClient(restConfig, mapping.GroupVersionKind.GroupVersion())
if err != nil {
return err
}
// Use the REST helper to create the object in the "default" namespace.
restHelper := resource.NewHelper(restClient, mapping)
return restHelper.Create("default", false, obj, &metav1.CreateOptions{})
}
func newRestClient(restConfig rest.Config, gv schema.GroupVersion) (rest.Interface, error) {
restConfig.ContentConfig = resource.UnstructuredPlusDefaultContentConfig()
restConfig.GroupVersion = &gv
if len(gv.Group) == 0 {
restConfig.APIPath = "/api"
} else {
restConfig.APIPath = "/apis"
}
return rest.RESTClientFor(&restConfig)
}
I was able to get this working in one of my projects. I had to use much of the source code from kubectl's apply command to get it working correctly.
https://github.com/billiford/go-clouddriver/blob/master/pkg/kubernetes/client.go#L63
I am doing some web develoment using the go programming language using the package html/template. At some point of the code, I need to call the function template.ParseFiles(...) so I can create a template from those files ad then execute it using temp.Execute(w,data). I would like to know if it is better to create the template on each request or to do it once in the main and declare a global variable.
Right now I do it on each request on my handle functions, like most tutorials do. However, I don't know If I'm wasting resources by doing it on each request instead of having them as global variables.
This is how it looks on each request
func ViewStats(w http.ResponseWriter, r *http.Request) {
//Get stuff from db and put them in data
//...
//return data to user
tmp, err := template.ParseFiles("views/guest-layout.html",
"views/stats.html")
if err != nil {
fmt.Println(err)
} else {
tmp.Execute(w,data)
}
}
I would like to know if this is better:
var temp1 template.Template
func main() {
temp1, err = template.ParseFiles("file1","file2")
//...
}
As usual: It depends.
But first some nuance:
You should never do template parsing (or anything else interesting) in your main() function. Instead, your main() function should call methods (or a single method) that kicks off the chain of interesting things in your program.
Go doesn't have globals, so it's not actually an option to store your parsed templates in a global variable in the first place. The closest Go has to global variables is package variables. You could store your parsed templates in a package variable in the main package, but this is also bad form, as your main package (except for tiny, trivial programs), should just be an entry point, and otherwise nearly empty.
But now, on to the core of your question:
Should you parse templates per request, or per run?
And here it depends.
If you have templates that change frequently (i.e. during development, when you're constantly editing your HTML files), once per request can be best.
But this is far less efficient than just parsing once, so in production, you may wish to parse the templates once on startup only. Then you can store the templates in a package variable, or better, in a struct that is initialized at runtime. But I leave that to you.
But what may be best is actually a bit of a compromise between the two approaches. It may be best to load your templates at start-up, and re-load them occasionally, either automatically (say, every 5 minutes), or watch your filesystem, and reload them whenever the on-disk representation of the templates changes.
How to do this is left as an exercise for the reader.
I am currently trying to work on a small Go project, and I have a problem I am trying to solve.
I'm currently using github.com/jinzhu/gorm to handle database operations for the backend of a GraphQL server, and I wanted to be able to store the DB connection in a global variable accessible throughout the entire project (including sub-packages).
My first attempt was at creating a variable named db by doing the following in my main.go file:
var db *gorm.DB
var err error
then inside the init() function:
func init() {
db, err = gorm.Open("postgres", "credential stuff here")
db.AutoMigrate(&modelStructHere)
defer db.Close()
}
There isn't any crashing, but I would assume due to scoping, the db variable is only usable inside main.go, but I also need to be able to use this inside gql/gql.go, where my GraphQL resolver is currently located.
Perhaps I should move this chunk of code (DB init) to the actual resolver file, since there's really no use for DB operations outside of such a thing anyway, so maybe that's the problem?
Thanks in advance!
Alex's comment is spot on. Create a folder named "database" and inside put a file called "database.go" containing this:
package database
// the imports go here
var DB *gorm.DB
Now you can use it wherever you like with database.DB. You should not make the error variable global, handle it in the same function that initializes the DB. The init function can be in any place, usually you want it at the beginning of your program in the main function.
I am essentially trying to walk through a folder of html files. I want to embed them into the binary file and be able to parse them upon request for template execution purposes. (Please excuse me if im not wording this properly).
Any ideas, tips, tricks or better way of accomplishing this is much appreciated.
// Template Files
type TempFiles struct {
Files map[string]string
}
// Loop through view files and load them
func LoadTempFiles() {
t := new(TempFiles)
// Load template files
filepath.Walk("application/views", func(path string, info os.FileInfo, err error) error {
if !info.IsDir() {
content, _ := ioutil.ReadFile(path)
t.Files[path] = string(content)
}
return nil
})
}
func ViewTemp(w http.ResponseWriter, path string) {
t := new(TempFiles)
temp, err := template.New().Parse(t.Files[path])
if err != nil {
http.Error(w, err.Error(), http.StatusInternalServerError)
} else {
temp.Execute(w, nil)
}
}
I do this with most of my Go web apps. I use go-bindata to auto-generate Go source code from all the files I want to embed and then compile them into the binary.
All this is done automatically during build.
One downside is that the current go build tools do not offer a way to hook into the build process, so I use a Makefile for this purpose. When the makefile is invoked, it runs go-bindata to generate the sources for all necessary files, then usually performs some additional code generation bits and bobs (notably, creating a Go source file which lists all the embedded files in a map.. A Table of Contents if you will). It then proceeds to compile the actual program.
This can become a little messy, but you only have to set it all up once.
Another downside, is that the use of a Makefile means the software is not compatible with the go get command. But since most of my web apps are not meant to be shared anyway, this has not been a problem so far.
When it comes to debugging/developing such an application, there is another issue that arises from embedding the static web content: I can't just edit an HTML or CSS file and refresh the browser to see its effects. I would have to stop the server, rebuild it and restart it with every edit. This is obviously not ideal, so I split the Makefile up into a debug and release mode. The release mode does what I described above. The debug mode, however, wil not actually embed the static files. It does generate source files for each of them, but instead of having them contain the actual file data, it contains a stub which simply loads the data from the filesystem.
As far as the server code is concerned, there is no difference in the generated code. All it does is call a function to fetch the contents of a given static file. It does not care whether that content is actually embedded in the binary, or if it's loaded from an external source. So the two build modes are freely interchangeable.
For example, the same generated function to fetch static file content in release and debug mode would look as follows:
Release mode:
func index_html() []byte {
return []byte {
....
}
}
Debug mode:
func index_html() []byte {
data, err := ioutil.ReadFile("index.html")
...
return data
}
The interface in both cases is identical. This allows for easy and care-free development and debugging.
Another tool to consider: Another recent good tool comes from esc: Embedding Static Assets in Go (GitHub repo)
a program that:
can take some directories and recursively embed all files in them in a way that was compatible with http.FileSystem
can optionally be disabled for use with the local file system for local development
will not change the output file on subsequent runs
has reasonable-sized diffs when files changed
is vendoring-friendly
Vendoring-friendly means that when I run godep or party, the static embed file will not change.
This means it must not have any third-party imports (since their import path will be rewritten during goimports, and thus different than what the tool itself produces), or a specifiable location for the needed third-party imports.
It generates nice, gzipped strings, one per file.
There is a simple flag to enable local development mode, which is smart enough to not strip directory prefixes off of filenames (an option in esc that is sometimes needed).
The output includes all needed code, and does not depend on any third-party libraries for compatibility with http.FileSystem.
I made a package that makes switching between debug and production easier. It also provides an http.FileSystem implementation, making it easy to server the files. And it has several ways of adding the files to the binary (generate go code, or append as zip).
https://github.com/GeertJohan/go.rice
Go now has builtin support for this:
package main
import (
"embed"
"os"
)
//go:embed *.html
var content embed.FS
func main() {
b, e := content.ReadFile("index.html")
if e != nil {
panic(e)
}
os.Stdout.Write(b)
}
https://golang.org/pkg/embed