I am having problems creating a Cloud Function in Golang and creating unit tests for it. Specifically, the environment variables that I want to inject during the test phase are not recognized, since the init() function always runs before the test. This init() function uses the environment variables to create a client. Is there any way around this problem?
Google recommends putting global variables like clients in the init() function, to reuse those objects in future invocations. This reduces startup time (source).
Directory structure
/function
do_something.go
do_something_test.go
do_something.go
package main
import (
"context"
"log"
"os"
"cloud.google.com/go/firestore"
)
var (
projectID = os.Getenv("GCLOUD_PROJECT")
client *firestore.Client
)
func init() {
ctx := context.Background()
var err error
client, err = firestore.NewClient(ctx, projectID)
if err != nil {
log.Fatalf("Firestore: %v", err)
}
}
func Main(w http.ResponseWriter, r *http.Request) {
// Do something
}
do_something_test.go
package main
import (
"os"
"testing"
)
func init() {
os.Setenv("GCLOUD_PROJECT", "my-test-project") // This does not work because of lexigraphical order, init() in do_something.go before this init().
}
func TestMain(t *testing.T) {
os.Setenv("GCLOUD_PROJECT", "my-test-project") // This does not work because init() is run before TestMain.
}
Possible solutions
Do not use init() in do_something.go. However, this gives a penalty in cloud function performance, latency is increased due to extended startup time.
Inject variables through a configuration file. However, I think this introduces the same problem.
Set environment variables before running go test. This is feasible, however I find this a suboptimal solution, since I don't want to run extra arguments before doing the actual test.
Always default to the development environment unless an environment variable specifies the code runs in production:
projectID = GetEnv("GCLOUD_PROJECT", "my-test-project")
func GetEnv(key string, defaultVal string) string {
if value, exists := os.LookupEnv(key); exists {
return value
}
return defaultVal
}
You should be exposing an environment variable at project level stating if the code is being executed in test settings or prod settings. Based on this, your init function will set the values of the variables.
Related
I am currently passing application/main context from main.go to repository.go so that I can use it as "parent" context to go with query context. Is this valid/idiomatic usage or not? Also, shall I ditch the idea and just use context.Background() as "parent" context to go with query context instead?
main.go
package main
import (
"context"
"internal/user"
)
func main() {
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
repo := user.NewRepository(ctx, db)
// HTTP server is running.
}
internal/user/repository.go
package user
import (
"context"
"database/sql"
"time"
)
type Repository struct {
*sql.DB
}
var appCTX context.Context
func NewRepository(ctx context.Context, db *sql.DB) Repository {
appCTX = ctx
return Repository{db}
}
func (r Repository) Insert(args ...interface{}) error {
ctx, cancel := context.WithTimeout(appCTX, 5 * time.Millisecond)
defer cancel()
// Run query etc.
res, err := r.ExecContext(ctx, `INSERT INTO .....`, args...)
}
The idiomatic use of context is to pass it as the first function argument, and not store in structs. This is from the context doc:
Do not store Contexts inside a struct type; instead, pass a Context explicitly to each function that needs it. The Context should be the first parameter, typically named ctx
So, even if you pass the main context down to your implementation, you should do that by passing the context to each operation.
Each self-contained operation (such as an HTTP request) should create a new context. If your main performs one such self-contained operation, you can pass the context down like this. However, if this is a server application, you should create a separate context for each request handler.
Today I try to program with context,code as follow:
package main
func main(){
ctx := context.Background()
ctx = context.WithValue(ctx,"appid","test111")
b.dosomething()
}
package b
func dosomething(ctx context.Context){
fmt.Println(ctx.Value("appid").(string))
}
Then my program has crashed.I think it's due to that these ctx is in different package
I suggest you to use context only in a lifetime of a single task and pass the same context through functions.
Also you should understand where to use context and where just to pass arguments to functions.
Another suggestion is to use custom types for setting and getting values from context.
According to all above, you program should look like this:
package main
import (
"context"
"fmt"
)
type KeyMsg string
func main() {
ctx := context.WithValue(context.Background(), KeyMsg("msg"), "hello")
DoSomething(ctx)
}
// DoSomething accepts context value, retrieves message by KeyMsg and prints it.
func DoSomething(ctx context.Context) {
msg, ok := ctx.Value(KeyMsg("msg")).(string)
if !ok {
return
}
fmt.Println("got msg:", msg)
}
You can move function DoSomething into another package and just call it as packagename.DoSomething it will change nothing.
I've the following project structure:
myGithubProject/
|---- cmd
|----- command
|----- hello.go
|---- internal
|----- template
|----- template.go
|----- log
|----- logger.go
main.go
The log and the template are on the same level (under internal package)
In logger.go Im using logrus as logger with some config sand I want to use the logger.go object inside the template package.
How should I do that in a clean way ?
Currently I use it with import logger inside my template.go file,
And under internal package I’ve 6 more packages which needs this logger. and each of them is depend on it.
(On the log package), is there a nice in go to handle it ?
update:
in case I've more things that I need to pass (like logger) what will be the approach/ pattern here ? maybe using dependency injection ? interface ? other clean approach...
I need some best practice full example how can I use logger inside the hello.go file and also in template.go.
this is my project
cliProject/main.go
package main
import (
"cliProject/cmd"
"cliProject/internal/logs"
)
func main() {
cmd.Execute()
logs.Logger.Error("starting")
}
**cliProject/cmd/root.go**
package cmd
import (
"fmt"
"github.com/spf13/cobra"
)
var rootCmd = &cobra.Command{
Use: "cliProject",
Short: "A brief description of your application",
}
func Execute() {
if err := rootCmd.Execute(); err != nil {
fmt.Println(err)
}
}
**cliProject/cmd/serve.go**
package cmd
import (
"cliProject/internal/logs"
"fmt"
"github.com/spf13/cobra"
)
// serveCmd represents the serve command
var serveCmd = &cobra.Command{
Use: "serve",
Short: "A brief description of your command",
Run: func(cmd *cobra.Command, args []string) {
fmt.Println("serve called")
startServe()
stoppingServe()
},
}
func init() {
rootCmd.AddCommand(serveCmd)
serveCmd.Flags().BoolP("toggle", "t", false, "Help message for toggle")
}
func startServe() {
logs.Logger.Error("starting from function serve")
}
func stoppingServe() {
logs.Logger.Error("stoping from function serve")
}
**cliProject/cmd/start.go**
package cmd
import (
"cliProject/internal/logs"
"github.com/spf13/cobra"
)
// startCmd represents the start command
var startCmd = &cobra.Command{
Use: "start",
Short: "Start command",
Run: func(cmd *cobra.Command, args []string) {
// Print the logs via logger from internal
logs.Logger.Println("start called inline")
// example of many functions which should use the logs...
start()
stopping()
},
}
func init() {
logs.NewLogger()
rootCmd.AddCommand(startCmd)
startCmd.Flags().BoolP("toggle", "t", false, "Help message for toggle")
}
func start() {
logs.Logger.Error("starting from function start")
}
func stopping() {
logs.Logger.Error("stoping from function start")
}
**cliProject/internal/logs/logger.go**
package logs
import (
"github.com/sirupsen/logrus"
"github.com/x-cray/logrus-prefixed-formatter"
"os"
)
var Logger *logrus.Logger
func NewLogger() *logrus.Logger {
var level logrus.Level
level = LogLevel("info")
logger := &logrus.Logger{
Out: os.Stdout,
Level: level,
Formatter: &prefixed.TextFormatter{
DisableColors: true,
TimestampFormat: "2009-06-03 11:04:075",
},
}
Logger = logger
return Logger
}
func LogLevel(lvl string) logrus.Level {
switch lvl {
case "info":
return logrus.InfoLevel
case "error":
return logrus.ErrorLevel
default:
panic("Not supported")
}
}
this is how it looks
And under internal package I’ve 6 more packages which needs this logger. and each of them is depend on it. (On the log package), is there a nice in go to handle it ?
A good general principle would be to respect the application's choice (whether to log or not) instead of setting policy.
Let Go pkgs in your internal directory be support packages that
will only return error in case of problems
won't log (console or otherwise)
won't panic
Let your application (packages in your cmd directory) decide what the appropriate behavior in case of an error (log / graceful shutdown / recover to 100% integrity)
This would streamline development by having logging at a particular layer only. Note: remember to give the application give enough context to determine action
internal/process/process.go
package process
import (
"errors"
)
var (
ErrNotFound = errors.New("Not Found")
ErrConnFail = errors.New("Connection Failed")
)
// function Process is a dummy function that returns error for certain arguments received
func Process(i int) error {
switch i {
case 6:
return ErrNotFound
case 7:
return ErrConnFail
default:
return nil
}
}
cmd/servi/main.go
package main
import (
"log"
p "../../internal/process"
)
func main() {
// sample: generic logging on any failure
err := p.Process(6)
if err != nil {
log.Println("FAIL", err)
}
// sample: this application decides how to handle error based on context
err = p.Process(7)
if err != nil {
switch err {
case p.ErrNotFound:
log.Println("DOESN'T EXIST. TRY ANOTHER")
case p.ErrConnFail:
log.Println("UNABLE TO CONNECT; WILL RETRY LATER")
}
}
}
in case I've more things that I need to pass (like logger) what will be the approach/ pattern here
Dependency injection is always a good first choice. Consider others only when the simplest implementation is insufficient.
The code below 'wires' the template and logger packages together using dependency injection and first-class function passing.
internal/logs/logger.go
package logger
import (
"github.com/sirupsen/logrus"
"github.com/x-cray/logrus-prefixed-formatter"
"os"
)
var Logger *logrus.Logger
func NewLogger() *logrus.Logger {
var level logrus.Level
level = LogLevel("info")
logger := &logrus.Logger{
Out: os.Stdout,
Level: level,
Formatter: &prefixed.TextFormatter{
DisableColors: true,
TimestampFormat: "2009-06-03 11:04:075",
},
}
Logger = logger
return Logger
}
func LogLevel(lvl string) logrus.Level {
switch lvl {
case "info":
return logrus.InfoLevel
case "error":
return logrus.ErrorLevel
default:
panic("Not supported")
}
}
internal/template/template.go
package template
import (
"fmt"
"github.com/sirupsen/logrus"
)
type Template struct {
Name string
logger *logrus.Logger
}
// factory function New accepts a logging function and some data
func New(logger *logrus.Logger, data string) *Template {
return &Template{data, logger}
}
// dummy function DoSomething should do something and log using the given logger
func (t *Template) DoSomething() {
t.logger.Info(fmt.Sprintf("%s, %s", t.Name, "did something"))
}
cmd/servi2/main.go
package main
import (
"../../internal/logs"
"../../internal/template"
)
func main() {
// wire our template and logger together
loggingFunc := logger.NewLogger()
t := template.New(loggingFunc, "Penguin Template")
// use the stuff we've set up
t.DoSomething()
}
Hope this helps. Cheers,
There are a few possibilities, each with their own tradeoffs.
Pass in dependencies explicitly
Pass in a context with all dependencies
Use a struct for context to methods
Use a package global and import
All of them have their place in different circumstances, and all have different tradeoffs:
This is very clear, but can get very messy and clutters your functions with lots of dependencies. It makes tests easy to mock if that is your thing.
This is my least favourite option, as it is seductive at first but quickly grows to a god object which mixes lots of unrelated concerns. Avoid.
This can be very useful in many cases, for example many people approach db access this way. Also easy to mock if required. This allows you to set/share dependencies without changing code at point of use - basically invert control in a neater way than passing in explicit parameters.
This has the virtue of clarity and orthogonality. It will require you to add separate setup for say tests vs production, initialising the package to the proper state prior to using it. Some dislike it for that reason.
I prefer the package global approach for something like logging, as long as a very simple signature is used. I don’t tend to test log output, or change logger often. Consider what you really need from logs, and whether it might be best just to use the built in log package, and perhaps try one of these approaches to see which suit you.
Examples in pseudo code for brevity:
// 1. Pass in dependencies explicitly
func MyFunc(log LoggerInterface, param, param)
// 2. Pass in a context with all dependencies
func MyFunc(c *ContextInterface, param, param)
// 3. Use a struct for context to methods
func (controller *MyController) MyFunc(param, param) {
controller.Logger.Printf("myfunc: doing foo:%s to bar:%s",foo, bar)
}
// 4. Use a package global and import
package log
var DefaultLogger PrintfLogger
func Printf(format, args) {DefaultLogger.Printf(format, args)}
// usage:
import "xxx/log"
log.Printf("myfunc: doing foo:%s to bar:%s",foo, bar)
I prefer this option 4 for logging and config, and even db access as it is explicit, flexible, and allows easily switching out another logger - just change the imports, no interfaces to change. The calculation on which is best depends on circumstances though - if you were writing a library for widespread use you might prefer to allow setting a logger at the struct level.
I'd usually require setup with an explicit setup on app startup, and always avoid using init functions as they are confusing and difficult to test.
I've always explicitly passed a *logrus.Logger into functions (or occasionally objects) that needed it. That avoids weird dependency cycles, makes it explicit that logging is part of the things this function does, and makes it easier to reuse the function or module elsewhere. The initial log object is created and configured in my main function (likely after some command-line argument handling).
I’ve application which should use log in state of debug. i.e. all the logs that I want to
provide is like log.debug
I’ve read about it and find the following
https://github.com/Sirupsen/logrus
https://github.com/uber-go/zap
My question is how should I “tell” to the program that now run at debug mode an then
all the logs will be printed since this I believe should come from outside …
example will be very helpful since Im new to golfing .
Ok, a really simple example of the approach I suggested in the comment:
package main
import (
"os"
"github.com/sirupsen/logrus"
)
func init() {
lvl, ok := os.LookupEnv("LOG_LEVEL")
// LOG_LEVEL not set, let's default to debug
if !ok {
lvl = "debug"
}
// parse string, this is built-in feature of logrus
ll, err := logrus.ParseLevel(lvl)
if err != nil {
ll = logrus.DebugLevel
}
// set global log level
logrus.SetLevel(ll)
}
func main() {
logrus.Debug("Will only be visible if the loglevel permits it")
}
The original comment:
Check codebases that are out there. In go, the common way to do that is to load configuration via environment variables (eg LOG_LEVEL, and use os.Getenv or a config package to load the value). That's why logrus for example allows you to set log levels via ints or strings.
Please, please: before you ask a question, read the basic info about the packages you use. Even the github repo for logrus' main README contains an example setting the log level to a specific level:
https://github.com/sirupsen/logrus#example
If the only thing you need is check level before printing you can create you own thin wrapper for standard logger.
This will help to better understand how they work. Feel free to ask any questions.
package main
import (
"log"
)
type MyLog struct {
PrintDebug bool
}
func (m *MyLog) Debug(args ...interface{}) {
if m.PrintDebug {
m.Print(args...)
}
}
func (m *MyLog) Print(args ...interface{}) {
log.Print(args...)
}
func main() {
ml := MyLog{}
ml.Debug("PrintDebig = false, so no oitput")
ml.Print("this will be printed anyway")
ml.PrintDebug = true
ml.Debug("Hello, playground")
}
https://play.golang.org/p/gKxQtC9NqX
I am currently writing a lot of unit tests for my package that runs on GAE Go. The package in question is focused on data saving and loading to and from appengine/datastore. As such, I have about 20 unit test files that look a bit like this:
package Data
import (
"appengine"
"appengine/aetest"
. "gopkg.in/check.v1"
"testing"
)
func TestUsers(t *testing.T) { TestingT(t) }
type UsersSuite struct{}
var _ = Suite(&UsersSuite{})
const UserID string = "UserID"
func (s *UsersSuite) TestSaveLoad(cc *C) {
c, err := aetest.NewContext(nil)
cc.Assert(err, IsNil)
defer c.Close()
...
As a result, each individual test file appears to be starting its own version of devappserver:
Repeat this 20 times and my unit tests run for over 10 minutes.
I am wondering, how can I speed up the execution of my testing suite? Should I have just one file that creates aetest.NewContext and passes that onwards, or is it due to me using separate Suites for each unit test? How can I speed this thing up?
You can use a custom TestMain function:
var ctx aetest.Context
var c aetest.Context
func TestMain(m *testing.M) {
var err error
ctx, err = aetest.NewContext(nil)
if err != nil {
panic(err)
}
code := m.Run() // this runs the tests
ctx.Close()
os.Exit(code)
}
func TestUsers(t *testing.T) {
// use ctx here
}
This way the dev server is started once for all the tests. More details on TestMain are available here: http://golang.org/pkg/testing/#hdr-Main.