How to use golang viper Watchconfig & onConfigChange - go

I am trying to read my application configuration using golang viper and would like to read the latest config always. Please find my code below
config.go
package config
import (
"github.com/spf13/viper"
"log"
"github.com/fsnotify/fsnotify"
"time"
)
type Reader interface {
GetAllKeys() []string
Get(key string) interface{}
GetBool(key string) bool
GetString(key string) string
}
type viperConfigReader struct {
viper *viper.Viper
}
var TestConfReader *viperConfigReader
func (v viperConfigReader) GetAllKeys() []string{
return v.viper.AllKeys()
}
func (v viperConfigReader) Get(key string) interface{} {
return v.viper.Get(key)
}
func (v viperConfigReader) GetBool(key string) bool {
return v.viper.GetBool(key)
}
func (v viperConfigReader) GetString(key string) string {
return v.viper.GetString(key)
}
func init() {
v:= viper.New()
v.SetConfigName("test")
v.AddConfigPath("/tmp/")
err := v.ReadInConfig()
if err != nil {
log.Panic("Not able to read configuration", err.Error())
}
TestConfReader = &viperConfigReader{
viper: v,
}
go func() {
for {
time.Sleep(time.Second * 5)
v.WatchConfig()
v.OnConfigChange(func(e fsnotify.Event) {
log.Println("config file changed", e.Name)
})
}
}()
}
main.go
package main
import (
"github.com/xxxx/xxxx/config"
"log"
"time"
)
func main() {
conf := config.TestConfReader
log.Println(conf.GetAllKeys())
log.Println(conf.GetString("test1"))
time.Sleep(20 * time.Second)
log.Println(conf.GetString("test1"))
}
When the main program is running, I tried to update the config and expected to see OnConfigChange log message but it never showed up.
How can I fix this program ?.
Can someone provide an example of using viper watchconfig & onconfigchange methods to read the latest config

The comment by ymonad is on the right track, depending on your OS you may be experiencing problems with viper/fsnotify.
For example, I ran your example code on Mac OS X (Sierra) and noticed the same symptom you described: when the config file is in /tmp the viper WatchConfig call was not causing the viper OnConfigChange function to be called.
However, when I change the AddConfigPath to use the current working directory or my home directory then I do see logging from your OnConfigChange function. For example, try:
v.AddConfigPath("./")
I'd recommend experimenting with different directory locations to see if this is some sort of viper/fsnotify bug or limitation. For some reason, it doesn't appear to detect changes from the /tmp directory on Mac OS X, at least, it doesn't for my setup. I couldn't find any mention of problems with /tmp on OS X, but the fsnotify CONTRIBUTING.md does mention limitations with "shared" directories under Vagrant so perhaps there are some filesystem configurations that do not trigger notifications:
Notice: fsnotify file system events won't trigger in shared folders. The tests get around this limitation by using the /tmp directory.
Also, you don't need to keep calling WatchConfig and OnConfigChange via your goroutine. You can eliminate the goroutine completely and just move the relevant lines into init.

Related

How to interact with Telegraf using Go external plugin?

I have a dummy GO plugin, using that, i want send data to telegraf. But, i'm not able to find any way to send data from the plugin to telegraf.
this external Go plugin looks like below
package main
import (
"fmt"
"time"
)
type greeting string
type n int
func (g greeting) Greet() {
for i := 0; i < 10; i++ {
timer1 := time.NewTimer(2 * time.Second)
<-timer1.C
fmt.Println("value ", i)
sum := 0
sum += i * 100
fmt.Println(sum)
}
}
// exported
var Greeter greeting
And the main file looks like
import (
"fmt"
"os"
"plugin"
)
type Greeter interface {
Greet()
}
func main() {
var mod string
mod = "./eng/eng.so"
// load module
// 1. open the so file to load the symbols
plug, err := plugin.Open(mod)
if err != nil {
fmt.Println(err)
os.Exit(1)
}
// 2. look up a symbol (an exported function or variable)
// in this case, variable Greeter
symGreeter, err := plug.Lookup("Greeter")
if err != nil {
fmt.Println(err)
os.Exit(1)
}
// 3. Assert that loaded symbol is of a desired type
// in this case interface type Greeter (defined above)
var greeter Greeter
greeter, ok := symGreeter.(Greeter)
if !ok {
fmt.Println("unexpected type from module symbol")
os.Exit(1)
}
// 4. use the module
greeter.Greet()
}
Can anyone help me find a way or a direction on how to make interaction between GO plugin and telegraf work. Any heads up is appreciated.
Currently, Telegraf does not support the use of external plugins.
An issue was made and closed regarding this requested feature.
If you were planning on writing code in Golang I would suggest creating a custom Telegraf plugin. Telegraf plugins do not use go external plugin package instead they are built with Telegraf itself. Your plugin must satisfy the specific interface based on the type of plugin you want to create. Based on your question I will assume that you desire to create an input plugin, Telegraf has examples for creating each type of plugin in their source code.

Using logger/configs in multi packages best practice for Golang productive

I've the following project structure:
myGithubProject/
|---- cmd
|----- command
|----- hello.go
|---- internal
|----- template
|----- template.go
|----- log
|----- logger.go
main.go
The log and the template are on the same level (under internal package)
In logger.go Im using logrus as logger with some config sand I want to use the logger.go object inside the template package.
How should I do that in a clean way ?
Currently I use it with import logger inside my template.go file,
And under internal package I’ve 6 more packages which needs this logger. and each of them is depend on it.
(On the log package), is there a nice in go to handle it ?
update:
in case I've more things that I need to pass (like logger) what will be the approach/ pattern here ? maybe using dependency injection ? interface ? other clean approach...
I need some best practice full example how can I use logger inside the hello.go file and also in template.go.
this is my project
cliProject/main.go
package main
import (
"cliProject/cmd"
"cliProject/internal/logs"
)
func main() {
cmd.Execute()
logs.Logger.Error("starting")
}
**cliProject/cmd/root.go**
package cmd
import (
"fmt"
"github.com/spf13/cobra"
)
var rootCmd = &cobra.Command{
Use: "cliProject",
Short: "A brief description of your application",
}
func Execute() {
if err := rootCmd.Execute(); err != nil {
fmt.Println(err)
}
}
**cliProject/cmd/serve.go**
package cmd
import (
"cliProject/internal/logs"
"fmt"
"github.com/spf13/cobra"
)
// serveCmd represents the serve command
var serveCmd = &cobra.Command{
Use: "serve",
Short: "A brief description of your command",
Run: func(cmd *cobra.Command, args []string) {
fmt.Println("serve called")
startServe()
stoppingServe()
},
}
func init() {
rootCmd.AddCommand(serveCmd)
serveCmd.Flags().BoolP("toggle", "t", false, "Help message for toggle")
}
func startServe() {
logs.Logger.Error("starting from function serve")
}
func stoppingServe() {
logs.Logger.Error("stoping from function serve")
}
**cliProject/cmd/start.go**
package cmd
import (
"cliProject/internal/logs"
"github.com/spf13/cobra"
)
// startCmd represents the start command
var startCmd = &cobra.Command{
Use: "start",
Short: "Start command",
Run: func(cmd *cobra.Command, args []string) {
// Print the logs via logger from internal
logs.Logger.Println("start called inline")
// example of many functions which should use the logs...
start()
stopping()
},
}
func init() {
logs.NewLogger()
rootCmd.AddCommand(startCmd)
startCmd.Flags().BoolP("toggle", "t", false, "Help message for toggle")
}
func start() {
logs.Logger.Error("starting from function start")
}
func stopping() {
logs.Logger.Error("stoping from function start")
}
**cliProject/internal/logs/logger.go**
package logs
import (
"github.com/sirupsen/logrus"
"github.com/x-cray/logrus-prefixed-formatter"
"os"
)
var Logger *logrus.Logger
func NewLogger() *logrus.Logger {
var level logrus.Level
level = LogLevel("info")
logger := &logrus.Logger{
Out: os.Stdout,
Level: level,
Formatter: &prefixed.TextFormatter{
DisableColors: true,
TimestampFormat: "2009-06-03 11:04:075",
},
}
Logger = logger
return Logger
}
func LogLevel(lvl string) logrus.Level {
switch lvl {
case "info":
return logrus.InfoLevel
case "error":
return logrus.ErrorLevel
default:
panic("Not supported")
}
}
this is how it looks
And under internal package I’ve 6 more packages which needs this logger. and each of them is depend on it. (On the log package), is there a nice in go to handle it ?
A good general principle would be to respect the application's choice (whether to log or not) instead of setting policy.
Let Go pkgs in your internal directory be support packages that
will only return error in case of problems
won't log (console or otherwise)
won't panic
Let your application (packages in your cmd directory) decide what the appropriate behavior in case of an error (log / graceful shutdown / recover to 100% integrity)
This would streamline development by having logging at a particular layer only. Note: remember to give the application give enough context to determine action
internal/process/process.go
package process
import (
"errors"
)
var (
ErrNotFound = errors.New("Not Found")
ErrConnFail = errors.New("Connection Failed")
)
// function Process is a dummy function that returns error for certain arguments received
func Process(i int) error {
switch i {
case 6:
return ErrNotFound
case 7:
return ErrConnFail
default:
return nil
}
}
cmd/servi/main.go
package main
import (
"log"
p "../../internal/process"
)
func main() {
// sample: generic logging on any failure
err := p.Process(6)
if err != nil {
log.Println("FAIL", err)
}
// sample: this application decides how to handle error based on context
err = p.Process(7)
if err != nil {
switch err {
case p.ErrNotFound:
log.Println("DOESN'T EXIST. TRY ANOTHER")
case p.ErrConnFail:
log.Println("UNABLE TO CONNECT; WILL RETRY LATER")
}
}
}
in case I've more things that I need to pass (like logger) what will be the approach/ pattern here
Dependency injection is always a good first choice. Consider others only when the simplest implementation is insufficient.
The code below 'wires' the template and logger packages together using dependency injection and first-class function passing.
internal/logs/logger.go
package logger
import (
"github.com/sirupsen/logrus"
"github.com/x-cray/logrus-prefixed-formatter"
"os"
)
var Logger *logrus.Logger
func NewLogger() *logrus.Logger {
var level logrus.Level
level = LogLevel("info")
logger := &logrus.Logger{
Out: os.Stdout,
Level: level,
Formatter: &prefixed.TextFormatter{
DisableColors: true,
TimestampFormat: "2009-06-03 11:04:075",
},
}
Logger = logger
return Logger
}
func LogLevel(lvl string) logrus.Level {
switch lvl {
case "info":
return logrus.InfoLevel
case "error":
return logrus.ErrorLevel
default:
panic("Not supported")
}
}
internal/template/template.go
package template
import (
"fmt"
"github.com/sirupsen/logrus"
)
type Template struct {
Name string
logger *logrus.Logger
}
// factory function New accepts a logging function and some data
func New(logger *logrus.Logger, data string) *Template {
return &Template{data, logger}
}
// dummy function DoSomething should do something and log using the given logger
func (t *Template) DoSomething() {
t.logger.Info(fmt.Sprintf("%s, %s", t.Name, "did something"))
}
cmd/servi2/main.go
package main
import (
"../../internal/logs"
"../../internal/template"
)
func main() {
// wire our template and logger together
loggingFunc := logger.NewLogger()
t := template.New(loggingFunc, "Penguin Template")
// use the stuff we've set up
t.DoSomething()
}
Hope this helps. Cheers,
There are a few possibilities, each with their own tradeoffs.
Pass in dependencies explicitly
Pass in a context with all dependencies
Use a struct for context to methods
Use a package global and import
All of them have their place in different circumstances, and all have different tradeoffs:
This is very clear, but can get very messy and clutters your functions with lots of dependencies. It makes tests easy to mock if that is your thing.
This is my least favourite option, as it is seductive at first but quickly grows to a god object which mixes lots of unrelated concerns. Avoid.
This can be very useful in many cases, for example many people approach db access this way. Also easy to mock if required. This allows you to set/share dependencies without changing code at point of use - basically invert control in a neater way than passing in explicit parameters.
This has the virtue of clarity and orthogonality. It will require you to add separate setup for say tests vs production, initialising the package to the proper state prior to using it. Some dislike it for that reason.
I prefer the package global approach for something like logging, as long as a very simple signature is used. I don’t tend to test log output, or change logger often. Consider what you really need from logs, and whether it might be best just to use the built in log package, and perhaps try one of these approaches to see which suit you.
Examples in pseudo code for brevity:
// 1. Pass in dependencies explicitly
func MyFunc(log LoggerInterface, param, param)
// 2. Pass in a context with all dependencies
func MyFunc(c *ContextInterface, param, param)
// 3. Use a struct for context to methods
func (controller *MyController) MyFunc(param, param) {
controller.Logger.Printf("myfunc: doing foo:%s to bar:%s",foo, bar)
}
// 4. Use a package global and import
package log
var DefaultLogger PrintfLogger
func Printf(format, args) {DefaultLogger.Printf(format, args)}
// usage:
import "xxx/log"
log.Printf("myfunc: doing foo:%s to bar:%s",foo, bar)
I prefer this option 4 for logging and config, and even db access as it is explicit, flexible, and allows easily switching out another logger - just change the imports, no interfaces to change. The calculation on which is best depends on circumstances though - if you were writing a library for widespread use you might prefer to allow setting a logger at the struct level.
I'd usually require setup with an explicit setup on app startup, and always avoid using init functions as they are confusing and difficult to test.
I've always explicitly passed a *logrus.Logger into functions (or occasionally objects) that needed it. That avoids weird dependency cycles, makes it explicit that logging is part of the things this function does, and makes it easier to reuse the function or module elsewhere. The initial log object is created and configured in my main function (likely after some command-line argument handling).

Logger not able to write when being called in other functions

Note: Newbie for golang language
Here is the sample program hello.go I wrote to check the behavior and I am seeing the issue where I don't see anything being written by Logger in any functions other than init().
package main
import (
"fmt"
"os"
"io"
"log"
)
var (
testLogger *log.Logger
)
func init() {
test_log := "/tmp/t1.log"
fmt.Printf("Logs are saved to %v\n", test_log)
f, err := os.OpenFile(test_log, os.O_CREATE|os.O_APPEND|os.O_WRONLY, 0600)
if err != nil {
fmt.Printf("ERROR Can't create log file! Reason: %v \n", err)
os.Exit(1)
}
defer f.Close()
multiWriter := io.MultiWriter(f)
testLogger = log.New(multiWriter, "", log.Ldate|log.Ltime|log.Lshortfile)
testLogger.Printf("in init..")
}
func main() {
pretest()
test()
testLogger.Printf("Back to main ... ")
}
func pretest() {
testLogger.Printf("In pretest ... ")
}
func test() {
testLogger.Printf("in test..")
}
Here is the output and content of the file being written to:
➜ ./hello
Logs are saved to /tmp/t1.log
➜ cat /tmp/t1.log
2018/06/28 11:23:25 hello.go:27: in init..
AFAIK, The testLogger being shared in the same package should be accessible by each function and able to being used. Please correct me if my understanding is wrong? Am I missing anything in the code? Please provide any pointer or reference on this issue? Thanks.
defer f.Close()
You are doing that defer in the init function, meaning that the file will be closed as soon as the init function finishes.
Since the init function runs before main, the file will be closed already when you try to write from all your other functions.
Move that defer to the main function so that it is closed when your program exits.
Note that in this specific case, you don't even need to close the os.File since by default File has a finalizer to close itself when it is collected by the GC (see newFile function here: https://golang.org/src/os/file_unix.go).

passing channel param to a dynamic loaded func in golang

I'm new to golang and golang plugins. I'm having trouble passing a chan object to this functions. If I switch to int it works. Not sure what exactly I am missing. Thanks for any help.
dataunit.go
package main
type DataUnit struct {
i int
s string
}
modcounter.go
package main
import (
"fmt"
// "strconv"
)
type module string
//func (m module) RunMod(i int) {
func (m module) RunMod(in <-chan *DataUnit) {
//fmt.Println("Hello Universe " + strconv.Itoa(i))
fmt.Println("Hello Universe ")
n := <-in
fmt.Printf(n.s)
}
var Module module
modmain.go
package main
import (
"fmt"
"os"
"plugin"
)
type DataUnit struct {
i int
s string
}
type Module interface {
//RunMod(i int)
RunMod(in <-chan *DataUnit)
}
func main() {
out := make(chan *DataUnit, 2000)
plug, err := plugin.Open("./modcounter.so")
if err != nil {
fmt.Printf("FATAL (plugin.Open): " + err.Error())
os.Exit(1)
}
symModule, err := plug.Lookup("Module")
if err != nil {
fmt.Printf(err.Error())
os.Exit(1)
}
var module Module
module, ok:= symModule.(Module)
if !ok {
fmt.Println("unexpected type from module symbol")
os.Exit(1)
}
//module.RunMod(5)
module.RunMod(out)
}
go build -buildmode=plugin -o modcounter.so modcounter.go dataunit.go
go build modmain.go dataunit.go
./modmain
unexpected type from module symbol
If you are still learning golang then plugins are definitely not the place to start. I believe plugin support is still experimental and doesn't work on all OS - in 1.10 I believe Darwin was added to previous Linux.
A quick search reveals a pre-existing issue reported similar to yours -looks like it's in go 1.10:
https://github.com/golang/go/issues/24351
You can maybe track through the repository and get a branch or version where this is fixed, but whilst your learning you aren't going to have any confidence in whether problems with this area of functionality are with your code or the plugin system. I recommend therefore sticking to learning go with core language features for now.
In go is possible to pass channels of objects!!
For example, if you to use <-chan http.Header will works fine. The question is that the params must be shared between modules and application. So if you reallocate DataUnit for another package will work.
My test was structured like:
My interface:
//in modmain.go
type Module interface {
RunMod(in <-chan *mydata.DataUnit)
}
My module:
//in modcounter.go
func (m module) RunMod(in <-chan *mydata.DataUnit) {
fmt.Println("Hello Universe ")
n := <-in
fmt.Printf("%v", n.S)
}
My data:
//in dataunit.go
type DataUnit struct {
I int //export field
S string //export field
}
The result:
P.S.: Docker with golang 1.10 was used for tests.
#in Dockerfile
FROM golang:1.10
COPY . /go/
RUN export GOPATH=$GOPATH:/go/
RUN cd /go/src/mydata && go build dataunit.go
RUN cd /go/src/app && go build modmain.go
RUN cd /go/src/app && go build -buildmode=plugin -o modcounter.so modcounter.go
WORKDIR /go/src/app
RUN ls -l
CMD ["/go/src/app/modmain"]

How to use debug log in golang

I’ve application which should use log in state of debug. i.e. all the logs that I want to
provide is like log.debug
I’ve read about it and find the following
https://github.com/Sirupsen/logrus
https://github.com/uber-go/zap
My question is how should I “tell” to the program that now run at debug mode an then
all the logs will be printed since this I believe should come from outside …
example will be very helpful since Im new to golfing .
Ok, a really simple example of the approach I suggested in the comment:
package main
import (
"os"
"github.com/sirupsen/logrus"
)
func init() {
lvl, ok := os.LookupEnv("LOG_LEVEL")
// LOG_LEVEL not set, let's default to debug
if !ok {
lvl = "debug"
}
// parse string, this is built-in feature of logrus
ll, err := logrus.ParseLevel(lvl)
if err != nil {
ll = logrus.DebugLevel
}
// set global log level
logrus.SetLevel(ll)
}
func main() {
logrus.Debug("Will only be visible if the loglevel permits it")
}
The original comment:
Check codebases that are out there. In go, the common way to do that is to load configuration via environment variables (eg LOG_LEVEL, and use os.Getenv or a config package to load the value). That's why logrus for example allows you to set log levels via ints or strings.
Please, please: before you ask a question, read the basic info about the packages you use. Even the github repo for logrus' main README contains an example setting the log level to a specific level:
https://github.com/sirupsen/logrus#example
If the only thing you need is check level before printing you can create you own thin wrapper for standard logger.
This will help to better understand how they work. Feel free to ask any questions.
package main
import (
"log"
)
type MyLog struct {
PrintDebug bool
}
func (m *MyLog) Debug(args ...interface{}) {
if m.PrintDebug {
m.Print(args...)
}
}
func (m *MyLog) Print(args ...interface{}) {
log.Print(args...)
}
func main() {
ml := MyLog{}
ml.Debug("PrintDebig = false, so no oitput")
ml.Print("this will be printed anyway")
ml.PrintDebug = true
ml.Debug("Hello, playground")
}
https://play.golang.org/p/gKxQtC9NqX

Resources