I've been exploring configuration management in Go using the viper package, which I learned about in another question. I'm having trouble understanding how to initialize a new configuration. What I want to do is to find system configurations if they exist, then user configuration, and then, if all else fails, to create (and then use) a new user configuration from defaults. The next time I run the app, I'd expect to find the configuration that I previously created. I'm using it like this:
import (
"fmt"
"github.com/spf13/viper"
"os"
"os/user"
"path/filepath"
)
usr, err := user.Current()
appMode := "test" // or "production" or...
configDir := filepath.Join(usr.HomeDir, ".myapp")
config := viper.New()
config.SetConfigName(appMode)
config.SetConfigType("json")
config.SetDefault("group1.key1", "value1.1")
config.SetDefault("group1.key2", 1234)
config.SetDefault("group2.key1", "value2.1")
config.SetDefault("group2.key2", 5678)
config.AddConfigPath("/usr/share/myapp/config")
config.AddConfigPath("/usr/local/share/myapp/config")
config.AddConfigPath(configDir)
if err := config.ReadInConfig(); err != nil {
filename := filepath(configDir, fmt.Sprintf("%s.json", appMode))
_, err := os.Create(filename)
if err != nil {
panic(fmt.Stringf("Failed to create %s", filename))
}
}
if err := config.ReadInConfig(); err != nil {
panic("Created %s, but Viper failed to read it: %s",
filename, err)
}
At this point, I would like for ~/.myapp/test.json to be created for me with the following:
{
"group1": {
"key1": "value1.1",
"key2": 1234
},
"group2": {
"key1": "value2.1",
"key2": 5678
}
}
The result is that the file is empty, and the second attempt to read the file also fails with the message "open : no such file or directory" even though it exists. If I hand-edit the file, Viper finds it and parses it. Obviously I can create the JSON file programatically, but this seems like such an obvious use case that I must be missing something here.
Related
I'm iterating through a mounted folder via filePath.Walk method in golang, but it returns the hidden files as well. I have to skip those hidden files.
For MaxOS and Linux, we can detect hidden file via .prefix in the filename, but for windows, when I'm trying to us this method GetFileAttributes, provided by "syscall", it's not detecting these methods and throwing an error.
Using below method to fetch the file
err := filepath.Walk(prefix, func(docPath string, f os.FileInfo, err error) error {
Below is how I'm trying to detect the hidden files
import (
"runtime"
"syscall"
)
func IsHiddenFile(filename string) (bool, error) {
if runtime.GOOS == "windows" {
pointer, err := syscall.UTF16PtrFromString(filename)
if err != nil {
return false, err
}
attributes, err := syscall.GetFileAttributes(pointer)
if err != nil {
return false, err
}
return attributes&syscall.FILE_ATTRIBUTE_HIDDEN != 0, nil
} else {
// unix/linux file or directory that starts with . is hidden
if filename[0:1] == "." {
return true, nil
}
}
return false, nil
}
Error :
.../ undefined: syscall.UTF16PtrFromString
.../ undefined: syscall.GetFileAttributes
.../ undefined: syscall.FILE_ATTRIBUTE_HIDDEN
I added this // +build windows in the start of the file before package name as suggested here : syscall variables undefined but it's sill not working and throwing the same error.
I need to know if go provides some common method to detect if a file is hidden or not? Or is there a way I can fetch all the files/folder in some mounted directory without receiving hidden files in the first place?
Really looking forward on receiving some feedback here, thanks.
EDIT : Fixed the above mentioned issue (please refer to the comments below), I also wants to know how can we detect the hidden file when we are connected with the remote server (SMB), the remoter system could be running any OS, and we compile these method based on the system it's running on. How can we detect Hidden files in that scenario ?
Conditional compilation is the right way to go, but it applies at source file level, so you need two separate files.
For example:
hidden_notwin.go:
//go:build !windows
package main
func IsHiddenFile(filename string) (bool, error) {
return filename[0] == '.', nil
}
hidden_windows.go:
//go:build windows
package main
import (
"syscall"
)
func IsHiddenFile(filename string) (bool, error) {
pointer, err := syscall.UTF16PtrFromString(filename)
if err != nil {
return false, err
}
attributes, err := syscall.GetFileAttributes(pointer)
if err != nil {
return false, err
}
return attributes&syscall.FILE_ATTRIBUTE_HIDDEN != 0, nil
}
Note that //go:build windows tag above is optional - the _windows source file suffix does the magic already. For more details see go command - Build constraints.
I want to create a yaml file from a current tmpl file. Basically I want to insert values in the sample.tmpl files stored in the /templates folder and create a new yaml file in the same folder sample.yml
My sample.tmpl looks like
url : {{ .host }}
namespace: {{ .namespace }}
I'm using the following function:
func ApplyTemplate(filePath string) (err error) {
// Variables - host, namespace
type Eingest struct {
host string
namespace string
}
ei := Eingest{host: "example.com", namespace: "finance"}
var templates *template.Template
var allFiles []string
files, err := ioutil.ReadDir(filePath)
if err != nil {
fmt.Println(err)
}
for _, file := range files {
filename := file.Name()
fullPath := filePath + "/" + filename
if strings.HasSuffix(filename, ".tmpl") {
allFiles = append(allFiles, fullPath)
}
}
fmt.Println("Files in path: ", allFiles)
// parses all .tmpl files in the 'templates' folder
templates, err = template.ParseFiles(allFiles...)
if err != nil {
fmt.Println(err)
}
s1 := templates.Lookup("sample.tmpl")
s1.ExecuteTemplate(os.Stdout, "sample.yml", ei)
fmt.Println()
return
}
s1.ExecuteTemplate() writes to stdout. How can I create a new file in the same folder? I believe something similar is used to build kubernetes yaml files. How do we achieve this using golang template package?
First: since you've already looked up the template, you should use template.Execute instead, but the same works with ExecuteTemplate.
text.Template.Execute takes a io.Writer as first argument. This is an interface with a single method: Write(p []byte) (n int, err error).
Any type that has that method implements the interface and can be used as a valid argument. One such type is a os.File. Simply create a new os.File object and pass it to Execute as follows:
// Build the path:
outputPath := filepath.Join(filepath, "sample.yml")
// Create the file:
f, err := os.Create(outputPath)
if err != nil {
panic(err)
}
defer f.Close() // don't forget to close the file when finished.
// Write template to file:
err = s1.Execute(f, ei)
if err != nil {
panic(err)
}
Note: don't forget to check whether s1 is nil, as documented in template.Lookup.
I'm trying to parse this yaml and I want to get the values of the run entry (test1 or test2) without success, here is my working example.
im a bit get lost with the map inside map :( ,
this is given yaml which I couldent change ...
any idea how could I got those values
package main
import (
"fmt"
"log"
"gopkg.in/yaml.v2"
)
var runContent = []byte(`
version: "3.2"
run-parameters:
before:
run-parameters:
run: test1
after:
run-parameters:
run: test2
`)
type FTD struct {
Version string `yaml:"version,omitempty"`
BuildParams *RunParams `yaml:"run-parameters,omitempty"`
}
type RunParams struct {
BeforeExec map[string]interface{} `yaml:"before,omitempty"`
AfterExec map[string]interface{} `yaml:"after,omitempty"`
}
func main() {
runners := &FTD{}
// parse mta yaml
err := yaml.Unmarshal(runContent, runners)
if err != nil {
log.Fatalf("Error : %v", err)
}
for k, v := range runners.BuildParams.BeforeExec {
fmt.Println(k, v.(interface{}))
}
}
This is working example
https://play.golang.org/p/qTqUJy3md0c
also I've tried with
this is working
run := runners.BuildParams.BeforeExec["run-parameters"].(map[interface{}]interface{})["run"]
fmt.Println("run: ", run)
what I've tried is this which works but what happens if the run value is empty or no entry at all,this will cause a dump how can I overcome this ?
what I've tried is this which works but what happens if the run value is empty or no entry at all,this will cause a dump how can I overcome this ?
You can do
runParams, ok := runners.BuildParams.BeforeExec["run-parameters"]
if !ok {
// handle lack of "run-parameters" in BeforeExec
}
runParamsMap, ok := runParams.(map[interface{}]interface{})
if !ok {
// handle "run-parameters" not being a map
}
run, ok := runParamsMap["run"]
if !ok {
// handle lack of "run" inside "run-parameters"
}
runStr, ok := run.(string)
if !ok {
// handle "run" not being a string
}
fmt.Println("run: ", runStr)
This is quite verbose so you could use something like https://github.com/jmoiron/jsonq, where you can specify a "path" to the desired value nested inside several levels of maps. Despite the "json" in the name, this library works with map[string]interface{} and not json files. But note that the library you use for yaml unmarshalling results in map[interface{}]interface{} instead of map[string]interface{} and you will have to use a different one in order for it to work with jsonq.
run, err := jsonq.NewQuery(runners.BuildParams.BeforeExec).String("run-parameters", "run")
if err != nil {
// handle all possible errors in one place
}
fmt.Println("run: ", run)
I am writing an application in which i need to record logs in two different files. For Example weblogs.go and debuglogs.go. I tried using the log4go but my requirement is i need the logger to be created in main file and be accessible in sub directory as major of decoding and logging is done in sub file. Can anybody please help with that?
Here's one way to do it, using the standard log package:
package main
import (
"io"
"log"
"os"
)
func main() {
f1, err := os.Create("/tmp/file1")
if err != nil {
panic(err)
}
defer f1.Close()
f2, err := os.Create("/tmp/file2")
if err != nil {
panic(err)
}
defer f2.Close()
w := io.MultiWriter(os.Stdout, f1, f2)
logger := log.New(w, "logger", log.LstdFlags)
myfunc(logger)
}
func myfunc(logger *log.Logger) {
logger.Print("Hello, log file!!")
}
Notes:
io.MultiWriter is used to combine several writers together. Here, it creates a writer w - a write to w will go to os.Stdout as well as two files
log.New lets us create a new log.Logger object with a custom writer
The log.Logger object can be passed around to functions and used by them to log things
This is how you can manage logs for debugging and prod envoirments with multi log files:
First, make a type for managing multiple loggers you can make a separate file for this like logging.go
type LOGGER struct {
debug *log.Logger
prod *log.Logger
.
.
.
}
For getting the LOGGER pointer anywhere in your project the best approach will be to get it through the singleton pattern like
var lock = &sync.Mutex{}
var loggers *LOGGER
func GetLoggerInstance() *LOGGER {
if singleInstance == nil {
lock.Lock()
defer lock.Unlock()
if loggers == nil {
fmt.Println("Creating LOGGER instance now.")
loggers = &LOGGER{}
} else {
fmt.Println("LOGGER instance already created.")
}
} else {
fmt.Println("LOGGER instance already created.")
}
return loggers
}
Now setup logger for debug env in any pkg or directory level better is to setup it at the root level
logger := GetLoggerInstance()
f, err := os.OpenFile("path/to/debug/log/file", os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
if err != nil {
fmt.Println("debug log file not created", err.Error())
}
loggers.debug = log.New(f, "[DEBUG]", log.Ldate|log.Ltime|log.Lmicroseconds|log.LUTC)
loggers.debug.Println("This is debug log message")
With the same approach, you can also create prod or as many as you want.
logger := GetLoggerInstance()
f, err = os.OpenFile("path/to/prod/log/file", os.O_RDWR|os.O_CREATE|os.O_APPEND, 0666)
if err != nil {
fmt.Println("prod log file not created", err.Error())
}
loggers.prod = log.New(f, "[PROD]", log.Ldate|log.Ltime|log.Lmicroseconds|log.LUTC)
loggers.prod.Println("This is prod log message")
As title, I want to know how to use toml files from golang.
Before that, I show my toml examples. Is it right?
[datatitle]
enable = true
userids = [
"12345", "67890"
]
[datatitle.12345]
prop1 = 30
prop2 = 10
[datatitle.67890]
prop1 = 30
prop2 = 10
And then, I want to set these data as type of struct.
As a result I want to access child element as below.
datatitle["12345"].prop1
datatitle["67890"].prop2
Thanks in advance!
First get BurntSushi's toml parser:
go get github.com/BurntSushi/toml
BurntSushi parses toml and maps it to structs, which is what you want.
Then execute the following example and learn from it:
package main
import (
"github.com/BurntSushi/toml"
"log"
)
var tomlData = `title = "config"
[feature1]
enable = true
userids = [
"12345", "67890"
]
[feature2]
enable = false`
type feature1 struct {
Enable bool
Userids []string
}
type feature2 struct {
Enable bool
}
type tomlConfig struct {
Title string
F1 feature1 `toml:"feature1"`
F2 feature2 `toml:"feature2"`
}
func main() {
var conf tomlConfig
if _, err := toml.Decode(tomlData, &conf); err != nil {
log.Fatal(err)
}
log.Printf("title: %s", conf.Title)
log.Printf("Feature 1: %#v", conf.F1)
log.Printf("Feature 2: %#v", conf.F2)
}
Notice the tomlData and how it maps to the tomlConfig struct.
See more examples at https://github.com/BurntSushi/toml
A small update for the year 2019 - there is now newer alternative to BurntSushi/toml with a bit richer API to work with .toml files:
pelletier/go-toml (and documentation)
For example having config.toml file (or in memory):
[postgres]
user = "pelletier"
password = "mypassword"
apart from regular marshal and unmarshal of the entire thing into predefined structure (which you can see in the accepted answer) with pelletier/go-toml you can also query individual values like this:
config, err := toml.LoadFile("config.toml")
if err != nil {
fmt.Println("Error ", err.Error())
} else {
// retrieve data directly
directUser := config.Get("postgres.user").(string)
directPassword := config.Get("postgres.password").(string)
fmt.Println("User is", directUser, " and password is", directPassword)
// or using an intermediate object
configTree := config.Get("postgres").(*toml.Tree)
user := configTree.Get("user").(string)
password := configTree.Get("password").(string)
fmt.Println("User is", user, " and password is", password)
// show where elements are in the file
fmt.Printf("User position: %v\n", configTree.GetPosition("user"))
fmt.Printf("Password position: %v\n", configTree.GetPosition("password"))
// use a query to gather elements without walking the tree
q, _ := query.Compile("$..[user,password]")
results := q.Execute(config)
for ii, item := range results.Values() {
fmt.Println("Query result %d: %v", ii, item)
}
}
UPDATE
There is also spf13/viper that works with .toml config files (among other supported formats), but it might be a bit overkill in many cases.
UPDATE 2
Viper is not really an alternative (credits to #GoForth).
This issue was solved using recommended pkg BurntSushi/toml!!
I did as below and it's part of code.
[toml example]
[title]
enable = true
[title.clientinfo.12345]
distance = 30
some_id = 6
[Golang example]
type TitleClientInfo struct {
Distance int `toml:"distance"`
SomeId int `toml:"some_id"`
}
type Config struct {
Enable bool `toml:"enable"`
ClientInfo map[string]TitleClientInfo `toml:"clientinfo"`
}
var config Config
_, err := toml.Decode(string(d), &config)
And then, it can be used as I expected.
config.ClientInfo[12345].Distance
Thanks!
With solution Viper you can use a configuration file in JSON, TOML, YAML, HCL, INI and others properties formats.
Create file:
./config.toml
First import:
import (config "github.com/spf13/viper")
Initialize:
config.SetConfigName("config")
config.AddConfigPath(".")
err := config.ReadInConfig()
if err != nil {
log.Println("ERROR", err.Error())
}
And get the value:
config.GetString("datatitle.12345.prop1")
config.Get("datatitle.12345.prop1").(int32)
Doc.: https://github.com/spf13/viper
e.g.: https://repl.it/#DarlanD/Viper-Examples#main.go
I am using this [1] go-toml library.
It works great for my uses. I wrote this [2] go util to deal with containerd config.toml file using go-toml
[1]https://github.com/pelletier/go-toml
[2]https://github.com/prakashmirji/toml-configer
I am using spf13/viper
3rd packages
Status
Project
Starts
Forks
Alive
spf13/viper
Alive
BurntSushi/toml
usage of viper
I tried to use a table to put the code and the contents of the configuration file together, but obviously, the editing did not match the final result, so I put the image up in the hope that it would make it easier for you to compare
package main
import (
"github.com/spf13/viper"
"log"
"os"
)
func main() {
check := func(err error) {
if err != nil {
panic(err)
}
}
myConfigPath := "test_config.toml"
fh, err := os.OpenFile(myConfigPath, os.O_RDWR, 0666)
check(err)
viper.SetConfigType("toml") // do not ignore
err = viper.ReadConfig(fh)
check(err)
// Read
log.Printf("%#v", viper.GetString("title")) // "my config"
log.Printf("%#v", viper.GetString("DataTitle.12345.prop1")) // "30"
log.Printf("%#v", viper.GetString("dataTitle.12345.prop1")) // "30" // case-insensitive
log.Printf("%#v", viper.GetInt("DataTitle.12345.prop1")) // 30
log.Printf("%#v", viper.GetIntSlice("feature1.userids")) // []int{456, 789}
// Write
viper.Set("database", "newuser")
viper.Set("owner.name", "Carson")
viper.Set("feature1.userids", []int{111, 222}) // overwrite
err = viper.WriteConfigAs(myConfigPath)
check(err)
}
title = "my config"
[datatitle]
[datatitle.12345]
prop1 = 30
[feature1]
userids = [456,789]
database = "newuser" # New
title = "my config"
[datatitle]
[datatitle.12345]
prop1 = 30
[feature1]
userids = [111,222] # Update
[owner] # New
name = "Carson"