(Go) How to use toml files? - go

As title, I want to know how to use toml files from golang.
Before that, I show my toml examples. Is it right?
[datatitle]
enable = true
userids = [
"12345", "67890"
]
[datatitle.12345]
prop1 = 30
prop2 = 10
[datatitle.67890]
prop1 = 30
prop2 = 10
And then, I want to set these data as type of struct.
As a result I want to access child element as below.
datatitle["12345"].prop1
datatitle["67890"].prop2
Thanks in advance!

First get BurntSushi's toml parser:
go get github.com/BurntSushi/toml
BurntSushi parses toml and maps it to structs, which is what you want.
Then execute the following example and learn from it:
package main
import (
"github.com/BurntSushi/toml"
"log"
)
var tomlData = `title = "config"
[feature1]
enable = true
userids = [
"12345", "67890"
]
[feature2]
enable = false`
type feature1 struct {
Enable bool
Userids []string
}
type feature2 struct {
Enable bool
}
type tomlConfig struct {
Title string
F1 feature1 `toml:"feature1"`
F2 feature2 `toml:"feature2"`
}
func main() {
var conf tomlConfig
if _, err := toml.Decode(tomlData, &conf); err != nil {
log.Fatal(err)
}
log.Printf("title: %s", conf.Title)
log.Printf("Feature 1: %#v", conf.F1)
log.Printf("Feature 2: %#v", conf.F2)
}
Notice the tomlData and how it maps to the tomlConfig struct.
See more examples at https://github.com/BurntSushi/toml

A small update for the year 2019 - there is now newer alternative to BurntSushi/toml with a bit richer API to work with .toml files:
pelletier/go-toml (and documentation)
For example having config.toml file (or in memory):
[postgres]
user = "pelletier"
password = "mypassword"
apart from regular marshal and unmarshal of the entire thing into predefined structure (which you can see in the accepted answer) with pelletier/go-toml you can also query individual values like this:
config, err := toml.LoadFile("config.toml")
if err != nil {
fmt.Println("Error ", err.Error())
} else {
// retrieve data directly
directUser := config.Get("postgres.user").(string)
directPassword := config.Get("postgres.password").(string)
fmt.Println("User is", directUser, " and password is", directPassword)
// or using an intermediate object
configTree := config.Get("postgres").(*toml.Tree)
user := configTree.Get("user").(string)
password := configTree.Get("password").(string)
fmt.Println("User is", user, " and password is", password)
// show where elements are in the file
fmt.Printf("User position: %v\n", configTree.GetPosition("user"))
fmt.Printf("Password position: %v\n", configTree.GetPosition("password"))
// use a query to gather elements without walking the tree
q, _ := query.Compile("$..[user,password]")
results := q.Execute(config)
for ii, item := range results.Values() {
fmt.Println("Query result %d: %v", ii, item)
}
}
UPDATE
There is also spf13/viper that works with .toml config files (among other supported formats), but it might be a bit overkill in many cases.
UPDATE 2
Viper is not really an alternative (credits to #GoForth).

This issue was solved using recommended pkg BurntSushi/toml!!
I did as below and it's part of code.
[toml example]
[title]
enable = true
[title.clientinfo.12345]
distance = 30
some_id = 6
[Golang example]
type TitleClientInfo struct {
Distance int `toml:"distance"`
SomeId int `toml:"some_id"`
}
type Config struct {
Enable bool `toml:"enable"`
ClientInfo map[string]TitleClientInfo `toml:"clientinfo"`
}
var config Config
_, err := toml.Decode(string(d), &config)
And then, it can be used as I expected.
config.ClientInfo[12345].Distance
Thanks!

With solution Viper you can use a configuration file in JSON, TOML, YAML, HCL, INI and others properties formats.
Create file:
./config.toml
First import:
import (config "github.com/spf13/viper")
Initialize:
config.SetConfigName("config")
config.AddConfigPath(".")
err := config.ReadInConfig()
if err != nil {
log.Println("ERROR", err.Error())
}
And get the value:
config.GetString("datatitle.12345.prop1")
config.Get("datatitle.12345.prop1").(int32)
Doc.: https://github.com/spf13/viper
e.g.: https://repl.it/#DarlanD/Viper-Examples#main.go

I am using this [1] go-toml library.
It works great for my uses. I wrote this [2] go util to deal with containerd config.toml file using go-toml
[1]https://github.com/pelletier/go-toml
[2]https://github.com/prakashmirji/toml-configer

I am using spf13/viper
3rd packages
Status
Project
Starts
Forks
Alive
spf13/viper
Alive
BurntSushi/toml
usage of viper
I tried to use a table to put the code and the contents of the configuration file together, but obviously, the editing did not match the final result, so I put the image up in the hope that it would make it easier for you to compare
package main
import (
"github.com/spf13/viper"
"log"
"os"
)
func main() {
check := func(err error) {
if err != nil {
panic(err)
}
}
myConfigPath := "test_config.toml"
fh, err := os.OpenFile(myConfigPath, os.O_RDWR, 0666)
check(err)
viper.SetConfigType("toml") // do not ignore
err = viper.ReadConfig(fh)
check(err)
// Read
log.Printf("%#v", viper.GetString("title")) // "my config"
log.Printf("%#v", viper.GetString("DataTitle.12345.prop1")) // "30"
log.Printf("%#v", viper.GetString("dataTitle.12345.prop1")) // "30" // case-insensitive
log.Printf("%#v", viper.GetInt("DataTitle.12345.prop1")) // 30
log.Printf("%#v", viper.GetIntSlice("feature1.userids")) // []int{456, 789}
// Write
viper.Set("database", "newuser")
viper.Set("owner.name", "Carson")
viper.Set("feature1.userids", []int{111, 222}) // overwrite
err = viper.WriteConfigAs(myConfigPath)
check(err)
}
title = "my config"
[datatitle]
[datatitle.12345]
prop1 = 30
[feature1]
userids = [456,789]
database = "newuser" # New
title = "my config"
[datatitle]
[datatitle.12345]
prop1 = 30
[feature1]
userids = [111,222] # Update
[owner] # New
name = "Carson"

Related

Parsing prometheus metrics from file and updating counters

I've a go application that gets run periodically by a batch. Each run, it should read some prometheus metrics from a file, run its logic, update a success/fail counter, and write metrics back out to a file.
From looking at How to parse Prometheus data as well as the godocs for prometheus, I'm able to read in the file, but I don't know how to update app_processed_total with the value returned by expfmt.ExtractSamples().
This is what I've done so far. Could someone please tell me how should I proceed from here? How can I typecast the Vector I got into a CounterVec?
package main
import (
"fmt"
"net/http"
"strings"
"time"
"github.com/prometheus/client_golang/prometheus"
"github.com/prometheus/client_golang/prometheus/promhttp"
dto "github.com/prometheus/client_model/go"
"github.com/prometheus/common/expfmt"
"github.com/prometheus/common/model"
)
var (
fileOnDisk = prometheus.NewRegistry()
processedTotal = prometheus.NewCounterVec(prometheus.CounterOpts{
Name: "app_processed_total",
Help: "Number of times ran",
}, []string{"status"})
)
func doInit() {
prometheus.MustRegister(processedTotal)
}
func recordMetrics() {
go func() {
for {
processedTotal.With(prometheus.Labels{"status": "ok"}).Inc()
time.Sleep(5 * time.Second)
}
}()
}
func readExistingMetrics() {
var parser expfmt.TextParser
text := `
# HELP app_processed_total Number of times ran
# TYPE app_processed_total counter
app_processed_total{status="ok"} 300
`
parseText := func() ([]*dto.MetricFamily, error) {
parsed, err := parser.TextToMetricFamilies(strings.NewReader(text))
if err != nil {
return nil, err
}
var result []*dto.MetricFamily
for _, mf := range parsed {
result = append(result, mf)
}
return result, nil
}
gatherers := prometheus.Gatherers{
fileOnDisk,
prometheus.GathererFunc(parseText),
}
gathering, err := gatherers.Gather()
if err != nil {
fmt.Println(err)
}
fmt.Println("gathering: ", gathering)
for _, g := range gathering {
vector, err := expfmt.ExtractSamples(&expfmt.DecodeOptions{
Timestamp: model.Now(),
}, g)
fmt.Println("vector: ", vector)
if err != nil {
fmt.Println(err)
}
// How can I update processedTotal with this new value?
}
}
func main() {
doInit()
readExistingMetrics()
recordMetrics()
http.Handle("/metrics", promhttp.Handler())
http.ListenAndServe("localhost:2112", nil)
}
I believe you would need to use processedTotal.WithLabelValues("ok").Inc() or something similar to that.
The more complete example is here
func ExampleCounterVec() {
httpReqs := prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "http_requests_total",
Help: "How many HTTP requests processed, partitioned by status code and HTTP method.",
},
[]string{"code", "method"},
)
prometheus.MustRegister(httpReqs)
httpReqs.WithLabelValues("404", "POST").Add(42)
// If you have to access the same set of labels very frequently, it
// might be good to retrieve the metric only once and keep a handle to
// it. But beware of deletion of that metric, see below!
m := httpReqs.WithLabelValues("200", "GET")
for i := 0; i < 1000000; i++ {
m.Inc()
}
// Delete a metric from the vector. If you have previously kept a handle
// to that metric (as above), future updates via that handle will go
// unseen (even if you re-create a metric with the same label set
// later).
httpReqs.DeleteLabelValues("200", "GET")
// Same thing with the more verbose Labels syntax.
httpReqs.Delete(prometheus.Labels{"method": "GET", "code": "200"})
}
This is taken from the Promethus examples on Github
To use the value of vector you can do the following:
vectorFloat, err := strconv.ParseFloat(vector[0].Value.String(), 64)
if err != nil {
panic(err)
}
processedTotal.WithLabelValues("ok").Add(vectorFloat)
This is assuming you will only ever get a single vector value in your response. The value of the vector is stored as a string but you can convert it to a float with the strconv.ParseFloat method.

How to parse yaml and get value for interface- deep structue

I'm trying to parse this yaml and I want to get the values of the run entry (test1 or test2) without success, here is my working example.
im a bit get lost with the map inside map :( ,
this is given yaml which I couldent change ...
any idea how could I got those values
package main
import (
"fmt"
"log"
"gopkg.in/yaml.v2"
)
var runContent = []byte(`
version: "3.2"
run-parameters:
before:
run-parameters:
run: test1
after:
run-parameters:
run: test2
`)
type FTD struct {
Version string `yaml:"version,omitempty"`
BuildParams *RunParams `yaml:"run-parameters,omitempty"`
}
type RunParams struct {
BeforeExec map[string]interface{} `yaml:"before,omitempty"`
AfterExec map[string]interface{} `yaml:"after,omitempty"`
}
func main() {
runners := &FTD{}
// parse mta yaml
err := yaml.Unmarshal(runContent, runners)
if err != nil {
log.Fatalf("Error : %v", err)
}
for k, v := range runners.BuildParams.BeforeExec {
fmt.Println(k, v.(interface{}))
}
}
This is working example
https://play.golang.org/p/qTqUJy3md0c
also I've tried with
this is working
run := runners.BuildParams.BeforeExec["run-parameters"].(map[interface{}]interface{})["run"]
fmt.Println("run: ", run)
what I've tried is this which works but what happens if the run value is empty or no entry at all,this will cause a dump how can I overcome this ?
what I've tried is this which works but what happens if the run value is empty or no entry at all,this will cause a dump how can I overcome this ?
You can do
runParams, ok := runners.BuildParams.BeforeExec["run-parameters"]
if !ok {
// handle lack of "run-parameters" in BeforeExec
}
runParamsMap, ok := runParams.(map[interface{}]interface{})
if !ok {
// handle "run-parameters" not being a map
}
run, ok := runParamsMap["run"]
if !ok {
// handle lack of "run" inside "run-parameters"
}
runStr, ok := run.(string)
if !ok {
// handle "run" not being a string
}
fmt.Println("run: ", runStr)
This is quite verbose so you could use something like https://github.com/jmoiron/jsonq, where you can specify a "path" to the desired value nested inside several levels of maps. Despite the "json" in the name, this library works with map[string]interface{} and not json files. But note that the library you use for yaml unmarshalling results in map[interface{}]interface{} instead of map[string]interface{} and you will have to use a different one in order for it to work with jsonq.
run, err := jsonq.NewQuery(runners.BuildParams.BeforeExec).String("run-parameters", "run")
if err != nil {
// handle all possible errors in one place
}
fmt.Println("run: ", run)

Read config in a custom format from yaml file

I have a configuration file in YAML format. I am trying to read the configuration in some custom format. I couldn't guess any pattern that I can go for like tree, json etc.
Eg. application.yaml
organization:
products:
product1:
manager: "Rob"
engineer: "John"
product2:
manager: "Henry"
lead: "patrick"
The configuration file can have huge information and that can vary from file to file. I want to construct data in the following format,
organization/products/product1/manager = Rob
organization/products/product1/engineer = John
organization/products/product2/lead = patrick
OR
{"organization/products/product1/manager":"Rob","organization/products/product2/lead":"patrick"}
Any idea how I can achieve this pattern?
This is essentially an exercise in printing trees. The exact implementation will depend on the particular YAML parser you pick, but pretty much all of them will have some kind of "map of anything" type. In the very popular gopkg.in/yaml.v2 this type is named MapSlice (don't let the name confuse you; it leaks its implementation which has to deal with flexible key types).
Just throw it at your favorite tree traversal algorithm to render the text file. Here is a simple example that works with only string keys and only some scalar leaf nodes:
package main
import (
"bytes"
"fmt"
"io"
"log"
"path/filepath"
)
func main() {
var tree yaml.MapSlice
if err := yaml.Unmarshal(input, &tree); err != nil {
log.Fatal(err)
}
var buf bytes.Buffer
if err := render(&buf, tree, ""); err != nil {
log.Fatal(err)
}
}
func render(w io.Writer, tree yaml.MapSlice, prefix string) error {
for _, branch := range tree {
key, ok := branch.Key.(string)
if !ok {
return fmt.Errorf("unsupported key type: %T", branch.Key)
}
prefix := filepath.Join(prefix, key)
switch x := branch.Value.(type) {
default:
return fmt.Errorf("unsupported value type: %T", branch.Value)
case yaml.MapSlice:
// recurse
if err := render(w, x, prefix); err != nil {
return err
}
continue
// scalar values
case string:
case int:
case float64:
// ...
}
// print scalar
if _, err := fmt.Fprintf(w, "%s = %v\n", prefix, branch.Value); err != nil {
return err
}
}
return nil
}

Unmarshal hcl to struct using viper

Trying to Unmarshal a hcl config file to a struct, using viper, this error is returned: 1 error(s) decoding:\n\n* 'NATS' expected a map, got 'slice'. What is missing?
The code:
func lab() {
var c conf
// config file
viper.SetConfigName("draft")
viper.AddConfigPath(".")
viper.SetConfigType("hcl")
if err := viper.ReadInConfig(); err != nil {
log.Error(err)
return
}
log.Info(viper.Get("NATS")) // gives [map[port:10041 username:cl1 password:__Psw__4433__ http_port:10044]]
if err := viper.Unmarshal(&c); err != nil {
log.Error(err)
return
}
log.Infow("got conf", "conf", c)
}
type conf struct {
NATS struct {
HTTPPort int
Port int
Username string
Password string
}
}
And the config file (draft.hcl inside current directory):
NATS {
HTTPPort = 10044
Port = 10041
Username = "cl1"
Password = "__Psw__4433__"
}
Edit
Have checked this struct with hcl package and it gets marshaled/unmarshalled correctly. Also this works correctly with yaml and viper.
There is a difference between these two where log.Info(viper.Get("NATS")) is called. While the hcl version returns a slice of maps, the yaml version returns a map: map[password:__psw__4433__ httpport:10044 port:10041 username:cl1].
Your conf struct is not matching the HCL. When converted to json the HCL looks like below
{
"NATS": [
{
"HTTPPort": 10044,
"Password": "__Psw__4433__",
"Port": 10041,
"Username": "cl1"
}
]
}
So the Conf Struct should look like this
type Conf struct {
NATS []struct{
HTTPPort int
Port int
Username string
Password string
}
}
Modified code
package main
import (
"log"
"github.com/spf13/viper"
"fmt"
)
type Conf struct {
NATS []struct{
HTTPPort int
Port int
Username string
Password string
}
}
func main() {
var c Conf
// config file
viper.SetConfigName("draft")
viper.AddConfigPath(".")
viper.SetConfigType("hcl")
if err := viper.ReadInConfig(); err != nil {
log.Fatal(err)
}
fmt.Println(viper.Get("NATS")) // gives [map[port:10041 username:cl1 password:__Psw__4433__ http_port:10044]]
if err := viper.Unmarshal(&c); err != nil {
log.Fatal(err)
}
fmt.Println(c.NATS[0].Username)
}
I know this question is more than two years old now, but I came across the same issue recently.
I'm using viper to be able to load different configuration files into a Go struct, allowing configuration in JSON, YAML, TOML, HCL, just pick your favourite :)
HCL file format does wrap a map into a slice because it allows redefining a section like:
section = {
key1 = "value"
}
section = {
key2 = "value"
}
which is something that is not supported by the other formats.
And here's how I fixed it:
My solution implies each new block will override any previous definition of the same key, and keep all the others. You can do some merging magic but I didn't need to.
You need to make a hook to convert a slice of maps into a map:
// sliceOfMapsToMapHookFunc merges a slice of maps to a map
func sliceOfMapsToMapHookFunc() mapstructure.DecodeHookFunc {
return func(from reflect.Type, to reflect.Type, data interface{}) (interface{}, error) {
if from.Kind() == reflect.Slice && from.Elem().Kind() == reflect.Map && (to.Kind() == reflect.Struct || to.Kind() == reflect.Map) {
source, ok := data.([]map[string]interface{})
if !ok {
return data, nil
}
if len(source) == 0 {
return data, nil
}
if len(source) == 1 {
return source[0], nil
}
// flatten the slice into one map
convert := make(map[string]interface{})
for _, mapItem := range source {
for key, value := range mapItem {
convert[key] = value
}
}
return convert, nil
}
return data, nil
}
}
then you need to create a DecodeHook:
configOption := viper.DecodeHook(mapstructure.ComposeDecodeHookFunc(
sliceOfMapsToMapHookFunc(),
mapstructure.StringToTimeDurationHookFunc(),
mapstructure.StringToSliceHookFunc(","),
))
the two other hooks are the default ones so you might want to keep them
then you pass the option to the Unmarshal method
viper.Unmarshal(&c, configOption)
With this method you don't need a slice around your structs or your maps. Also that makes it compatible with the other configuration file formats

Delete objects in s3 using wildcard matching

I have the following working code to delete an object from Amazon s3
params := &s3.DeleteObjectInput{
Bucket: aws.String("Bucketname"),
Key : aws.String("ObjectKey"),
}
s3Conn.DeleteObjects(params)
But what i want to do is to delete all files under a folder using wildcard **. I know amazon s3 doesn't treat "x/y/file.jpg" as a folder y inside x but what i want to achieve is by mentioning "x/y*" delete all the subsequent objects having the same prefix. Tried amazon multi object delete
params := &s3.DeleteObjectsInput{
Bucket: aws.String("BucketName"),
Delete: &s3.Delete{
Objects: []*s3.ObjectIdentifier {
{
Key : aws.String("x/y/.*"),
},
},
},
}
result , err := s3Conn.DeleteObjects(params)
I know in php it can be done easily by s3->delete_all_objects as per this answer. Is the same action possible in GOlang.
Unfortunately the goamz package doesn't have a method similar to the PHP library's delete_all_objects.
However, the source code for the PHP delete_all_objects is available here (toggle source view): http://docs.aws.amazon.com/AWSSDKforPHP/latest/#m=AmazonS3/delete_all_objects
Here are the important lines of code:
public function delete_all_objects($bucket, $pcre = self::PCRE_ALL)
{
// Collect all matches
$list = $this->get_object_list($bucket, array('pcre' => $pcre));
// As long as we have at least one match...
if (count($list) > 0)
{
$objects = array();
foreach ($list as $object)
{
$objects[] = array('key' => $object);
}
$batch = new CFBatchRequest();
$batch->use_credentials($this->credentials);
foreach (array_chunk($objects, 1000) as $object_set)
{
$this->batch($batch)->delete_objects($bucket, array(
'objects' => $object_set
));
}
$responses = $this->batch($batch)->send();
As you can see, the PHP code will actually make an HTTP request on the bucket to first get all files matching PCRE_ALL, which is defined elsewhere as const PCRE_ALL = '/.*/i';.
You can only delete 1000 files at once, so delete_all_objects then creates a batch function to delete 1000 files at a time.
You have to create the same functionality in your go program as the goamz package doesn't support this yet. Luckily it should only be a few lines of code, and you have a guide from the PHP library.
It might be worth submitting a pull request for the goamz package once you're done!
Using the mc tool you can do:
mc rm -r --force https://BucketName.s3.amazonaws.com/x/y
it will delete all the objects with the prefix "x/y"
You can achieve the same with Go using minio-go like this:
package main
import (
"log"
"github.com/minio/minio-go"
)
func main() {
config := minio.Config{
AccessKeyID: "YOUR-ACCESS-KEY-HERE",
SecretAccessKey: "YOUR-PASSWORD-HERE",
Endpoint: "https://s3.amazonaws.com",
}
// find Your S3 endpoint here http://docs.aws.amazon.com/general/latest/gr/rande.html
s3Client, err := minio.New(config)
if err != nil {
log.Fatalln(err)
}
isRecursive := true
for object := range s3Client.ListObjects("BucketName", "x/y", isRecursive) {
if object.Err != nil {
log.Fatalln(object.Err)
}
err := s3Client.RemoveObject("BucketName", object.Key)
if err != nil {
log.Fatalln(err)
continue
}
log.Println("Removed : " + object.Key)
}
}
Since this question was asked, the AWS GoLang lib for S3 has received some new methods in S3 Manager to handle this task (in response to #Itachi's pr).
See Github record: https://github.com/aws/aws-sdk-go/issues/448#issuecomment-309078450
Here is their example in v1: https://github.com/awsdocs/aws-doc-sdk-examples/blob/main/go/s3/DeleteObjects/DeleteObjects.go#L36
To get "wildcard matching" on paths inside the bucket, add the Prefix param to the example's ListObjectsInput call, as shown here:
iter := s3manager.NewDeleteListIterator(svc, &s3.ListObjectsInput{
Bucket: bucket,
Prefix: aws.String("somePathString"),
})
A bit late in the game, but since I was having the same problem, I created a small pkg that you can copy to your code base and import as needed.
func ListKeysInPrefix(s s3iface.S3API, bucket, prefix string) ([]string, error) {
res, err := s.Client.ListObjectsV2(&s3.ListObjectsV2Input{
Bucket: aws.String(bucket),
Prefix: aws.String(prefix),
})
if err != nil {
return []string{}, err
}
var keys []string
for _, key := range res.Contents {
keys = append(keys, *key.Key)
}
return keys, nil
}
func createDeleteObjectsInput(keys []string) *s3.Delete {
rm := []*s3.ObjectIdentifier{}
for _, key := range keys {
rm = append(rm, &s3.ObjectIdentifier{Key: aws.String(key)})
}
return &s3.Delete{Objects: rm, Quiet: aws.Bool(false)}
}
func DeletePrefix(s s3iface.S3API, bucket, prefix string) error {
keys, err := s.ListKeysInPrefix(bucket, prefix)
if err != nil {
panic(err)
}
_, err = s.Client.DeleteObjects(&s3.DeleteObjectsInput{
Bucket: aws.String(bucket),
Delete: s.createDeleteObjectsInput(keys),
})
if err != nil {
return err
}
return nil
}
So, in the case you have a bucket called "somebucket" with the following structure: s3://somebucket/foo/some-prefixed-folder/bar/test.txt and wanted to delete from some-prefixed-folder onwards, usage would be:
func main() {
// create your s3 client here
// client := ....
err := DeletePrefix(client, "somebucket", "some-prefixed-folder")
if err != nil {
panic(err)
}
}
This implementation only allows to delete a maximum of 1000 entries from the given prefix due ListObjectsV2 implementation - but it is paginated, so it's a matter of adding the functionality to keep refreshing results until results are < 1000.

Resources