I have a configuration file that looks like the following:
apps:
customer1:
upload_path: "/opt/uploads/customer1"
local_path: "/opt/ready/customer1"
bucket: "b1"
customer2:
upload_path: /opt/uploads/customer2
local_path: opt/ready/customer2,
bucket: "b2"
I am using Viper to load and read the configuration file.
I am unmarshalling the above config and mapping it to the following struct:
type AppConfig struct {
UploadPath string `mapstructure:"upload_path"`
LocalPath string `mapstructure:"local_path"`
Bucket string `mapstructure:"bucket"`
}
appconfigs []*AppConfig
viper.SetConfigName(configName)
viper.SetConfigType("yaml")
viper.AddConfigPath(".")
viper.UnmarshalKey("apps", &appconfigs)
The problem I am trying to solve is getting the entry header (ie customer1 and customer2) without having to have a redundant field in my config file and ending up with:
apps:
customer1:
name: customer1
upload_path: "/opt/uploads/customer1"
local_path: "/opt/ready/customer1"
bucket: "b1"
The configuration can be unmarshaled to a map:
var appconfigs map[string]*AppConfig
viper.UnmarshalKey("apps", &appconfigs)
You can get the names from the map key.
Related
In a golang CLI I'm programming I collect information on how to configure the tool and I marhsal that as a YAML file. However, I'm not sure how I would add line breaks to make the file more readable?
type Config struct {
Author string `yaml:"author"`
License string `yaml:"license"`
// Marhsal a line break here
Workspace string `yaml:"workspace"`
Description string `yaml:"description"`
// Marhsal a line break here
Target string `yaml:"target"`
}
One way to implement this that allows format (and comments) is to use a template engine.
Here is a running example that generates a string with the formatted yaml, that can be then saved to a .yml file.
No additional libraries are needed and the template is included inside the go file.
package main
import (
"bytes"
"fmt"
"text/template"
)
type Config struct {
Author string
License string
Workspace string
Description string
Target string
}
const cfg_template = `
conf:
author: {{ .Author }}
licence: {{ .License }}
workspace: {{ .Workspace }}
description: {{ .Description }}
# you can even add comments to the template
target: {{ .Target }}
# other hardcoded config
foo: bar
`
func generate(config *Config) string {
t, err := template.New("my yaml generator").Parse(cfg_template)
if err != nil {
panic(err)
}
buf := &bytes.Buffer{}
err = t.Execute(buf, config)
if err != nil {
panic(err)
}
return buf.String()
}
func main() {
c := Config{
Author: "Germanio",
License: "MIT",
Workspace: "/home/germanio/workspace",
Description: "a cool description",
Target: "/home/germanio/target",
}
yaml := generate(&c)
fmt.Printf("yaml:\n%s", yaml)
}
The result looks like this:
$ go run yaml_generator.go
yaml:
conf:
author: Germanio
licence: MIT
workspace: /home/germanio/workspace
description: a cool description
# you can even add comments to the template
target: /home/germanio/target
# other hardcoded config
foo: bar
I'm sure there are better ways to implement it, just want to show a quick working example.
As empty line don't have a meaning in yaml, the default library does not create them, and does not expose an option to do so in the struct field tag.
However, if you want fine grained control of how a type is marshalled in yaml, you can always make it implements yaml.Marshaller by defining a method MarshalYAML() (interface{}, error)
I am trying to read a config in a json file using the viper library
//config.json
{
"currency": {
"btc": [{
"api_endpoint": "api.blockcypher.com/v1/btc/main/addrs/$address/balance",
"balance_threshold": 234234.34,
"wallet_address": "0xsdrsdf",
"alerts":{
"slack": "put slack config here",
"sms": "put sms config here"
},
"enable_monitoring": true
}],
"eth": [{
"api_endpoint": "some endpoint",
"balance_threshold" : 2234234.234,
"wallet_address": "0xsdrsdf",
"alerts":{
"slack": "put slack config here",
"sms": "put sms config here"
},
"enable_monitoring": true
}]
}
}
and config.go like this
func GetConfig() {
viper.SetConfigName("config") // name of config file (without extension)
viper.SetConfigType("json") // REQUIRED if the config file does not have the extension in the name
viper.AddConfigPath(".") // path to look for the config file in
viper.AddConfigPath("$HOME/") // path to look for the config file in
err := viper.ReadInConfig() // Find and read the config file
if err != nil { // Handle errors reading the config file
panic(fmt.Errorf("Fatal error config file: %s", err))
}
var config config
viper.Unmarshal(&config)
fmt.Println(viper.Get("currency"))
fmt.Printf("Config: %v\n", config["currency"])
}
type coreconfig struct {
APIEndpoint string `json:"api_endpoint"`
BalanceThreshold float64 `json:"balance_threshold"`
WalletAddress string `json:"wallet_address"`
EnableMonitoring bool `json:"enable_monitoring"`
Alerts map[string]alerts
}
type alerts struct {
Sms string `json:"sms"`
Slack string `json:"slack"`
}
type currencyconfig map[string][]coreconfig
type config map[string]currencyconfig
The output is printed as empty config
[map[btc:[map[alerts:[map[slack:put slack config here sms:put sms config here]] api_endpoint:api.blockcypher.com/v1/btc/main/addrs/$address/balance balance_threshold:234234 enable_monitoring:true wallet_address:0xsdrsdf]] eth:[map[alerts:[map[slack:put slack config here sms:put sms config here]] api_endpoint:some endpoint balance_threshold:2.234234e+06 enable_monitoring:true wallet_address:0xsdrsdf]]]] <= This shows that config file is read correctly
Config: map[] <= actual output
Use map[string]string or alerts only in coreconfig struct as your json stucture.
The json tag is used by the json package, not by viper. Viper use github.com/mitchellh/mapstructure package for unmarshaling, so use mapstructure tag.
type coreconfig struct {
APIEndpoint string `mapstructure:"api_endpoint"`
BalanceThreshold float64 `mapstructure:"balance_threshold"`
WalletAddress string `mapstructure:"wallet_address"`
EnableMonitoring bool `mapstructure:"enable_monitoring"`
Alerts alerts
}
We are using Viper to read and parse our config file and all of that works without any issues.
However we are not able to override some of our config values using env variables. These are specific use cases where the config is bound to a struct or an array of structs.
Here is an example from our config.yaml:
app:
verifiers:
- name: "test1"
url: "http://test1.url"
cache: "5000ms"
- name: "test2"
url: "http://test2.url"
cache: "10000ms"
Which is bound to the following structs (golang):
type App struct {
AppConfig Config `yaml:"app" mapstructure:"app"`
}
type Config struct {
Verifiers []VerifierConfig `json:"verifiers" yaml:"verifiers" mapstructure:"verifiers"`
}
type VerifierConfig struct {
Name string `json:"name" yaml:"name" mapstructure:"name"`
URL string `json:"url,omitempty" yaml:"url,omitempty" mapstructure:"url"`
cache jsontime.Duration `json:"cache" yaml:"cache" mapstructure:"cache"`
}
We are unable to override the value of verifiers using env variables.
Here are the Viper options we have used:
viper.AutomaticEnv()
viper.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))
Has anyone experienced a similar issue or can confirm that Viper does not support such a use case?
Any pointers would be greatly appreciated.
Thanks
viper cannot read env variables inside config file. You have to replace them in .go file where value is being used. Here is an example on how I achieved:
Ask Viper to consider system environment variable with "ENV" prefix, that means viper expects "ENV_MONGO_USER" and "ENV_MONGO_PASSWORD" as system environment variables
viper.AutomaticEnv()
viper.SetEnvPrefix(viper.GetString("ENV"))
viper.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))
In yml file:
mongodb:
url: "mongodb+srv://${username}:${password}#xxxxxxx.mongodb.net/"
database: "test"
In mongodb connection file: (before making db connection, replace ${username} and ${password} with respective environment variables but viper will read those variables without prefix )
const BasePath = "mongodb"
mongoDbUrl := viper.GetString(fmt.Sprintf("%s.%s", BasePath, "url"))
username := viper.GetString("MONGO_USER")
password := viper.GetString("MONGO_PASSWORD")
replacer := strings.NewReplacer("${username}", username, "${password}", password)
mongoDbUrl = replacer.Replace(mongoDbUrl)
I am trying to integrate Terraform go-SDK with my go-code where I already have an terraform template file and I need to create the infrastructure by importing that file to the go-SDK. I am not able to find any related documents. I tried with the godoc for terraform. With a little understanding, I tried to implement a simple terraform template. My terraform template file is,
provider "aws" {
access_key = "xxxx"
secret_key = "xxxx"
region = "xxxx"
}
resource "aws_instance" "example" {
ami = "ami-xxxx"
instance_type = "t2.micro"
# The name of our SSH keypair.
key_name = "xxxx"
# Security groupID
vpc_security_group_ids = ["sg-xxxx"]
#Subnet ID
subnet_id = "subnet-xxxx"
}
I tried via commandLine and I am able to bring up an instance with this file.
I tried the same via my go-code. My understanding is that I need to create a Context in order to apply it. For creating a Context, I need a module.Tree. So I tried creating a module.Tree with config.Config. I imported the above terraform template file to my go-code to create a config and a tree. My go-code looks like,
package main
import (
"github.com/hashicorp/terraform/config"
"github.com/hashicorp/terraform/config/module"
"fmt"
)
func main() {
con, err := config.LoadFile(path to .tf file)
fmt.Println("Config: ", con)
fmt.Println("Error: ", err)
fmt.Println("Validate: ", con.Validate())
tree := module.NewTree("testTree", con)
fmt.Println("Tree: ", tree.String())
}
When I execute my code, I am not getting any errors for loading the config from file, or validating the same. But I am getting the output as "tree not loaded",
Output:
Config: &{ <nil> <nil> [] [0xc4201cd0c0] [0xc4201f03c0] [] [] [] []}
Error: <nil>
Validate: <nil>
Tree: testTree not loaded
Can someone help me out to implement this? Thanks in advance.
I have a Go struct for which I want to generate an OpenAPI schema automatically. Once I have an OpenAPI definition of that struct I wanna generate JSONSchema of it, so that I can validate the input data that comes and is gonna be parsed into those structs.
The struct looks like the following:
// mySpec: io.myapp.MinimalPod
type MinimalPod struct {
Name string `json:"name"`
// k8s: io.k8s.kubernetes.pkg.api.v1.PodSpec
v1.PodSpec
}
Above struct is clearly an augmentation of what Kubernetes PodSpec is.
Now the approach that I have used is to generate definition part for my struct MinimalPod, the definition for PodSpec will come from upstream OpenAPI spec of Kubernetes. PodSpec has a key io.k8s.kubernetes.pkg.api.v1.PodSpec in the upstream OpenAPI spec, this definition is injected from there in my Properties. Now in my code that parses above struct I have templates of what to do if struct field is string.
If the field has a comment that starts with k8s: ... the next part is Kubernetes object's OpenAPI definition key. In our case the OpenAPI definition key is io.k8s.kubernetes.pkg.api.v1.PodSpec. So I retrieve that field's definition from the upstream OpenAPI definition and embed it into the definition of my struct.
Once I have generated an OpenAPI definition for this struct which is injected in Kubernetes OpenAPI schema's definition with key being io.myapp.MinimalPod. Now I can use the tool openapi2jsonschema to generate JSONSchema out of this one. Which generates a JSONSchema file named MinimalPod.json.
Now jsonschema tool and the file MinimalPod.json can be used for validating input given to my tool parser to see if all fields were given right.
Is this the right approach of doing things, or is there a tool/library and if I feed Go structs to it, it gives me OpenAPI schema? It would be fine if it does not identify where to inject Kubernetes OpenAPI schema from even automatic parsing of Go structs and giving OpenAPI definition would be much appreciated.
Update 1
After following #mehdy 's instructions, this is what I have tried:
I have used this import path github.com/kedgeproject/kedge/vendor/k8s.io/client-go/pkg/api/v1 to import the PodSpec definition instead of k8s.io/api/core/v1 and code looks like this:
package foomodel
import "github.com/kedgeproject/kedge/vendor/k8s.io/client-go/pkg/api/v1"
// MinimalPod is a minimal pod.
// +k8s:openapi-gen=true
type MinimalPod struct {
Name string `json:"name"`
v1.PodSpec
}
Now when I generate the same with flag -i changed from k8s.io/api/core/v1 to github.com/kedgeproject/kedge/vendor/k8s.io/client-go/pkg/api/v1
$ go run example/openapi-gen/main.go -i k8s.io/kube-openapi/example/model,github.com/kedgeproject/kedge/vendor/k8s.io/client-go/pkg/api/v1 -h example/foomodel/header.txt -p k8s.io/kube-openapi/example/foomodel
This is what is generated:
$ cat openapi_generated.go
// +build !ignore_autogenerated
/*
======
Some random text
======
*/
// This file was autogenerated by openapi-gen. Do not edit it manually!
package foomodel
import (
spec "github.com/go-openapi/spec"
common "k8s.io/kube-openapi/pkg/common"
)
func GetOpenAPIDefinitions(ref common.ReferenceCallback) map[string]common.OpenAPIDefinition {
return map[string]common.OpenAPIDefinition{
"k8s.io/kube-openapi/example/model.Container": {
Schema: spec.Schema{
SchemaProps: spec.SchemaProps{
Description: "Container defines a single application container that you want to run within a pod.",
Properties: map[string]spec.Schema{
"health": {
SchemaProps: spec.SchemaProps{
Description: "One common definitions for 'livenessProbe' and 'readinessProbe' this allows to have only one place to define both probes (if they are the same) Periodic probe of container liveness and readiness. Container will be restarted if the probe fails. Cannot be updated. More info: https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle#container-probes",
Ref: ref("k8s.io/client-go/pkg/api/v1.Probe"),
},
},
"Container": {
SchemaProps: spec.SchemaProps{
Ref: ref("k8s.io/client-go/pkg/api/v1.Container"),
},
},
},
Required: []string{"Container"},
},
},
Dependencies: []string{
"k8s.io/client-go/pkg/api/v1.Container", "k8s.io/client-go/pkg/api/v1.Probe"},
},
}
}
I get only this much of the configuration generated. While when I switch back to "k8s.io/api/core/v1" I get config code auto generated which is more than 8k lines. What am I missing here?
Here definition of k8s.io/client-go/pkg/api/v1.Container and k8s.io/client-go/pkg/api/v1.Probe is missing while when I use k8s.io/api/core/v1 as import everything is generated.
Note: To generate above steps, please git clone https://github.com/kedgeproject/kedge in GOPATH.
You can use kube-openapi package for this. I am going to add a sample to the repo but I've tested this simple model:
// Car is a simple car model.
// +k8s:openapi-gen=true
type Car struct {
Color string
Capacity int
// +k8s:openapi-gen=false
HiddenFeature string
}
If you assume you created this file in
go run example/openapi-gen/main.go -h example/model/header.txt -i k8s.io/kube-openapi/example/model -p k8s.io/kube-openapi/example/model
(you also need to add a header.txt file). You should see a new file created in example/model folder called openapi_generated.go. This is an intermediate generated file that has your OpenAPI model in it:
func GetOpenAPIDefinitions(ref common.ReferenceCallback) map[string]common.OpenAPIDefinition {
return map[string]common.OpenAPIDefinition{
"k8s.io/kube-openapi/example/model.Car": {
Schema: spec.Schema{
SchemaProps: spec.SchemaProps{
Description: "Car is a simple car model.",
Properties: map[string]spec.Schema{
"Color": {
SchemaProps: spec.SchemaProps{
Type: []string{"string"},
Format: "",
},
},
"Capacity": {
SchemaProps: spec.SchemaProps{
Type: []string{"integer"},
Format: "int32",
},
},
},
Required: []string{"Color", "Capacity"},
},
},
Dependencies: []string{},
},
}
}
From there you should be able to call the generated method, get the model for your Type and get its Schema.
With some go get magic and changing the command line a little, I was able to generate the model for your model. Here is what you should change in your code:
package model
import "k8s.io/api/core/v1"
// MinimalPod is a minimal pod.
// +k8s:openapi-gen=true
type MinimalPod struct {
Name string `json:"name"`
v1.PodSpec
}
and then change the run command a little to include PodSpec in the generation:
go run example/openapi-gen/main.go -h example/model/header.txt -i k8s.io/kube-openapi/example/model,k8s.io/api/core/v1 -p k8s.io/kube-openapi/example/model
Here is what I got: https://gist.github.com/mbohlool/e399ac2458d12e48cc13081289efc55a