We are using Viper to read and parse our config file and all of that works without any issues.
However we are not able to override some of our config values using env variables. These are specific use cases where the config is bound to a struct or an array of structs.
Here is an example from our config.yaml:
app:
verifiers:
- name: "test1"
url: "http://test1.url"
cache: "5000ms"
- name: "test2"
url: "http://test2.url"
cache: "10000ms"
Which is bound to the following structs (golang):
type App struct {
AppConfig Config `yaml:"app" mapstructure:"app"`
}
type Config struct {
Verifiers []VerifierConfig `json:"verifiers" yaml:"verifiers" mapstructure:"verifiers"`
}
type VerifierConfig struct {
Name string `json:"name" yaml:"name" mapstructure:"name"`
URL string `json:"url,omitempty" yaml:"url,omitempty" mapstructure:"url"`
cache jsontime.Duration `json:"cache" yaml:"cache" mapstructure:"cache"`
}
We are unable to override the value of verifiers using env variables.
Here are the Viper options we have used:
viper.AutomaticEnv()
viper.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))
Has anyone experienced a similar issue or can confirm that Viper does not support such a use case?
Any pointers would be greatly appreciated.
Thanks
viper cannot read env variables inside config file. You have to replace them in .go file where value is being used. Here is an example on how I achieved:
Ask Viper to consider system environment variable with "ENV" prefix, that means viper expects "ENV_MONGO_USER" and "ENV_MONGO_PASSWORD" as system environment variables
viper.AutomaticEnv()
viper.SetEnvPrefix(viper.GetString("ENV"))
viper.SetEnvKeyReplacer(strings.NewReplacer(".", "_"))
In yml file:
mongodb:
url: "mongodb+srv://${username}:${password}#xxxxxxx.mongodb.net/"
database: "test"
In mongodb connection file: (before making db connection, replace ${username} and ${password} with respective environment variables but viper will read those variables without prefix )
const BasePath = "mongodb"
mongoDbUrl := viper.GetString(fmt.Sprintf("%s.%s", BasePath, "url"))
username := viper.GetString("MONGO_USER")
password := viper.GetString("MONGO_PASSWORD")
replacer := strings.NewReplacer("${username}", username, "${password}", password)
mongoDbUrl = replacer.Replace(mongoDbUrl)
Related
I have a configuration file that looks like the following:
apps:
customer1:
upload_path: "/opt/uploads/customer1"
local_path: "/opt/ready/customer1"
bucket: "b1"
customer2:
upload_path: /opt/uploads/customer2
local_path: opt/ready/customer2,
bucket: "b2"
I am using Viper to load and read the configuration file.
I am unmarshalling the above config and mapping it to the following struct:
type AppConfig struct {
UploadPath string `mapstructure:"upload_path"`
LocalPath string `mapstructure:"local_path"`
Bucket string `mapstructure:"bucket"`
}
appconfigs []*AppConfig
viper.SetConfigName(configName)
viper.SetConfigType("yaml")
viper.AddConfigPath(".")
viper.UnmarshalKey("apps", &appconfigs)
The problem I am trying to solve is getting the entry header (ie customer1 and customer2) without having to have a redundant field in my config file and ending up with:
apps:
customer1:
name: customer1
upload_path: "/opt/uploads/customer1"
local_path: "/opt/ready/customer1"
bucket: "b1"
The configuration can be unmarshaled to a map:
var appconfigs map[string]*AppConfig
viper.UnmarshalKey("apps", &appconfigs)
You can get the names from the map key.
I have a function that works locally and I have a config file that loads into the app using Viper, I also have viper.AutomaticEnv() set.
After deploying to AWS Lambda, seems like env vars are ignored. I went over to the Viper issues page and found this: https://github.com/spf13/viper/issues/584
Looks like Viper requires a config file to load or it will just stop working even though we can set env vars.
How do you handle local dev vs deployment for lambda secrets in Go?
I would like to avoid AWS Secrets Manager if possible
There are a lot of options how to handle secrets in AWS Lambdas. I'd recommend to not use Viper or any of those tools. Building a Lambda that reads configuration from environment Lambdas is simple.
That said, I would also recommend reading secrets from AWS SSM parameter store.
main.go
func (h handler) handleRequest() error {
fmt.Printf("My secret: %s", h.config.secret)
return nil
}
type configuration struct {
secret string
}
type handler struct {
config configuration
}
func newConfig() (configuration, error) {
secret, ok := os.LookupEnv("SECRET")
if !ok {
return configuration{}, errors.New("can not read environment variable 'SECRET'")
}
return configuration{
secret: secret
}, nil
}
func main() {
cfg, err := newConfig()
if err != nil {
fmt.Printf("unable to create configuration: %v\n", err)
os.Exit(1)
}
h := handler{
config: cfg,
}
lambda.Start(h.handleRequest)
}
No need to use Viper and increase your binaries size unnecessarily. Remember: larger binary, longer cold-start time.
How do you handle local dev vs deployment for lambda secrets in Go?
Usually, we only use unit tests locally that use mocked services that do not require secrets. Most of the "integration" testing is done in AWS. Every developer has their own "environment" that they can deploy. To manage this we use Terraform.
If you really need to do test something locally, I'd recommend to create a test file that you do "gitignore" to avoid committing it. In this file I just hard-code the secret.
So for example, you can have a playground_test.go which you ignore in your .gitignore file.
I am trying to read an environment variable is being set from within Gitlab configuration when the application is being built, I am doing this for achieving that purpose:
I setup a variable in the application.properties.yml of my spring boot app:
sf:
apiKey: $SF_API_KEY
In the .gitlab-ci.yml, I defined the variable to be set, as follows:
variables:
SF_API_KEY: $SF_API_KEY
all I want, is to be able to read that variable from within one of my services, as the below code depicts:
#Service
class MyService(#Value("\${sf.apiKey}") val apiKey: String)
{
fun doSomething(){
//i am seeing the variable is being set by gitlab in the build logs but
// it is not being read here properly
var result = apiKey;
logger.info { "***check apiKey: $apiKey" }
//This line lgs $SF_API_KEY as a value of my variable, but not the
// real value
}
}
Am I doing something wrong? I would appreciate any help.
Try (note the {} around SF_API_KEY):
sf:
apiKey: ${SF_API_KEY}
Take a look at the docs where this placeholder notation is detailed.
I have deployed my app on Heroku and on the start of my app I check a config file and in there I want to access a config var I have created on Heroku API_KEY to define my Firebase config:
module.exports = {
fireConfig: {
apiKey: process.env.API_KEY,
authDomain: "my-app.firebaseapp.com",
databaseURL: "https://my-app.firebaseio.com",
projectId: "my-project-id",
storageBucket: "my-app.appspot.com",
messagingSenderId: "my-messaging-sender-id"
}
};
this process.env.API_KEY is undefined. Should I use a different way to access it?
You can define environment variables in your nuxt.config.js file, e.g.
export default {
env: {
firebaseApiKey: process.env.API_KEY || 'default value'
}
}
Here we use the API_KEY environment variable, if it's available, and assign that to firebaseApiKey.
If that environment variable isn't set, e.g. maybe in development (you can set it there too if you want), we fall back to 'default value'. Please don't put your real API key in 'default value'. You could use a separate throwaway Firebase account key here or just omit it (take || 'default value' right out) and rely on the environment variable being set.
These will be processed at build time and then made available using the name you give them, e.g.
module.exports = {
fireConfig: {
apiKey: process.env.firebaseApiKey, # Not API_KEY
// ...
};
I want to hide my secret credential from my yaml, i need to use .env, so how to call .env file from my yaml, so that every I call this YAML, YAML will automatic call .env file. Please help me. thx
Instead of using an .env file, which is a simple properties file if you're following dotenv package, you can do the following:
create additional .yml file, for example .secrets.yml. you can store the secrets per stage:
prod:
MY_SECRET: foo
dev:
MY_SECRET: bar
store your secrets/configurations there
Then in serverless.yml:
load this file into an object:
custom:
secrets: ${file(.secrets.yml):${self:provider.stage}}
load object fields as environment variables:
provider:
environment:
MY_SECRET: ${self:custom.secrets.MY_SECRET}
How to test locally
In your tests you can load the secrets file this way:
const yaml = require('js-yaml');
const fs = require('fs');
const _ = require('lodash');
module.exports.loadSecrets = function (env = 'dev', path = './.secrets.yml') {
const secrets = yaml.load(fs.readFileSync(path));
_.forEach(secrets[env], (value, key) => {
process.env[key] = value;
});
}
Reference: http://www.goingserverless.com/blog/using-environment-variables-with-the-serverless-framework