I have a project in Laravel 8 and I have some secret env parameters and I do not want to ship them with my application to github. I will deploy my application with github actions to AWS beanstalk. How do I keep all the secrets secure and put them to EC2 instance when all application deployed.
There are multiple ways to do that and you should not send your env file with your application to github.
You can use beanstalk's own parameter store page. However, if you do that another developer who has access to your AWS account can see all the env parameters. It is simple key value store page.
Benastalk Panel -> (Select Your Environment) -> Configuration -> Software
Under the systems manager there is a service called Parameter Store (this is my prefered way)
In here, You can add as much as parameter as you like securely. You can simply add string parameters as well as secure (like password or api keys) strings also integers but string and secure types are my favorites.
You can split all you parameters by path like "APP_NAME/DB_NAME" etc.
You should get all the parameters from Parameter Store to your EC2 instance and put them on newly created .env file.
There is github secrets in github actions and you can put all your secret parameters to github secrets page. You can get all the secrets from github secrets and put your secrets to your application and ship from github to AWS directly.
You can go to settings in your repository and see this page:
I am looking for tools/software for dynamically changing Linux conf/YAML files via API or tool like Consul.
If you have any experience on consul, please give feedback about creating templates for conf/YAML files, and without using service can it be done via consul?
Consul Template or Gomplate can be used to template configuration files based on changes in a backend data source.
https://learn.hashicorp.com/tutorials/consul/consul-template provides a basic example of a template which regenerates file contents when keys are added to Consul's key-value store.
I feel that I'm going around in circles here, so, please, bear with me. I want to deploy my Spring Boot application to App Engine but unlike the simple sample Google provides, mine, requires a database and that means credentials. I'm running Java 11 on Standard on Google App Engine.
I managed to make my app successfully connect by having this in the application.properties:
spring.datasource.url=jdbc:postgresql://google/recruiters_wtf?cloudSqlInstance=recruiters-wtf:europe-west2:recruiters-wtf&socketFactory=com.google.cloud.sql.postgres.SocketFactory&user=the_user&password=monkey123
The problem is that I don't want to commit any credentials to the repository, so, this is not acceptable. I could use an environment variable, but then I'll have to define them in the app.yaml file. I either keep a non-committed app.yaml file that is needed to deploy, which is cumbersome, or I commit it and I'm back at square one, committing credentials to the repository.
Since apparently Google App Engine cannot have environment variables defined in any other way (unlike Heroku), does this mean it's impossible to deploy a Spring Boot app to App Engine and have it connect to the database without using some unsafe/cumbersome practices? I feel I'm missing something here.
Based on my understanding of what you have described, you would like to essentially connect your Spring boot application running on Google App Engine to a database without exposing the sensitive information. If that is the case, I was able to find out that Cloud KMS offers users the ability for secret management. Specifically, applications which require small pieces of sensitive data at build or runtime are referred to as secrets. These secrets can be encrypted and decrypted with a symmetric key. In your case you can store the database credentials as secrets. You may find further details of the process for encrypting / decrypting a secret here.
They are currently three ways to manage secrets:
Storing secrets in code, encrypted with a key from Cloud KMS. This solution is implementing secrets at the application layer.
Storing secrets in a storage bucket in Cloud Storage, encrypted at rest. You can use Cloud Storage: Bucket to store your database
credentials and can also grant that bucket a specific Service Account.
This solution allows for separation of systems. In the case that the
code repository is breached, your secrets would themselves may still
be protected.
Using third-party secret management system.
In terms of storing secrets themselves, I found the follow steps outlined here quite useful for this. This guide walks users through setting up and storing secrets within a Cloud Storage bucket. The secret is encrypted at the application layer with an encryption key from Cloud KMS. Given your use case, this would be a great option as your secret would be stored within a bucket instead of your app.yaml file. Also, the secret being stored in a bucket would grant you the ability to restrict access to it with service account roles.
Essentially, your app will need to perform an API call to Google Cloud Storage in order to download the KMS encrypted file that contains the secret. It would then use the KMS generated key to decrypt the file so that it would be able to read out the password and use it to make a manual connection to the database. Adding these extra steps would be implementing more security layers, which is the entire idea noted in “'Note: Saving credentials in environment variables is convenient, but not secure - consider a more secure solution such as Cloud KMS to help keep secrets safe.'” in the Google example repository for Cloud SQL.
I hope this helps!
Assuming you are OK to use KMS or GCS to get the credentials, you can programatically set them in Spring Boot. See this post
Configure DataSource programmatically in Spring Boot
As you pointed out there's no built-in way to set environment variables in App Engine other than with the app.yaml file. I'm not an expert in Spring Boot but unless you can set/override some hook to initialize env var from Java code prior to application.properties evaluation, you'll need to set these at build time.
Option 1: Using Cloud Build
I know you're not really keen to use Cloud Build but that would be something like this.
First, following the instructions here, (after creating KeyRing and CryptoKey in KMS and grant access to Cloud Build service account) from you terminal encrypt your environment variable using KMS and get back its base64 representation:
echo -n $DB_PASSWORD | gcloud kms encrypt \
--plaintext-file=- \ # - reads from stdin
--ciphertext-file=- \ # - writes to stdout
--location=global \
--keyring=[KEYRING-NAME] \
--key=[KEY-NAME] | base64
Next, let's say you have an app.yaml file like this:
runtime: java11
instance_class: F1
env_variables:
USER: db_user
PASSWORD: db_passwd
create a cloudbuild.yaml file to define your build steps:
steps:
# replace env vars in app.yaml by their values from KMS
- name: 'gcr.io/cloud-builders/gcloud'
entrypoint: 'bash'
args: ['-c', 'sed -i "s/TEST/$$PASSWORD/g" src/main/appengine/app.yaml']
secretEnv: ['PASSWORD']
- name: 'gcr.io/cloud-builders/mvn'
args: ['clean']
- name: 'gcr.io/cloud-builders/mvn'
args: ['package']
- name: 'gcr.io/cloud-builders/mvn'
args: ['appengine:deploy']
timeout: '1600s'
secrets:
- kmsKeyName: projects/<PROJECT-ID>/locations/global/keyRings/<KEYRING_NAME>/cryptoKeys/<KEY_NAME>
secretEnv:
PASSWORD: <base64-encoded encrypted password>
timeout: '1600s'
You ould then deploy your app by running the following command:
gcloud builds submit .
The advantage of this method is that your local app.yaml file only contains placehoder values and can be safely committed. Or you can even set this build to trigger automatically every time you commit to a remote repository.
Option 2: Locally with a bash script
Instead of running mvn appengine:deploy to deploy your app, you could create a bash script that would replace the values in app.yaml, deploy the app, and remove the values straight away.. Something like:
#!/bin/bash
sed -i "s/db_passwd/$PASSWORD/g" src/main/appengine/app.yaml'
mvn appengine:deploy
sed -i "s/$PASSWORD/db_passwd/g" src/main/appengine/app.yaml'
and execute that bash script instead of running the maven command.
I would probably suggest the combination of Spring Cloud Config & Google Runtime Configuration API with your Spring Boot App.
Spring Cloud Config is component which is responsible for retrieving the configuration from remote locations and serving that configuration to your Spring Boot during initialization/boot up. The remote locations can be anything. e.g, GIT repository is widely used but for your use case, you can store the configuration in Google Runtime Configuration API.
So a sample flow will be like this.
Your Spring Boot App(with Config Client) --> Spring Cloud Config Server --> Google Runtime Configuration API
This requires for you to bring up a Spring Cloud Config Server as another App in GCP and have communications enabled from many of your Spring Boot Apps to a centralized Config Server which interacts with Google Runtime API.
Some documentation links.
https://cloud.spring.io/spring-cloud-config/reference/html/
https://docs.spring.io/spring-cloud-gcp/docs/1.1.0.M1/reference/html/_spring_cloud_config.html
https://cloud.google.com/deployment-manager/runtime-configurator/reference/rest/
Sample Spring Cloud GCP Config Example.
https://github.com/spring-cloud/spring-cloud-gcp/tree/master/spring-cloud-gcp-samples/spring-cloud-gcp-config-sample
You can try to encrypt the password in the application properties. Take a look at http://mbcoder.com/spring-boot-how-to-encrypt-properties-in-application-properties/
You should use GCP's Key Management Service: https://cloud.google.com/kms/
We use a few options:
1 - Without Docker
Can you use this approach via the env or console?
We use (Spring Boot defined) environment variables. This is the default way of doing this:
SPRING_DATASOURCE_USERNAME=myusername
SPRING_DATASOURCE_PASSWORD=mypassword
According to the Spring Boot spec this will overrule any value of application.properties variables. So you can specify default username and passwords for development and have this overruled at (test or) production deploy time.
Another way is documented in this post:
spring.datasource.url = ${OPENSHIFT_MYSQL_DB_HOST}:${OPENSHIFT_MYSQL_DB_PORT}/"nameofDB"
spring.datasource.username = ${OPENSHIFT_MYSQL_DB_USERNAME}
spring.datasource.password = ${OPENSHIFT_MYSQL_DB_PASSWORD}
2 - A Docker like approach via a console
Your question is described in this post. The default solution is working with 'secrets'. They are specifically made for this. You can convert any secret (as a file) to an environment variable in your process of building and deploying your application. This is a simple action that is described in many posts. Look for newer approaches.
I feel like it is easy to do. I used below properties in my spring boot app which connect to google mysql instance which is public one.
spring.datasource.url=jdbc:postgresql://IP:5432/yourdatabasename
spring.datasource.platform = postgres
spring.datasource.username = youruser
spring.datasource.password = yourpass
Rest will be taken care by the JpaRepository. I hope this would help you.
Let me know if any help is needed.
I'm running contract driven development tests using dredd.
I know how to configure dredd tests to run either against a local or remote server, given a blueprint apib file. Typically, the relevant fields in my dredd config file will read like
blueprint: myblueprintfile.apib
endpoint: localhost:3000 <or any remote server>
I didn't find a way to automatically refer to the remote blueprint hosted on apiary though. what i would like to achieve is something along the lines of
blueprint: <remote apiary apib file>
endpoint: localhost:3000 <or any remote server>
I can basically achieve the same result by manually fetching the blueprint using apiary CLI and saving it to a local file, before running the actual dredd tests
export APIARY_API_KEY=<key>
apiary fetch --api-name=<name>
Is there a way to achieve this step directly from dredd configuration file?
Notice:
i'm working with an authenticated apiary private account
i'm not worry about the endpoint field above, my problem is having the blueprint field pointing to a remote apiary source automatically
Maybe this question is a duplicated one, but i've looked at previously related questions and didn't find anything
While it's possible to point to a remotely stored .apib file, it will not work for a private documentation. At this point you can either:
Use the GitHub Sync to get the document on your machine
Automate fetching the document before testing with the Apiary CLI
Heroku provides its own database name and other credentials, but my local database name is different.How can I change the database name according to the database credentials provided by heroku during production?
Use a package like dotenv. dotenv and variants of it likely exist for whatever language you're using.
Basically, you want to use environment variables instead of hard coding values into your code. So, instead of writing something like this:
my_database_connect('my_username', 'abc123')
You'd write:
my_database_connect(process.env.DB_USERNAME, process.env.DB_PASSWORD)
Heroku will already have these environment variables set on the "config" tab of your app. Then for local development, you'll create a file called .env and have this text in it:
DB_USERNAME=my_username
DB_PASSWORD=abc123
Don't commit .env to your git repository – it should only live on your machine where you develop. Now your code will run locally as well as on Heroku, and connect to the proper database depending on the environment it's running in.
Here's an article that explains this more thoroughly for node.js, although this is basically the best practice for general development: https://medium.com/#rafaelvidaurre/managing-environment-variables-in-node-js-2cb45a55195f
First I created an application name on Heroku. Then I deployed my app to heroku by connecting to github.
Heroku provides the database credentials after we deploy our applications. Then I redeployed the app through github by changing the configuration in application.properties file as follows:
#localhost configuration
SPRING_DATASOURCE_DRIVER_CLASS_NAME=org.postgresql.Driver
SPRING_DATASOURCE_URL=jdbc:postgresql://localhost/transactions?useSSL=false
SPRING_DATASOURCE_USER=postgres
SPRING_DATASOURCE_PASSWORD=some_pass
#server database configuration
SPRING_DATASOURCE_DRIVER_CLASS_NAME=org.postgresql.Driver
SPRING_DATASOURCE_URL=jdbc:postgresql://ec2-23-23-247-222.compute-1.amazonaws.com/d6kk9c4s7onnu?useSSL=false
SPRING_DATASOURCE_USER=rimjvlxrdswwou
SPRING_DATASOURCE_PASSWORD=dd903753bc0adffb96ce541b1d55fb043472e32e28031ddc334175066aa42f69
Then you have to edit the config vars according to your application.properties files as shown in the figure below
config_var.png