I am trying to revamp my microservice to maintain a single application yaml rather maintaining multiple profiles. Initially I was maintaining different profiles and the common configurations were repeated across the helm environment specific values yaml. Now the strategy is to move everything to values.yml and maintain environment specific values in config map. The problem I face is my application yml now looks very generic with placeholders and for the same reason the test runs fails as I cannot give a default value for each of the configurations in application.yml. The reason being, For Eg: mongodb cluster endpoint format is different in local to other environment. I managed to place a local specific yaml file under test/resources, but not sure it's the right approach. I need to anyway maintain a local specific yaml under main/resources for running locally. So essentially I am duplicating it under test resources as well. Is there any better way of pointing test to load the application-local.yml under main resources so that I can avoid the duplication or is there any better way of doing this as a whole?
1. Working with multiple configs in One File
You can add all your configurations in one property file as illustrated below
spring.application.name: test. ## Used for all profiles
---
spring.config.active.on-profile:dev
spring.database.host: localhost
spring.database.name: testing
---
spring.config.active.on-profile:prod. ##You can use spring.profiles:prod
spring.database.host: localhost
spring.database.name: testing
---
--- marks where yml document splits, while #--- marks where properties file splits.
Multi-document property files are often used in conjunction with the following activation properties
spring.config.activate.on-profile
spring.config.activate.on.on-cloud-platform
All property definitions defined without specifying the profile name are used on all profiles. In the above case spring.application.name will be used on all profiles dev or prod.
When running the application you can manually specify profile or you can set in within the yml or properties on properties that are used throughout the application.
spring.application.name: test
spring.profiles.active: prod
2. Testing your application
when running tests that need to access properties in yml(property)file there is
no need to redefine your configurations.Just add #ActiveProfile("profile-name")
on your tests.
for example:
#ActiveProfiles("dev")
#SpringBootTest(webEnvironmentSpringBootTest.WebEnvironment.RANDOM_PORT)
Related
I have seen projects where a standalone application.properties is defined under src/test/resources containing all the properties from the default application.properties defined under src/main/resources while overriding properties for testing purposes. The disadvantage of this approach is that every change (adding/removing/updating of properties) must be done in both application.properties, otherwise tests may fail.
On the other hand I have seen projects where only a context based property file is defined under src/test/resources, say application-test.properties containing only the overriden properties for testing purposes. This approach merges the default application.properties under src/main/resources and the application-test.properties under src/test/resources. The respective context must be activated in test classes via #ActiveProfiles, say #ActiveProfiles("test")
What is the approach you follow and why? Is there any best practice or recommendation for this?
Thanks for your replies in advance.
Profiles are generally used to "map" different Spring beans accordingly. I've seen that people usually stick to three of them for most of the workflows: development, test, and production — mainly in tutorials and blogs.
I use profiles aligned with the actual environments the application is going to be deployed at since it's more realistic. So I always run the tests with the local profile. If I need any specific settings for running the tests, I just redefine an application.properties file under src/test/resources and overwrite what's needed.
In this way, you would always effectively have what's going to be used to run the application in each environment within src/main/resources, and if anything needs to be overwritten only for testing, the src/test/resources/application.properties could take care of that.
For instance, let's say your src/main/resources/application.yaml file looks like:
application:
key: value-common
another-key: value-common-another
spring:
profiles:
active: default
---
spring:
config:
activate:
on-profile: local
main:
banner-mode: console
---
spring:
config:
activate:
on-profile: test
This defines effectively two profiles: local and test — because default would have what's common to both. Now, if you place an application.properties file with the following content in src/test/resources:
application.key: value-common-test
...everything will stay the same, but application.key will be bound to value-common-test instead when you run the tests (for any profile). If you happen to just run the application, it will be bound to value-common instead.
I do have a lot of shared properties across my main and test module. It seems that when running the modules the only properties that are picked up are located inside the <module>/resources/application.properties the documentation does not mention the resource folder, while there is nothing in my grade configuration.
Have anyone had any success with sharing properties across modules without having to:
* use spring.config.location
* copy the configuration as part of the gradle script
* have a custom configuration in IntelliJ to use the correct config file
You could create an application-test.properties file in src/test/resources and make all your tests use the test profile, which will overlay the default application.properties file values.
This allows you to use your defaults but change only the props you need different for testing.
This also allows you to create different test profiles for different types of tests - by simply creating a new file and specifying a different profile.
See Also: https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-profiles.html
I understand there's multiple ways of handling application property files and profiles in Spring Boot and I've seen multiple questions and answers on how to handle each but I'm trying to find the "best" way of handling it for a new project, if there is one.
The application is hosted in Weblogic 12c on production/pre-prod (with a jndi database connection) and ran locally in tomcat (with hardcoded database details) for development. I'd like it so that when the project is built via gradle and deployed to production it uses the jndi properties file and when ran locally it defaults to the hardcoded datasource with minimal changes required.
src/main/resources/application.properties
# DEV
spring.datasource.url=
spring.datasource.username=
spring.datasource.password=
spring.datasource.driver-class-name=oracle.jdbc.driver.OracleDriver
# DEV
# PROD
# spring.datasource.jndi-name=
# spring.datasource.driver-class-name=oracle.jdbc.driver.OracleDriver
# PROD
From my understanding the recommended way is to externalize the property files and place the required one in a config directory alongside the WAR file for any differing config which is then automatically picked up and used?
You should consider creating multiple profiles. This means: Either multiple properties-Files, or multiple profiles in one file:
See https://docs.spring.io/spring-boot/docs/current/reference/html/howto-properties-and-configuration.html
I would recommend to use multiple application-ENV.properties, e.g.
application-prod.properties and application-preprod.properties.
There is always one active profile and settings from the application.properties (without any profile suffix) are used as default values if not overwritten in a specific profile-file.
Depdending on your environment (local, prod etc.) you should set an environment variable (start the java-process/application server with that environment variable), e.g.:
SPRING_PROFILES_ACTIVE=prod
On your local machine you would set:
SPRING_PROFILES_ACTIVE=dev
With this variable you can control, which profile is currently active.
Also consider integrating the active profile into you Continious Integration/Deployment settings.
Please note that putting plain text passwords hardcoded into committed files is not a good idea. Consider using jasypt or a spring cloud config server for your prod database configuraiton or use any mechanism that your cloud provider provides to you if you use any. Microsoft Azure for example provides a KeyVault for sensitive data.
https://cloud.spring.io/spring-cloud-config/multi/multi_spring-cloud-config.html
http://www.jasypt.org/
If you use gradle good solution is to set application.properties and test.properties files and put into them properties for prod and preprod respectively.
Then run application with different vm arguments: -Dspring.profiles.active=test for test.properties and without arguments for application.properties
Use gradle instruments and configure them once for test and prod. For example: gradle bootWar configure without vm arguments, gradle bootWarTest with vm arguments -Dspring.profiles.active=test. Save once you configs and you will create war for different environments only selecting between two buttons in gradle.
I currently have the following config setup in spring boot:
application.properties
app.database.host=${DB_HOST}
app.database.port=${DB_PORT}
app.database.name=${DB_NAME}
app.database.user=${DB_USER}
app.database.password=${DB_PASSWORD}
app.database.schema=${DB_SCHEMA:public}
spring.datasource.url=jdbc:postgresql://${app.database.host}:${app.database.port}/${app.database.name}
spring.datasource.username=${app.database.user}
spring.datasource.password=${app.database.password}
spring.datasource.driver-class-name=org.postgresql.Driver
spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.PostgreSQLDialect
application-local-dev.properties:
app.database.host=${DB_HOST:localhost}
app.database.port=${DB_PORT:5432}
app.database.name=${DB_NAME:db_name}
app.database.user=${DB_USER:root}
app.database.password=${DB_PASSWORD:root}
app.database.schema=${DB_SCHEMA:public}
application-load-fixtures.properties:
spring.profiles.include=local-dev
spring.profiles.active=load-fixtures,local-dev
app.database.name=${DB_NAME:db_name}_fixtures
The idea here is that when starting the app in default mode, it will fail to boot when critical properties like database name are missing.
They should be passed via environment variables.
For development purposes, this is unnecessary overhead when setting up the project because we have a docker container with static credentials and I'd like to provide them as defaults. Therefore, I created a profile local-dev that will use default values to be able to connect to our docker database and still have the ability to override them via environment variables in case someone needs to.
Until here, everything works fine.
But now, we also have a profile that is used to load fixtures into the database (drop all tables, recreate and fill them with data).
For obvious reasons, I want to ensure that this cannot be done on an arbitrary database, so I created a profile load-fixtures that should inherit all properties from local-dev and override the database name. However, this approach seems to be wrong. I can see in the spring log that the profiles are loaded properly:
2017-11-16 13:32:11.508 INFO 23943 --- [ main] Main:
The following profiles are active: load-fixtures,local-dev
But it still uses the database name provided by the local-dev profile.
When I remove the line
app.database.name=${DB_NAME:db_name}
from the local-dev config file, it works.
However, what I want to avoid is having to add new properties to both, local-dev and load-fixtures, whenever we add a new configuration property to the project.
I understand that profile specific properties take precedence over non-profile specific ones. And also that non-default location properties take precedence over properties from the default locations. But here, both profiles (local-dev and load-fixtures) are in the same location, and they are also both profile specific.
What are proper ways to go about this problem?
Thanks in advance!
I recently came across quite the same problem and had to figure out which precedence Spring applies to several profile specific property files. Unfortunately this is not well documented and I did not find the location of the code that is responsible for that.
However after some tests and tries I'm pretty sure it works like this (or at least in a similar way):
Probably some kind of map is used to gather up all properties of all the different places and possibilites where you could define them like documented here. So for example a property my.value is defined in application.properties and so stored in the mentioned map. Then the same property is found as Java system property. Since this way of defining a property is higher in the PropertySource-order it will override the value found before in the map. Until here it is clear according to the documentation that the Java system property will win.
But as we come to two different sources on the same precedence level like two different profile specific property files the documentation is not a 100% clear in my opinion. However it says in 24.4:
If several profiles are specified, a last-wins strategy applies. For example, profiles specified by the spring.profiles.active property are added after those configured through the SpringApplication API and therefore take precedence.
Maybe it is just the example that is not optimal here or I just do not understand it correctly. But I guess the "last-wins" strategy also applies to all profiles defined for example in spring.profiles.active. That means if you run java -jar -Dspring.profiles.active=dev,fix application.jar, the properties in application-fix.properties will overwrite the values of properties having the same key in application-dev.properties.
So in your case considering the output of your application I guess you specified something like java -jar -Dspring.profiles.active=load-fixtures,local-dev application.jar. If I was correct, you would just have to change that into java -jar -Dspring.profiles.active=local-dev,load-fixtures application.jar.
I am trying to find the best way to configure my Spring Boot Web application to easily switching between the following data sources for both local testing and deployment.
H2 in memory db. Local testing only.
Dev oracle. Local testing and deployment.
Prod oracle. Deployment only.
By local testing, I mean to test in IDE environment (Eclipse). Dev and prod oracle databases are set up on two remote servers.
After some research, there are different ways to switch from one data source to another.
Use Spring profile. Using H2 and Oracle with Spring Boot. Set up the following files in classpath, application.properties, application-h2. properties and application-dev.properties. While connections for h2 and dev are defined in corresponding properties files, spring.profiles.active is set in application.properties. My understanding is this property can be overridden during build process by specifying spring.profiles.active. However, it seems to be a JVM variable, how do I set it running maven?
Maven profile. Create multiple profiles in pom and a filter pointing to application properties files. The profile specified by -P option during maven build will determine which application properties file to look. However, according to maven application with multi environment configuration can't deploy on tomcat, this will generate multiple wars for different deployment. So method 1 is preferred. Plus, it does not apply to switching datasources while testing locally.
Persistence units. Define different persistence units for different data sources in persistence.xml. Use EntityManager by choosing a specific unit. Variation of this method include having a variable in unit names which is determined in application.properties.
JNDI lookup. Set up a jndi name in application.properties with spring.datasource.jndi-name. The actual database information including url and credentials will be specified in context.xml in the tomcat folder where the war will be deployed.
My mind is set on local testing environment. Gonna go with method 1. Switching between H2 in memory and oracle is so easy just by changing the property in application.properties. Since the testing is usually done in IDE, war does not need to be generated, although answers are welcome for run maven install with spring.profiles.active.
As far as deployment, JNDI is definitely the way to go. However, I am concerned that the two properties in application.properties: spring.profiles.active and spring.datasource.jndi-name may be conflicting with each other. If I have spring.profiles.active=h2 and then tried to deploy the war to prod server, does it try to connect to h2 based on the spring profile or to prod db based on jdni-name? What is the best practice to accommodate all scenarios with enough flexibility?
Also is a explicit configuration class for DataSource required such as Configure Mutiple DataSource in Spring Boot with JNDI? My understanding is application.properties and spring profile should be enough to handle it, right?
Definitely use Spring profiles.
You don't want to use Maven profiles as it creates different artifacts. Ask your QA/Release engineers how they feel about having different artifacts for different environments :). They wouldn't be happy.
H2 is what you want to use in CI server integration testing as well. Such integration testing is fast and easy.
Instead of changing profile in application.properties, consider defining profile via command line parameter. So that configuration file changes are not required to run your application in different profiles.