SpringBoot 2.7.1 with Spring Cloud Config
I have a test
#ActiveProfiles({"local","h2dbmem"})
#SpringBootTest(properties = {"server.ssl.enabled=false"})
#AutoConfigureTestDatabase(replace = AutoConfigureTestDatabase.Replace.NONE)
#TestPropertySource(locations = "classpath:application-integrationtest.properties")
public class SystemTest {
...
}
and I then run Maven to run the test :
mvn -Dspring.profiles.include=cloudconfig clean verify
When the test runs, the console says :
The following 3 profiles are active: "cloudconfig", "local", "h2dbmem"
Although this is exactly what I wanted, that it ends up using the JDBC URL and credentials from the h2dbmem profile and not the JDBC URL and credentials from the cloud-config profile, I could not find documentation on why cloudconfig ( supplied via command-line ) was added as the first entry in the list of active profiles after the profiles supplied via the #ActiveProfiles annotation.
To be honest, I was actually expecting the following order of profiles :
The following 3 profiles are active: "local", "h2dbmem", "cloudconfig"
... where the JDBC URL and credentials from cloudconfig are used, but I am glad it was not.
So question is:
Is there specific documentation on the sequence / order of profiles when they are supplied in multiple ways ( annotation, command-line, etc ) ?
Note I am aware of the order and sequence / preference of properties, which is not what I am asking.
Related
There are several ways present to define "active profiles" for a Spring Boot application.
The default one is to pass it through a command line, like this:
java -Dspring.profiles.active=dev,local -jar myapp.jar
it works just fine (as expected): All three sets of profile-related properties will be loaded in proper order:
application.yaml
application-dev.yaml will override the previous one
application-local.yaml will override the previous one as well (these properties will have the most priority)
Based on the idea, that my "local" profile should always "use and overrides" properties from the "dev", let's "hardcode" such behavior.
Let's use the 'spring.profiles.include' feature for this. So, the following lines are added to the 'application-local.yaml':
spring.profiles:
include:
- dev
I expect, now I can pass the "local" profile only in the command line, and the "dev" profile will be applied automatically (with his properties, of course):
java -Dspring.profiles.active=local -jar myapp.jar
But ooop!*: properties from the 'application-dev.yaml' are ignored.
Why? Is it a bug? Is it a feature that forces me to list all profiles in a command line directly?
I'm sure that the behavior around profiles activation should be the same without any difference in how the active-profiles list was passed to Spring Boot framework.
The application:
#SpringBootApplication #EnableConfigurationProperties( MyProps::class )
class SpringApp4
#ConfigurationProperties("my.db") #ConstructorBinding
data class MyProps(val name: String, val url: String, val user: String)
#Component
class MyRunner(val myProps: MyProps, val env: Environment) : CommandLineRunner {
override fun run(vararg args: String) {
println("myProps = $myProps")
println("activeProfiles = ${env.activeProfiles.joinToString()}")
exitProcess(0)
}
}
fun main() { runApplication<SpringApp4>() }
application.yaml:
my.db:
name: "default-name"
url: "default-url"
user: "default-user"
application-dev.yaml:
my.db:
url: "dev-url"
user: "dev-user"
application-local.yaml:
spring.profiles.include:
- dev
my.db:
user: "local-user"
Run1: java -Dspring.profiles.active=dev,local -jar myapp.jar
Correct output:
myProps = MyProps(name=default-name, url=dev-url, user=local-user)
activeProfiles = dev, local
it's correct because the url=dev-url
Run2: java -Dspring.profiles.active=local -jar myapp.jar
Incorrect output:
myProps = MyProps(name=default-name, url=default-url, user=local-user)
activeProfiles = local
It's not correct because the url=default-url and the activeProfiles doesn't contain the "dev" at all.
Help me please to figure out how to use the spring.profiles.include feature in yaml to build a kind of top level profiles that will activate other automatically.
In Run2 You are giving profile as local
i.e
-Dspring.profiles.active=local
So spring will first load application.yml and then application-local.yml
I can see the output is expected.
Since some properties like name and url are not present in application-local.yml, so the values of these fields will be same as present in application.yml
FYI : application.yml is always called irrespective of profile, and then it gets overridden by the profile mentioned in -Dspring.profiles.active property
spring.profiles.include deprecated in Spring Boot 2.4 and no longer works: https://spring.io/blog/2020/08/14/config-file-processing-in-spring-boot-2-4
It caused recursive resource loading; that broke Kubernates ConfigMap so they removed recursion.
Use spring.profiles.active or spring.profiles.group.
It's my first time implementing a Spring application using Spring Cloud Config framework. The application is working fine, getting configuration properties from repository through the server application. So, at the moment I must write the integration tests to test connection between client application and server and repository. To do it I mean get a value from repository over properties and check the value, also make a request to the server, or if exists, other method inside spring cloud config library to make it.
The problem now, when executing the integrationTest the application can't read the properties from remote. I've created a bootstrap.yml on the integrationTest context but not works. So, I got the errors like: Could not resolve placeholder.
MyIntegrationTest
#RunWith(SpringJUnit4ClassRunner.class)
#SpringBootTest()
#ActiveProfiles("integrationTest")
public class MyIntegrationTest {
#Value("${throttling.request.rate.minute}")
private int MAX_REQUEST_PER_MINUTE;
#Test
public void validateGetRequestRateLimitFromRepository() {
Assert.assertNotEquals(MAX_REQUEST_PER_MINUTE,1L);
}
}
bootstrap.yml
spring:
application:
name: sample-name
---
spring:
profiles: local #also tried integrationTest
cloud:
config:
enabled: true
uri: https://endpoint-uri:443
skipSslValidation: true
name: ${spring.application.name},${spring.application.name}-${spring.profiles}
Also tried:
Set application-integrationTest.yml
Set application.yml
Set bootstrap-integrationTest.yml
Create a *-integrationTest.yml at the repo.
Set #ActiveProfiles("local") on integrationTest class.
Nothing works.
I am using to configure spring boot with an external YAML configuration and CMD.
-> application.yml file
spring:
profiles: integration-test
datasource:
driverClassName: ${SPRING_DATASOURCE_DRIVER_CLASS_NAME}
url: ${SPRING_DATASOURCE_URL}
username: ${SPRING_DATASOURCE_USERNAME}
password: ${SPRING_DATASOURCE_PASSWORD}
-> cmd
mvn clean install
-> Result
Caused by: java.lang.IllegalStateException: Cannot load driver class: ${SPRING_DATASOURCE_DRIVER_CLASS_NAME}
Can anyone explain this to me?
When you use the syntax ${}, you are actually telling Spring Boot to use the value of the property whose name is between brackets. In your case, Spring Boot tries to resolve the property SPRING_DATASOURCE_DRIVER_CLASS_NAME. When it fails, it uses the string as is, which leads to the error you mentioned, since no driver exists under the name ${SPRING_DATASOURCE_DRIVER_CLASS_NAME}.
To solve the issue, you can either :
replace the ${} by the real values, e.g. driverClassName: org.postgresql.Driver and do the same for the other properties (url, username and password)
provide properties SPRING_DATASOURCE_DRIVER_CLASS_NAME,SPRING_DATASOURCE_URL and the two others. These can passed in the command line with -D options (e.g. -DSPRING_DATASOURCE_DRIVER_CLASS_NAME=org.postgresql.Driver) or through environment variables. You can look at spring Boot documentation for more details.
Pass those variables in your launch configuration of your program or at commandline when you run your app with java YourMainClass, e.g.
java -DSPRING_DATASOURCE_DRIVER_CLASS_NAME=<full_qualified_name_of_your_jdbc_driver_class> -DSPRING_DATASOURCE_URL=<jdbc_url> YourMainClass
also pass the other two variables the same way, username & password!
your can even set those enviroment variables on OS level, so you don't have to set them each time you start your application...
if your using Spring Boot also have a look at this one: https://docs.spring.io/spring-boot/docs/current/reference/html/boot-features-external-config.html
Updated Question based upon feedback:
I have a spring-boot application that has three databases: H2 for integration testing, and Postgresql for qa & production. Since spring-boot creates a default datasource for you, I don't have anything defined for my integration tests. I thought I would use application.properties to define my datasource connection values but I am not certain what is the best way to handle this.
I have two files:
src/main/resources/application.properties
spring.profiles.active=production
appName = myProduct
serverPort=9001
spring.datasource.url=jdbc:postgresql://localhost/myDatabase
spring.datasource.username=user
spring.datasource.password=password
spring.datasource.driverClassName=org.postgresql.Driver
spring.jpa.hibernate.hbm2ddl.auto=update
spring.jpa.hibernate.ejb.naming_strategy=org.hibernate.cfg.EJB3NamingStrategy
spring.jpa.hibernate.show_sql=true
spring.jpa.hibernate.format_sql=true
spring.jpa.hibernate.use_sql_comments=false
spring.jpa.hibernate.type=all
spring.jpa.hibernate.disableConnectionTracking=true
spring.jpa.hibernate.default_schema=dental
src/main/resources/application-test.properties
spring.profiles.active=test
serverPort=9002
spring.datasource.url = jdbc:h2:~/testdb
spring.datasource.username = sa
spring.datasource.password =
spring.datasource.driverClassName = org.h2.Driver
liquibase.changeLog=classpath:/db/changelog/db.changelog-master.sql
I used to run my tests with with gradle (using "gradle build test") or within IntelliJ. I updated my gradle file to use:
task setTestEnv {
run { systemProperty "spring.profiles.active", "test" }
}
But when I run gradle clean build setTestEnv test I get errors that seem to indicate the test is trying to connect to an actual Postgresql database:
Caused by: org.postgresql.util.PSQLException: Connection refused. Check that the hostname and port are correct and that the postmaster is accepting TCP/IP connections.
at org.postgresql.core.v3.ConnectionFactoryImpl.openConnectionImpl(ConnectionFactoryImpl.java:138)
at org.postgresql.core.ConnectionFactory.openConnection(ConnectionFactory.java:66)
at org.postgresql.jdbc2.AbstractJdbc2Connection.<init>(AbstractJdbc2Connection.java:125)
at org.postgresql.jdbc3.AbstractJdbc3Connection.<init>(AbstractJdbc3Connection.java:30)
at org.postgresql.jdbc3g.AbstractJdbc3gConnection.<init>(AbstractJdbc3gConnection.java:22)
at org.postgresql.jdbc4.AbstractJdbc4Connection.<init>(AbstractJdbc4Connection.java:32)
at org.postgresql.jdbc4.Jdbc4Connection.<init>(Jdbc4Connection.java:24)
at org.postgresql.Driver.makeConnection(Driver.java:393)
at org.postgresql.Driver.connect(Driver.java:267)
at org.apache.tomcat.jdbc.pool.PooledConnection.connectUsingDriver(PooledConnection.java:278)
at org.apache.tomcat.jdbc.pool.PooledConnection.connect(PooledConnection.java:182)
at org.apache.tomcat.jdbc.pool.ConnectionPool.createConnection(ConnectionPool.java:701)
at org.apache.tomcat.jdbc.pool.ConnectionPool.borrowConnection(ConnectionPool.java:635)
at org.apache.tomcat.jdbc.pool.ConnectionPool.init(ConnectionPool.java:486)
at org.apache.tomcat.jdbc.pool.ConnectionPool.<init>(ConnectionPool.java:144)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.pCreatePool(DataSourceProxy.java:116)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.createPool(DataSourceProxy.java:103)
at org.apache.tomcat.jdbc.pool.DataSourceProxy.getConnection(DataSourceProxy.java:127)
at liquibase.integration.spring.SpringLiquibase.afterPropertiesSet(SpringLiquibase.java:288)
... 42 more
Caused by: java.net.ConnectException: Connection refused
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339)
I haven't figured out how to set the default system.property == "test" within IntelliJ yet...
See section 21.3 of the Spring Boot documentation. This section describes how to define profile specific property files that use the format application-{profile}.properties. This can help you isolate properties on a per profile basis.
You can annotate your tests adding #ActiveProfiles in the following way:
#SpringApplicationConfiguration(classes = {Application.class, TestSpringConfiguration.class})
#Test(groups = "integration")
#ActiveProfiles("test")
public class MyServiceTest extends AbstractTransactionalTestNGSpringContextTests {
...
#Test
public void testSomething() {
...
}
}
I'm using TestNG but JUnint wouldn't be much different.
You can also specify additional configuration as showed in the example above.
That way you won't need to set the profile in build.gradle or launch configuration in IntelliJ
i'm trying to pass command line arguments in my Spring Boot Yarn application and am having difficulties. i understand that i can set these in the yml document spring.yarn.appmaster.launchcontext.arguments but how can it from the command line? like java -jar MyYarnApp.jar {arg0} {arg1} and get access to it from my #YarnContainer?
i've discovered that #YarnProperties maps to spring.yarn.appmaster.launchcontext.arguments but i want to set them from the command line, not in the yml
You are pretty close on this when you found spring.yarn.client.launchcontext.arguments and spring.yarn.appmaster.launchcontext.arguments. We don't have settings which would automatically pass all command-line arguments from a client into an appmaster which would then pass them into a container launch context. Not sure if we even want to do that because you surely want to be on control what happens with YARN container launch context. User using a client could then potentially pass a rogue arguments along a food chain.
Having said that, lets see what we can do with our Simple Single Project YARN Application Guide.
We still need to use those launch context arguments to define our command line parameters to basically map how things are passed from a client into an appmaster into a container.
What I added in application.yml:
spring:
yarn:
client:
launchcontext:
arguments:
--my.appmaster.arg1: ${my.client.arg1:notset1}
appmaster:
launchcontext:
arguments:
--my.container.arg1: ${my.appmaster.arg1:notset2}
Modified HelloPojo in Application class:
#YarnComponent
#Profile("container")
public static class HelloPojo {
private static final Log log = LogFactory.getLog(HelloPojo.class);
#Autowired
private Configuration configuration;
#Value("${my.container.arg1}")
private String arg1;
#OnContainerStart
public void onStart() throws Exception {
log.info("Hello from HelloPojo");
log.info("Container arg1 value is " + arg1);
log.info("About to list from hdfs root content");
FsShell shell = new FsShell(configuration);
for (FileStatus s : shell.ls(false, "/")) {
log.info(s);
}
shell.close();
}
}
Notice how I added arg1 and used #Value to map it with my.container.arg1. We can either use #ConfigurationProperties or #Value which are normal Spring and Spring Boot functionalities and there's more in Boot's reference docs how to use those.
You could then modify AppIT unit test:
ApplicationInfo info = submitApplicationAndWait(Application.class, new String[]{"--my.client.arg1=arg1value"});
and run build with tests
./gradlew clean build
or just build it without running test:
./gradlew clean build -x test
and then submit into a real hadoop cluster with your my.client.arg1.
java -jar build/libs/gs-yarn-basic-single-0.1.0.jar --my.client.arg1=arg1value
Either way you see arg1value logged in container logs:
[2014-07-18 08:49:09.802] boot - 2003 INFO [main] --- ContainerLauncherRunner: Running YarnContainer with parameters [--spring.profiles.active=container,--my.container.arg1=arg1value]
[2014-07-18 08:49:09.806] boot - 2003 INFO [main] --- Application$HelloPojo: Container arg1 value is arg1value
Using format ${my.client.arg1:notset1} also allows you to automatically define a default value notset1 if my.client.arg1 is omitted by user. We're working on Spring Application Context here orchestrated by Spring Boot so all the goodies from there are in your disposal
If you need more precise control of those user facing arguments(using args4j, jopt, etc) then you'd need to have a separate code/jar for client/appmaster/container order to create a custom client main method. All the other Spring YARN getting started guides are pretty much using multi-project builds so look at those. For example if you just want to have first and second argument value without having a need to use full --my.client.arg1=arg1value on a command line.
Let us know if this works for you and if you have any other ideas to make things simpler.