I want to configure the properties of the Tomcat JDBC Pool with custom parameter values. The pool is bootstrapped by the spring-cloud (Spring Cloud Connector) environment (Cloud Foundry) and connected to a PostgreSQL database. In particular, I want to set the minIdle, maxIdle and initialSize properties for the given pool.
In a "spring-vanilla" environment (non-cloud) the properties can be easily set by using
application.properties / .yaml files with environment properties,
#ConfigurationProperties annotation.
However, this approach doesn't transfer to my Cloud environment, where the URL (and other parameters) are injected from the environment variable VCAP_SERVICES (via the ServiceInfo instances). I don't want to re-implement the logic which Spring Cloud already did with its connectors.
After some searching I also stumbled over some tutorials / guides, which suggest to make use of the PoolConfig object (e.g. http://cloud.spring.io/spring-cloud-connectors/spring-cloud-spring-service-connector.html#_relational_database_db2_mysql_oracle_postgresql_sql_server). However, that way one cannot set the properties I need but merely the following three:
minPoolSize,
maxPoolSize,
maxWaitTime.
Note that I don't want to set connection-related properties (such as charset), but the properties are associated with the pool itself.
In essence, I would like to do the configuration similarly to https://www.baeldung.com/spring-boot-tomcat-connection-pool (using spring.datasource.tomcat.* properties). The problem with that approach is that the properties are not considered if the datasource was created by Spring Cloud. The article https://dzone.com/articles/binding-data-services-spring, section "Using a CloudFactory to create a DataSource", claims that the following code snippet makes it so that the configuration "can be tweaked using application.properties via spring.datasource.* properties":
#Bean
#ConfigurationProperties(DataSourceProperties.PREFIX)
public DataSource dataSource() {
return cloud().getSingletonServiceConnector(DataSource.class, null);
}
However, my own local test (with spring-cloud:Greenwich.RELEASE and spring-boot-starter-parent:2.1.3.RELEASE) showed that those property values are simply ignored.
I found an ugly way to solve my problem but I think it's not appropriate:
Let spring-cloud create the DataSource, which is not the pooled DataSource directly,
check that the reference is a descendant of a DelegatingDataSource,
resolve the delegate, which is then the pool itself,
change the properties programmatically directly at the pool itself.
I do not believe that this is the right way as I am using internal knowledge (on the layering of datasources). Additionally, this approach does not work for the property initialSize, which is only considered when the pool is created.
Related
I use WAS Liberty Server container which provides server.xml and server.env file for configuring many things in addition to configuring DataSource properties such as url, username, password etc.
For security reasons, these properties are not known to developers for production Liberty servers. However, developers use same server.xml/server.evn files but with different DataSource properties so they can do their work.
So, I might have two server.env files like:
PRODUCTION server.env: dataSourceUrl="server-A-URL" (this URL is not known to developers)
DEVELOPMENT server.env: dataSourceUrl="server-B-URL" (this URL is known to developers)
, then the dataSourceUrl is used in server.xml files in production and development to set the url accordingly.
So, the structure of server.xml/server.env file is same for developers and production, only the DataSource url, username, password are different. This way developers can work using their own DataSource properties and once ready to deploy they app, it is handed to other team which then just changes the DataSource properties to the production values and deploys the application to production server.
With Springboot JPA, I know we can use application.properties file to set these DataSource values. But, we would like to be able to set these to the values located in server.env file. Basically to have something like this in application.properties file:
spring.datasource.url=dataSourceUrl //dataSourceUrl is set in server.env
, then be able to use this value in Java code using #Value injection like:
public class MyClass {
#Value("${spring.datasource.url}")
String dsUrl;
...
}
I have been reading about externalizing properties but I am not able to figure out how to do this
You can use Liberty's jndiEntry elements to make configured values available in JNDI. You will need the jndi-1.0 feature, after which you can configure,
<jndiEntry jndiName="spring/datasource/url" value="${dataSourceUrl}"/>
and access it in your application as:
String dsUrl = InitialContext.doLookup("spring/datasource/url");
Or, from a web or ejb component as:
#Resource(lookup = "spring/datasource/url")
String dsUrl;
I have spring boot micro-service with database credentials define in the application properties.
spring.datasource.url=<<url>>
spring.datasource.username=<<username>>
spring.datasource.password=<<password>>
We do not use spring data source to create the connection manually. Only Spring create the database connection with JPA.(org.springframework.boot.autoconfigure.orm.jpa.HibernateJpaAutoConfiguration)
We only provide the application properties, but spring create the connections automatically to use with the database connection pool.
Our requirement to enhance the security without using db properties in clear text. Two possible methods.
Encrypt the database credentials
Use the AWS secret manager. (then get the credential with the application load)
For the option1, jasypt can be used, since we are just providing the properties only and do not want to create the data source manually, how to do to understand by the spring framework is the problem. If better I can get some working sample or methods.
Regarding the option-2,
first we need to define secretName.
use the secertName and get the database credentials from AWS secret manager.
update the application.properties programatically to understand by spring framework. (I need to know this step)
I need to use either option1 and option2. Mentioned the issues with each option.
What you could do is use environment variables for your properties. You can use them like this:
spring.datasource.url=${SECRET_URL}
You could then retrieve these and start your Spring process using a ProcessBuilder. (Or set the variables any other way)
I have found the solution for my problem.
We need to define org.springframework.context.ApplicationListenerin spring.factories file. It should define the required application context listener like below.
org.springframework.context.ApplicationListener=com.sample.PropsLoader
PropsLoader class is like this.
public class PropsLoader implements ApplicationListener<ApplicationEnvironmentPreparedEvent> {
#Override
public void onApplicationEvent(ApplicationEnvironmentPreparedEvent event) {
ConfigurableEnvironment environment = event.getEnvironment();
String appEnv = environment.getProperty("application.env");
//set new properties based on the application environment.
// calling other methods and depends on the enviornment and get the required value set
Properties props = new Properties();
props.put("new_property", "value");
environment.getPropertySources().addFirst(new PropertiesPropertySource("props", props));
}
}
spring.factories file should define under the resources package and META-INF
folder.
This will set the application context with new properties before loading any other beans.
Can I dynamically refresh properties that are used by Spring Boot's auto configuration setup?
For example, I have the following properties set (via cloud config) to auto configure a dataSource:
spring.datasource.username=user1
spring.datasource.password=test
Now if I change the password prop on the config server, and hit the /refresh endpoint, I can see that the updated prop is retrieved but the DataSource is not refreshed.
I know I can manually configure the DataSource beans and make sure they fall under a RefreshScope, but I was hoping to find a way to mark the auto configured properties as "refreshable". I have some use cases where I'd want to refresh props used by Spring Boot for other beans besides DataSources, and setting up some of those beans manually could be a pain.
I think I spoke too soon, at least as far as my DataSource example goes. A new db connection was being created with the updated props.
Which makes sense especially when looking at the docs here
This didn't re-connect some of my spring.cloud.stream.bindings properties I had, but in that case I can probably solve the issue with #RefreshScope.
There's a configuration property to set in case of the Autoconfigured bean is immutable (don't change the properties after initialized)
You can put a list (set) of classes that you need to be refreshed and you don't have control over the source code, you can put them under the property: spring.cloud.refresh.extra-refreshable
e.g.:
spring
cloud
refresh
extra-refreshable:
- org.springframework.mail.javamail.JavaMailSenderImpl
see: https://cloud.spring.io/spring-cloud-static/Greenwich.SR1/single/spring-cloud.html#refresh-scope
I am working on a project that has a requirement of Pivotal GemFire.
I am unable to find a proper tutorial about how to configure gemFire with Spring Boot.
I have created a partitioned Region and I want to configure Locators as well, but I need only server-side configuration as client is handled by someone else.
I am totally new to Pivotal GemFire and really confused. I have tried creating a cache.xml but then somehow a cache.out.xml gets created and there are many issues.
#Priyanka-
Best place to start is with the Guides on spring.io. Specifically, have a look at...
"Accessing Data with GemFire"
There is also...
"Cache Data with GemFire", and...
"Accessing GemFire Data with REST"
However, these guides focus mostly on "client-side" application concerns, "data access" (over REST), "caching", etc.
Still, you can use Spring Data GemFire (in a Spring Boot application even) to configure a GemFire Server. I have many examples of this. One in particular...
"Spring Boot GemFire Server Example"
This example demonstrates how to bootstrap a Spring Boot application as a GemFire Server (technically, a peer node in the cluster). Additionally, the GemFire properties are specified Spring config and can use Spring's normal conventions (property placeholders, SpEL expression) to configure these properties, like so...
https://github.com/jxblum/spring-boot-gemfire-server-example/blob/master/src/main/java/org/example/SpringBootGemFireServer.java#L59-L84
This particular configuration makes the GemFire Server a "GemFire Manager", possibly with an embedded "Locator" (indicated by the start-locator GemFie property, not to be confused with the "locators" GemFire property which allows our node to join and "existing" cluster) as well as a GemFire CacheServer to serve GemFire cache clients (with a ClientCache).
This example creates a "Factorials" Region, with a CacheLoader (definition here) to populate the "Factorials" Region on cache misses.
Since this example starts an embedded GemFire Manager in the Spring Boot GemFire Server application process, you can even connect to it using Gfsh, like so...
gfsh> connect --jmx-manager=localhost[1099]
Then you can run "gets" on the "Factorial" Region to see it compute factorials of the numeric keys you give it.
To see more advanced configuration, have a look at my other repos, in particular the Contacts Application RI (here).
Hope this helps!
-John
Well, I had the same problem, let me share with you what worked for me, in this case I'm using Spring Boot and Pivotal GemFire as cache client.
Install and run GemFire
Read the 15 minutes quick start guide
Create a locator(let's call it locator1) and a server(server1) and a region(region1)
Go to the folder where you started the 'Gee Fish'(gfsh) and then go to the locator's folder and open the log file, in that file you can get the port your locator is using.
Now let's see the Spring boot side:
In you Application with the main method add the #EnablegemFireCaching annotation
In the method(wherever it is) you want to cache, add the #Cacheable("region1") annotation.
Now let's create a configuration file for the caching:
//this is my working class
#Configuration
public class CacheConfiguration {
#Bean
ClientCacheFactoryBean gemfireCacheClient() {
return new ClientCacheFactoryBean();
}
#Bean(name = GemfireConstants.DEFAULT_GEMFIRE_POOL_NAME)
PoolFactoryBean gemfirePool() {
PoolFactoryBean gemfirePool = new PoolFactoryBean();
gemfirePool.addLocators(Collections.singletonList(new ConnectionEndpoint("localhost", HERE_GOES_THE_PORT_NUMBER_FROM_STEP_4)));
gemfirePool.setName(GemfireConstants.DEFAULT_GEMFIRE_POOL_NAME);
gemfirePool.setKeepAlive(false);
gemfirePool.setPingInterval(TimeUnit.SECONDS.toMillis(5));
gemfirePool.setRetryAttempts(1);
gemfirePool.setSubscriptionEnabled(true);
gemfirePool.setThreadLocalConnections(false);
return gemfirePool;
}
#Bean
ClientRegionFactoryBean<Long, Long> getRegion(ClientCache gemfireCache, Pool gemfirePool) {
ClientRegionFactoryBean<Long, Long> region = new ClientRegionFactoryBean<>();
region.setName("region1");
region.setLookupEnabled(true);
region.setCache(gemfireCache);
region.setPool(gemfirePool);
region.setShortcut(ClientRegionShortcut.PROXY);
return region;
}
That's all!, also do not forget to serialize(implements Serializable) the class is being cached(The class your cached method is returning)
At the project with Spring Boot we use application.properties but need to configure some of these properties (like port number of logging level) based on an external configuration. We access the configuration via API so it is known only at runtime.
Is there a way to override or set some Spring properties at runtime (for example using a bean) and if yes how can this be achieved?
You could do this with Spring Cloud Config
Just for the purpose of illustration, here's a relatively quick way to see dynamic property overrides at runtime:
First, for your bean to be able to pick up changed properties, you need to annotate it with
#RefreshScope
Add the spring cloud dependency to your spring boot app, eg for gradle
compile group: 'org.springframework.cloud', name: 'spring-cloud-starter', version: '1.1.1.RELEASE'
( NB You also need the spring boot actuator dependency.)
With the app running, you can view your current config at eg
http://localhost:8080/env
eg if you have a property 'my.property' in application.properties, you'll see something like:
"applicationConfig: [classpath:/application.properties]": {
"my.property": "value1",
etc
To change the value, POST my.property=value2 to /env as application/x-www-form-urlencoded
eg
curl -X POST http://localhost:8080 -d my.property=value2
GET /env again and you'll see the new value appears under the "manager" section
To apply the changed properties, do an empty POST to /refresh. Now your bean will have the new value.
Could you use system properties to pass in the variable? If you configure the PropertyPlaceholderConfigurer you can set the precedence of system properties vs file properties.
For example, something like:
#Bean public PropertyPlaceholderConfigurer placeHolderConfigurer() {
PropertyPlaceholderConfigurer props = new PropertyPlaceholderConfigurer()
props.setSystemPropertiesMode( PropertyPlaceholderConfigurer.SYSTEM_PROPERTIES_MODE_OVERRIDE )
props.setLocations(new
PathMatchingResourcePatternResolver().getResources("classpath:/**.properties"));
props
}
The above would load your .properties file, but we set the priority to be system variables first, so if you set a system variable that will override the same variable in the config.
Alternatively, looking at the docs, Spring recommends defining a search order in your Environment:
[PropertyPlaceholderConfigurer is still appropriate for use when]
existing configuration makes use of the "systemPropertiesMode" and/or "systemPropertiesModeName" properties. Users are encouraged to
move away from using these settings, and rather configure property
source search order through the container's Environment; however,
exact preservation of functionality may be maintained by continuing to
use PropertyPlaceholderConfigurer.
Hopefully one of the above should sort out what you need?