I have N Servers, N DBs and N configuration. see the scenario below
So, on every request , I need to access server and db based on configuration.
How can implement dynamically data source in spring data jpa?
You can try AbstractRoutingDatasource provided by Spring since version 2.0.1. using which you can dynamically use appropriate data-source . For integration with Spring data JPA check this very good example. In your case since your configurations are in DB instead of properties file you would need to perform an extra first database lookup to get the appropriate database configuration and return appropriate data-source object.
Another simple approach can be reloading different properties per environment;
Indeed, it might be overkill for DB switching, but it keeps your app simple and maintainable, and most importantly, keeps your environments completely isolated.
Step 1: Configure different properties file per each configuration you have (keep them in src/main/resources with the this naming convention: application-profile.properties)
Step 2: In runtime, change the application context to reload your app based on a given profile
Sample code:
In ProfileController:
#RestController
#RequestMapping("/profile")
public class ProfileController {
#Value("${spring.profiles.active}")
private String profile;
#GetMapping("/profile")
public String getProfile() {
System.out.println("Current profile is: " + profile);
return "Current profile is: " + profile;
}
#GetMapping("/switch/{profile}")
public String switchProfile(#PathVariable String profile) {
System.out.println("Switching profile to: " + profile);
**MyApplication.restartWithNewProfile(profile);**
return "Switched to profile: " + profile;
}
}
In MyApplication.java:
/**
* Switching profile in runtime
*/
public static void restartWithNewProfile(String profile) {
Thread thread = new Thread(() -> {
context.close();
context = SpringApplication.run(MyApplication.class, "--spring.profiles.active=" + profile);
});
thread.setDaemon(false);
thread.start();
}
Related
We are trying to achieve parallelism through the spark executors below are the steps which we are following -
Read from the hive
Data transformation (custom spring library).
Ship it via rest endpoint in batches (1000 records per batch).
Problem - We want to do all these steps in parallel and want to use a spring-boot based library.
Understanding - If we are using the custom code for transformation (the code that we want to run parallelly most probably inside rdd.map() method) then our classes and composite dependencies need to be serialized or those classes need to implement serialization.
We know we can achieve this by performing these tasks in sequence over the driver in such case we need to collect the data over the driver again + again and then pass it to the next step. In this case, we are not leveraging the power of executors.
Needs your assistance here -
If we ship this spring boot dependency to executors then is there any way that the executor understands the spring boot code and resolves the annotations over there ?
sample code -
Code from spring boot library -
public class Process{
String convert(Row row) {
return row.mkString();
}
}
#Component
#ConditionalOnProperty(name = "process.dummy.serialize", havingValue = "true")
class ProcessNotSerialized extends Process {
#Autowired
private RecordService recorService; //not-serialized
public void setName(String name) {
this.name = name;
}
private String name;
#Override
public String toString() {
return "ProcessNotSerialized";
}
}
Code from my spark spring boot application -
Dataset<Row> sqlDf = sparkSession.sql(sqlQuery); // millions of data
ProcessNotSerialized process = new ProcessNotSerialized();
System.out.println("object name=>" + process.toString());
List<String> listColumns = sqlDf.select(column)
.javaRDD()
.map(row -> {
return process.convert(row);
})
.collect();
here is the code inside map() will execute in parallel.
Please let me know if you have any better way other than serialization or running over a driver.
Is it possible to have a Spring Boot properties file depend on two or more profiles? Something like application-profile1-profile2.properties?
Spring Boot does not support this out of the box. It only supports a single profile as described here.
However, it does provide enough flexibility to add your own property sources using EnvironmentPostProcessor.
Here is an example of how to implement this:
public class MultiProfileEnvironmentPostProcessor implements EnvironmentPostProcessor, Ordered {
private final ResourceLoader resourceLoader = new DefaultResourceLoader();
#Override
public void postProcessEnvironment(ConfigurableEnvironment environment, SpringApplication application) {
String[] activeProfiles = environment.getActiveProfiles();
for (int i = 2; i <= activeProfiles.length; i++) {
Generator.combination(activeProfiles).simple(i)
.forEach(profileCombination -> {
String propertySourceName = String.join("-", profileCombination);
String location = "classpath:/application-" + propertySourceName + ".properties";
if (resourceLoader.getResource(location).exists()) {
try {
environment.getPropertySources().addFirst(new ResourcePropertySource(propertySourceName, location));
} catch (IOException e) {
throw new RuntimeException("could not add property source '" + propertySourceName + "'", e);
}
}
});
}
}
#Override
public int getOrder() {
return Ordered.LOWEST_PRECEDENCE;
}
}
Couple of things to note:
This implementation only supports .properties files but can easily be extended to .yml files as well.
getActiveProfiles already returns the profiles in an order where the last one wins. This implementation relies on this order and builds the different file names leveraging this order. i.e. if active profiles are: profile1,profile2,profile3 then application-profile1-profile3.properties is supported but application-profile3-profile1.properties isn't, and application-profile1-profile3.properties will override properties defined in application-profile1.properties or application-profile3.properties.
This implementation uses a third party library com.github.dpaukov:combinatoricslib3 to create the different sets of profiles.
The property sources are added to the front of the property source list to override existing sources. But if you have custom property sources that should take precedence you need to modify this a bit to consider them in the order, i.e. by leveraging methods like Environment.addAfter.
Registering an EnvironmentPostProcessor is done using the spring.factories file.
There are 4 ways I know.
insert .yaml or .properties programmatically like Asi Bross Said. Use ResourceLoader or YamlPropertySourceLoader to insert.
Use .yaml. but it will be replace when you have a another spring project to dependent it.
Use properties instead of profiles. (For api project)
Use one #PropertySource to define properties file A.
Get the variables from properties file A and assign them to the parameters in another #PropertySource file path expression.
For example:
resources
/-application.properties <-- remove or empty,because it will be override by application project
/-moduleA
/-application.properties <-- Intellij can identify properties files started with application-
/-application-mysql-dev.properties
/-application-psql-dev.properties
/-application-psql-prod.properties
The content of resources/moduleA/application.properties :
moduleA.custom.profile1=mysql
moduleA.custom.profile2=dev
Content of Java Config file:
#SpringBootApplication
#PropertySources({
#PropertySource("/moduleA/application.properties"),
#PropertySource("/moduleA/application-${moduleA.custom.profile1}-${moduleA.custom.profile2}.properties"),
})
public class ModuleConfig {}
Use properties instead of profiles. (For application project)
resources
/-application.properties
/-application-mysql-dev.properties
/-application-psql-dev.properties
/-application-psql-prod.properties
The content of resources/application.properties :
moduleA.custom.profile1=mysql
moduleA.custom.profile2=dev
The content of SpringMvcApplication.java:
#SpringBootApplication
#PropertySource("/application-${moduleA.custom.profile1}-${moduleA.custom.profile2}.properties")
public class SpringMvcApplication {...}
I build a Spring-Boot application that accesses a Database and extracts data from it. Everything is working fine, but I want to configure the table names from an external .properties file.
like:
#Entity
#Table(name = "${fleet.table.name}")
public class Fleet {
...
}
I tried to find something but I didn't.
You can access external properties with the #Value("...") annotation.
So my question is: Is there any way I can configure the table names? Or can I change/intercept the query that is sent by hibernate?
Solution:
Ok, hibernate 5 works with the PhysicalNamingStrategy. So I created my own PhysicalNamingStrategy.
#Configuration
public class TableNameConfig{
#Value("${fleet.table.name}")
private String fleetTableName;
#Value("${visits.table.name}")
private String visitsTableName;
#Value("${route.table.name}")
private String routeTableName;
#Bean
public PhysicalNamingStrategyStandardImpl physicalNamingStrategyStandard(){
return new PhysicalNamingImpl();
}
class PhysicalNamingImpl extends PhysicalNamingStrategyStandardImpl {
#Override
public Identifier toPhysicalTableName(Identifier name, JdbcEnvironment context) {
switch (name.getText()) {
case "Fleet":
return new Identifier(fleetTableName, name.isQuoted());
case "Visits":
return new Identifier(visitsTableName, name.isQuoted());
case "Result":
return new Identifier(routeTableName, name.isQuoted());
default:
return super.toPhysicalTableName(name, context);
}
}
}
}
Also, this Stackoverflow article over NamingStrategy gave me the idea.
Table names are really coming from hibernate itself via its strategy interfaces. Boot configures this as SpringNamingStrategy and there were some changes in Boot 2.x how things can be customised. Worth to read gh-1525 where these changes were made. Configure Hibernate Naming Strategy has some more info.
There were some ideas to add some custom properties to configure SpringNamingStrategy but we went with allowing easier customisation of a whole strategy beans as that allows users to to whatever they need to do.
AFAIK, there's no direct way to do config like you asked but I'd assume that if you create your own strategy you can then auto-wire you own properties to there. As in those customised strategy interfaces you will see the entity name, you could reserve a keyspace in boot's configuration properties to this and match entity names.
mytables.naming.fleet.name=foobar
mytables.naming.othertable.name=xxx
Your configuration properties would take mytables and within that naming would be a Map. Then in your custom strategy it would simply be by checking from mapping table if you defined a custom name.
Spring boot solution:
Create below class
#Configuration
public class CustomPhysicalNamingStrategy extends SpringPhysicalNamingStrategy{
#Value("${table.name}")
private String tableName;
#Override
public Identifier toPhysicalTableName(final Identifier identifier, final JdbcEnvironment jdbcEnv) {
return Identifier.toIdentifier(tableName);
}
}
Add below property to application.properties:
spring.jpa.properties.hibernate.physical_naming_strategy=<package.name>.CustomPhysicalNamingStrategy
table.name=product
I would like to port two projects to Spring Boot 1.1.6. The are each part of a larger project. They both need to make SQL connections to 1 of 7 production databases per web request based region. One of them persists configuration setting to a Mongo database. They are both functional at the moment but the SQL configuration is XML based and the Mongo is application.properties based. I'd like to move to either xml or annotation before release to simplify maintenance.
This is my first try at this forum, I may need some guidance in that arena as well. I put the multi-database tag on there. Most of those deal with two connections open at a time. Only one here and only the URL changes. Schema and the rest are the same.
In XML Fashion ...
#Controller
public class CommonController {
private CommonService CommonService_i;
#RequestMapping(value = "/rest/Practice/{enterprise_id}", method = RequestMethod.GET)
public #ResponseBody List<Map<String, Object>> getPracticeList(#PathVariable("enterprise_id") String enterprise_id){
CommonService_i = new CommonService(enterprise_id);
return CommonService_i.getPracticeList();
}
#Service
public class CommonService {
private ApplicationContext ctx = null;
private JdbcTemplate template = null;
private DataSource datasource = null;
private SimpleJdbcCall jdbcCall = null;
public CommonService(String enterprise_id) {
ctx = new ClassPathXmlApplicationContext("database-beans.xml");
datasource = ctx.getBean(enterprise_id, DataSource.class);
template = new JdbcTemplate(datasource);
}
Each time a request is made, a new instance of the required service is created with the appropriate database connection.
In the spring boot world, I've come across one article that extended TomcatDataSourceConfiguration.
http://xantorohara.blogspot.com/2013/11/spring-boot-jdbc-with-multiple.html That at least allowed me to create a java configuration class however, I cannot come up with a way to change the prefix for the ConfigurationProperties per request like I am doing with the XML above. I can set up multiple configuration classes but the #Qualifier("00002") in the DAO has to be a static value. //The value for annotation attribute Qualifier.value must be a constant expression
#Configuration
#ConfigurationProperties(prefix = "Region1")
public class DbConfigR1 extends TomcatDataSourceConfiguration {
#Bean(name = "dsRegion1")
public DataSource dataSource() {
return super.dataSource();
}
#Bean(name = "00001")
public JdbcTemplate jdbcTemplate(DataSource dsRegion1) {
return new JdbcTemplate(dsRegion1);
}
}
On the Mongo side, I am able to define variables in the configurationProperties class and, if there is a matching entry in the appropriate application.properties file, it overwrites it with the value in the file. If not, it uses the value in the code. That does not work for the JDBC side. If you define a variable in your config classes, that value is what is used. (yeah.. I know it says mondoUrl)
#ConfigurationProperties(prefix = "spring.mongo")
public class MongoConnectionProperties {
private String mondoURL = "localhost";
public String getMondoURL() {
return mondoURL;
}
public void setMondoURL(String mondoURL) {
this.mondoURL = mondoURL;
}
There was a question anwsered today that got me a little closer. Spring Boot application.properties value not populating The answer showed me how to at least get #Value to function. With that, I can set up a dbConfigProperties class that grabs the #Value. The only issue is that the value grabbed by #Value is only available in when the program first starts. I'm not certain how to use that other than seeing it in the console log when the program starts. What I do know now is that, at some point, in the #Autowired of the dbConfigProperties class, it does return the appropriate value. By the time I want to use it though, it is returning ${spring.datasource.url} instead of the value.
Ok... someone please tell me that #Value is not my only choice. I put the following code in my controller. I'm able to reliably retrieve one value, Yay. I suppose I could hard code each possible property name from my properties file in an argument for this function and populate a class. I'm clearly doing something wrong.
private String url;
//private String propname = "${spring.datasource.url}"; //can't use this
#Value("${spring.datasource.url}")
public void setUrl( String val) {
this.url = val;
System.out.println("==== value ==== " + url);
}
This was awesome... finally some progress. I believe I am giving up on changing ConfigurationProperties and using #Value for that matter. With this guy's answer, I can access the beans created at startup. Y'all were probably wondering why I didn't in the first place... still learning. I'm bumping him up. That saved my bacon. https://stackoverflow.com/a/24595685/4028704
The plan now is to create a JdbcTemplate producing bean for each of the regions like this:
#Configuration
#ConfigurationProperties(prefix = "Region1")
public class DbConfigR1 extends TomcatDataSourceConfiguration {
#Bean(name = "dsRegion1")
public DataSource dataSource() {
return super.dataSource();
}
#Bean(name = "00001")
public JdbcTemplate jdbcTemplate(DataSource dsRegion1) {
return new JdbcTemplate(dsRegion1);
}
}
When I call my service, I'll use something like this:
public AccessBeans(ServletRequest request, String enterprise_id) {
ctx = RequestContextUtils.getWebApplicationContext(request);
template = ctx.getBean(enterprise_id, JdbcTemplate.class);
}
Still open to better ways or insight into foreseeable issues, etc but this way seems to be about equivalent to my current XML based ways. Thoughts?
I am using spring embedded jdbc database configuration in spring context file for Junit testing. I am using in memory database for testing. I am trying to unit test only DAO layer and for unit testing I am using spring container's in memory database.
When I am running Junit test case I am not seeing first test case values in second test case (testAddressCreate in testAddressUpdate test case). I am not using #Before or #After in my Junit for now. I am not sure how spring is creating in memory database and starting it. According to behavior it seems that it is creating or resetting before each test case. Does anyone know about it? Also how can we connect to spring created in memory database. Please suggest.
Spring Configuration is :
<jdbc:embedded-database id="dataSource" type="HSQL">
<jdbc:script location="classpath:createdb.sql" />
Junit test case is :
#ContextConfiguration(locations="classpath:test-context.xml")
#Transactional
public class GenericDaoImplTest extends AbstractTransactionalJUnit4SpringContextTests {
#Autowired
private GenericDao<Address, Long> addressDao;
#Test
public void testAddressCreate() {
Address address = new Address();
address.setAddress1("first address");
address.setCity("First City");
address.setCountry("First one");
address.setPostalCode("22222");
boolean result = addressDao.create(address);
Assert.assertEquals(true, result);
List<Address> listOfAddress = addressDao.findAll();
Assert.assertNotNull(listOfAddress);
for(Address addressTemp : listOfAddress){
System.out.println(addressTemp.getAddress1());
System.out.println(addressTemp.getAddressId());
}
}
#Test
public void testAddressUpdate(){
Address address = new Address();
address.setAddress1("second address");
address.setCity("Second City");
address.setCountry("Second one");
address.setPostalCode("11111");
boolean result = addressDao.create(address);
Assert.assertEquals(true, result);
address.setAddress1("Updated Second Address");
Assert.assertNotNull(addressDao.update(address));
List<Address> listOfAddress = addressDao.findAll();
Assert.assertNotNull(listOfAddress);
for(Address addressTemp : listOfAddress){
System.out.println(addressTemp.getAddress1());
System.out.println(addressTemp.getAddressId());
}
}
}
By default, a #Transactional test method will rollback any changes made to the database during the test method. If you want to change this behavior you need to annotate your method with #Rollback(false). I don't think the documentation is specific about the default behavior, but the Javadocs mentions this here:
Retrieves the TransactionConfigurationAttributes for the specified class which may optionally declare or inherit #TransactionConfiguration. If TransactionConfiguration is not present for the supplied class, the default values for attributes defined in TransactionConfiguration will be used instead.
And the default value defined in TransactionConfiguration for rollback is "true".
These being said, you need something like this to be able to keep the values from the first #Test method in the second one:
#Test
#Rollback(false)
public void testAddressCreate() {
Address address = new Address();
...
}
For connecting at the in-memory HSQLDB, take a look at this post on Stackoverflow.