I have a Maven multi-module Spring Boot project with modules App, Domain1 and Domain2. App depends on Domain1 and Domain2.
Flyway is auto configured in the App module, with migrations in db/migrations. It runs without problems.
In the domain modules, I have AutoConfiguration classes with #AutoConfigureBefore(JpaAutoConfiguration::class) to run flyway manually 'per domain'. Each domain has it's migration files in different locations (to prevent collissions):
Domain1:
#Autowired
fun migrateFlyway(dataSource: DataSource) {
val cfg = FluentConfiguration()
.locations("classpath:migrations/domain1")
.baselineOnMigrate(true)
.table("flyway_domain1_history")
.dataSource(dataSource)
val flyway = Flyway(cfg)
flyway.migrate()
}
Domain2:
#Autowired
fun migrateFlyway(dataSource: DataSource) {
val cfg = FluentConfiguration()
.locations("classpath:migrations/domain2")
.baselineOnMigrate(true)
.table("flyway_domain2_history")
.dataSource(dataSource)
val flyway = Flyway(cfg)
flyway.migrate()
}
The problem is: both flyway instances from the domain run, creating their respective history tables. However: neither of them actually executes the sql files in them, although they exist (checked from IntelliJ in the target folder, and after building the whole thing with Maven, the SQL files exist in the JAR's).
Am I missing something?
Never mind, after years of using Flyway I just didn't understand the baseline property. De default is 1, and my first script has version 1. So Flyway saw it, but skipped. Setting baseline to '000' solved the problem.
Related
Created a Spring Boot application that will need to migrate from "Local Dev" to "Test", "QA" and "Prod" environments.
Application currently uses a "application.properties" for database connectivity and Kafka configuration.
I am wanting to deploy to "Test" and realized that the properties will not work for that enviornment. After reading the ref docs, it looks like I can simply copy the application.properties file and add a new one application-test.properties, so on, and then run the standalone jar with a -Dspring.profiles.active=test and that seems to work.
But by the time I am done, that means I h ave 4 different appliction-XXXXX.properties files in the jar which may or may not be bad. I know the ultimate configuration would be to use Spring Config server, but right now we are not there with regards to this.
Can anyone validate that using multiple properties files is viable and will work for a bit, or if I am looking at th is all wrong. I do not want to have configuration on the servers in each environment, as I am thinking these mini-services should be self-contained.
Any input would be appreciated.
in a word, your configuration file should be outside your source code.
#PropertySource(value = {"classpath:system.properties"})
public class EnvironmentConfig {
#Bean
public static PropertySourcesPlaceholderConfigurer properties() {
return new PropertySourcesPlaceholderConfigurer();
}
}
Let's say it's named "system.properties", which will be uploaded to server at deployment stage under your application classpath.
I am trying to use flyway for DB migrations and Spring boot's flyway support for auto-upgrading DB upon application start-up and subsequently this database will be used by my JPA layer
However this requires that schema be present in the DB so that primary datasource initialization is successful. What are the options available to run a SQL script that will create the required schema before flyway migrations happen.
Note that If I use flyway gradle plugin (and give the URL as jdbc:mysql://localhost/mysql. It does create the schema for me. Am wondering if I could make this happen from Java code on application startup.
Flyway does not support full installation when schema is empty, just migration-by-migration execution.
You could though add schema/user creation scripts in the first migration, though then your migration scripts need to be executed with sysdba/root/admin user and you need to set current schema at the beginning of each migration.
If using Flyway, the least problematic way is to install schema for the first time manually and do a baseline Flyway task (also manually). Then you are ready for next migrations to be done automatically.
Although Flyway is a great tool for database migrations it does not cover this particular use case well (installing schema for the first time).
"Am wondering if I could make this happen from Java code on application startup."
The simple answer is yes as Flyway supports programmatic configuration from with java applications. The starting point in the flyway documentation can be found here
https://flywaydb.org/documentation/api/
flyway works with a standard JDBC DataSource and so you can code the database creation process in Java and then have flyway handle the schema management. In many environment you are likely to require 2 steps anyway as the database/schema creation will need admin rights to the database, while the ongoing schema management will need an account with reduced access rights.
what you need is to implement the interface FlywayCallback
in order to kick start the migration manually from you code you can use the migrate() method on the flyway class
tracking the migration process can be done through the MigrationInfoService() method of the flyway class
Unfortunately if your app has a single datasource that expects the schema to exist, Flyway will not be able to use that datasource to create the scheme. You must create another datasource that is not bound to the schema and use the unbounded datasource by way of a FlywayMigrationStrategy.
In your properties file:
spring:
datasource:
url: jdbc:mysql://localhost:3306/myschema
bootstrapDatasource:
url: jdbc:mysql://localhost:3306
In your config file:
#Bean
#Primary
#ConfigurationProperties("spring.datasource")
public DataSourceProperties primaryDataSourceProperties() {
return new DataSourceProperties();
}
#Bean
#Primary
#ConfigurationProperties("spring.datasource")
public DataSource primaryDataSource() {
return primaryDataSourceProperties().initializeDataSourceBuilder().build();
}
#Bean
#ConfigurationProperties("spring.bootstrapDatasource")
public DataSource bootstrapDataSource() {
return DataSourceBuilder.create().build();
}
And in your FlywayMigrationStrategy file:
#Inject
#Qualifier("bootstrapDataSource")
public void setBootstrapDataSource(DataSource bootstrapDataSource) {
this.bootstrapDataSource = bootstrapDataSource;
}
#Override
public void migrate(Flyway flyway) {
flyway.setDataSource(bootstrapDataSource);
...
flyway.migrate()
}
Spring Boot's FlywayProperties.java supports many of the Flyway settings but not 'baselineVersion' or 'baselineOnMigrate'. I am converting an existing application to Flyway and these setting appear to be designed for this purpose. Our production environment is highly controlled and running a commandline version of flyway there to achieve this is not practical.
Is creating a custom Flyway #Bean the only option here?
You can set any of flyways properties be prefixing them with flyway in your application.yml/.properties.
It is made possible by org.springframework.boot.autoconfigure.flyway.FlywayAutoConfiguration.FlywayConfiguration#flyway which is annotated with #ConfigurationProperties(prefix = "flyway").
If you are using an application.yml add the following:
spring:
flyway:
baselineOnMigrate: true
If using an application.properties add the following:
spring.flyway.baselineOnMigrate = true
Update: added prefix spring (see #pdem comment).
It is impossible. I spent some time today analyzing code of Spring Boot to try to find a solution to this. There is nothing about setting any of these properties in FlywayAutoConfiguration. Also I found that Spring is never calling configure method on Flyway object what would be the only other option for flyway.properties to work. Spring is abusing flyway.properties a little bit and instead of providing this file further to Flyway they use it themselves as a source of properties. That is why the set of possible options when using FlywayAutoConfiguration is so limited. So using FlywayAutoConfiguration is not a good option if you need any more advanced features of Flyway. But using #Bean is not a tragedy here. Below you may see an example of using #Bean this way that implementing this behavior would be impossible with any property files:
#Profile(value = "!dbClean")
#Bean(name = "flyway", initMethod = "migrate")
public Flyway flywayNotADestroyer() {
Flyway flyway = new Flyway();
flyway.setDataSource(dataSource());
flyway.setBaselineOnMigrate(true);
return flyway;
}
#Profile(value = "dbClean")
#Bean(name = "flyway", initMethod = "migrate")
public Flyway flywayTheDestroyer() {
Flyway flyway = new Flyway();
flyway.setDataSource(dataSource());
flyway.setBaselineOnMigrate(true);
flyway.clean();
return flyway;
}
As you can see I have two Spring profiles here. One default that will not clean your database, and one with full clean of the database. Very handy.
I had success using a FlywayMigrationStrategy.
#Component
public class BaselineOnMigrateMigrationStrategy implements FlywayMigrationStrategy {
#Override
public void migrate(Flyway flyway) {
flyway.setBaselineOnMigrate(true);
flyway.migrate();
}
}
You can to use on application.properties file, but you need to add spring. prefix to it for springboot 2 like #pdem marked in this answer comments https://stackoverflow.com/a/39244097/273119.
spring.flyway.baseline-on-migrate=true
I am using flyway 5.1.4, for me adding these in application.properties worked
flyway.enabled = true
flyway.baseline-on-migrate = true
After digging into the source and running some experiments, it would appear that because the setBaselineVersion() is overloaded in the Flyway class, Spring is unable to inject the property value.
Changing to flyway.baselineVersionAsString=2 works as desired.
Answer of Seth is worked for me.
But I changed
flyway.setBaselineOnMigrate(true);
for
flyway.baseline();
Not sure if this is a Gradle question or a Spring Boot one, but here goes...
I am using Spring security and LDAP in a spring boot application.
I have the following dependencies in my build.gradle:
compile 'org.springframework.security:spring-security-ldap:3.2.4.RELEASE'
compile 'org.apache.directory.server:apacheds-server-jndi:1.5.5'
The second of these supplies an embedded LDAP server that is only needed during development.
I have established a SB #Profile and configure/load an LDIF file into the embedded server within a class that has the #Profile('development') annotation.
The question is: how to remove the second dependency when not in dev mode?
I establish the spring.profiles.active property within my config/application.yml file, thusly:
spring:
profiles:
active: development
Can I reference spring.profiles.active so that I can somehow exclude the unneeded dependency?
For posterity, what I ended up doing...
At the top of my build.grade file:
def readActiveProfile() {
final config = new org.yaml.snakeyaml.Yaml().loadAll(new File('config/application.yml').newReader())
final defaultPart = config?.take(1)
defaultPart?.spring?.profiles?.active
}
final activeProfile = readActiveProfile() ?: ['development']
This reads the config file that I am keeping my externalised settings in (one of which is the setting defining the active profile).
And then, in the dependencies section:
compile 'org.springframework.security:spring-security-ldap:3.2.4.RELEASE'
if( ! ('production' in activeProfile))
compile 'org.apache.directory.server:apacheds-server-jndi:1.5.5'
This works well enough for my purposes, but doesn't feel quite right; I was assuming that there would be a more idiomatic "Gradle way" of doing this.
I've been following the development of Spring Boot, and sometime between the initial version 0.0.5-BUILD-SNAPSHOT and the current version I am using 1.0.0.RC1 I am no longer running my import.sql script.
Here is my configuration for LocalContainerEntityManager and JpaVendorAdapter
#Bean
public LocalContainerEntityManagerFactoryBean entityManagerFactory(
DataSource dataSource, JpaVendorAdapter jpaVendorAdapter) {
LocalContainerEntityManagerFactoryBean lef = new LocalContainerEntityManagerFactoryBean();
lef.setDataSource(dataSource);
lef.setJpaVendorAdapter(jpaVendorAdapter);
lef.setPackagesToScan("foo.*");
return lef;
}
#Bean
public JpaVendorAdapter jpaVendorAdapter() {
HibernateJpaVendorAdapter hibernateJpaVendorAdapter = new HibernateJpaVendorAdapter();
hibernateJpaVendorAdapter.setShowSql(true);
hibernateJpaVendorAdapter.setGenerateDdl(true);
hibernateJpaVendorAdapter.setDatabase(Database.POSTGRESQL);
return hibernateJpaVendorAdapter;
}
Interesting the hibernate.hbm2ddl.auto still seems to run, which I think is part of the definition of my SpringBootServletInitializer
#Configuration
#ComponentScan
#EnableAutoConfiguration
public class Application extends SpringBootServletInitializer {
However, I also noticed that the tables generated no longer have underscores and changed their shape when generated?
However, that could be the result of updating my org.postgresql version like so:
Previously:
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<version>9.2-1004-jdbc41</version>
</dependency>
Now:
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<version>9.3-1100-jdbc41</version>
</dependency>
I also had to change pggetserialsequence to pg_get_serial_sequence to get the script to run at all from pgadmin?
I guess I'm confusing what's going on, but most importantly I want to get back to having my import.sql run.
I have been following the sample project: https://github.com/spring-projects/spring-boot/tree/master/spring-boot-samples/spring-boot-sample-data-jpa
And their import.sql isn't running either on 1.0.0-BUILD-SNAPSHOT
The import.sql script is a Hibernate feature I think (not Spring or Spring Boot). It must be running in the sample otherwise the tests would fail, but in any case it only runs if ddl-auto is set to create the tables. With Spring Boot you should ensure that spring.jpa.hibernate.ddl-auto is set to "create" or "create-drop" (the latter is the default in Boot for an embedded database, but not for others, e.g. postgres).
If you want to unconditionally run a SQL script, By default Spring Boot will run one independent of Hibernate settings if you put it in classpath:schema.sql (or classpath:schema-<platform>.sql where <platform> is "postgres" in your case).
I think you can probably delete the JpaVendorAdapter and also the LocalContainerEntityManagerFactoryBean (unless you are using persistence.xml) and let Boot take control. The packages to scan can be set using an #EntityScan annotation (new in Spring Boot).
The default table naming scheme was changed in Boot 1.0.0.RC1 (so nothing to do with your postgres dependency). I'm not sure that will still be the case in RC2, but anyway you can go back to the old Hibernate defaults by setting spring.jpa.hibernate.naming-strategy=org.hibernate.cfg.ImprovedNamingStrategy.
Hey I came across similar issue. My sql script was not getting invoked initially. Then I tried renaming the file from "import.sql" to "schema.sql", it worked. May be give this a shot. My code can be found here - https://github.com/sidnan/spring-batch-example
In addition to what was already said, it's worth noting you can use the data.sql file to import/intialize data into your tables. Just put your data.sql into the root of the classpath (eg: if you're running a Spring Boot app, you put it in the src/main/resources path).
Like was said before, use it together with the property ddl-auto=create-drop, so that it won't crash trying to insert the existing data.
You can also set up which specific file to execute using the spring.datasource.data property. Check out more info here: http://docs.spring.io/spring-boot/docs/current/reference/html/howto-database-initialization.html
Note: the schema.sql mentioned before would contain the whole DB definition. If you want to use this, ensure that Hibernate doesn't try to construct the DB for you based on the Java Entities from your project. This is what de doc says:
If you want to use the schema.sql initialization in a JPA app (with
Hibernate) then ddl-auto=create-drop will lead to errors if Hibernate
tries to create the same tables. To avoid those errors set ddl-auto
explicitly to "" (preferable) or "none"