I use Flyway in Spring project that has 2 databases: mysql and mongodb. I want to keep order of migrations to both data sources and be able to move data from one to other one (use mysql+mongo in one migration).
Is it ok to use Flyway java-based-migrations for mongodb migrations?
IMO it's completely fine to run MongoDB migrations as a flyway java migration, you only need to be sure that the migration is idempotent and prepare for possible failures, unlike SQL migrations Flyway won't perform a rollback if the transaction fails in the middle.
Another alternative would be Mongobee, it's like Flyway but for MongoDB. But it also relies on java based migrations.
Related
I have a SpringBoot application with in memory H2 database and Spring Data JPA.
I need to configure a #Scheduled job that drops and recreates the schema and loads it with fresh data from a file.
How can I programmatically recreate the schema in my application?
You can use database version control tool like eg Liquibase to create and maintain database schema definition as well as initial data. Than, you will be able to easily invoke database migration including drop of whole schema during applicaiton runtime. IT has some integration with Spring Boot already.
Keep in mind, that you will have to lock database access in order to execute migration - DDL is not transactional, so database will be of no use anyway during the migration process and you app can yeld many errors during that time.
If locking is not an option - you should be able to create another instance or at least separate schema in running instance, run migration against it and if everything is done, "switch" peristence context to use brand new schema (and probably remove the old one)
I'm trying to populate my database with around 150 different values (one for each row).
So far, I've found two different ways to implement the inserts, but none of them seems to be the best way to do it.
Flyway + Postgres: One of them is to create a migration file and make use of the COPY command from postgres but to do so, I need to give superuser permissions to the user and that doesn't seem to be a good choice.
Spring boot: place a data.sql file in the classpath with a lot of inserts. If I'm not wrong I would have to write 150 insert into... statements.
In previous projects, I have used liquibase and it has a loadData command which is very convenient to do what is says it does. You just give the file, table name and that's it. You end up with your csv file values in your table rows.
Is there an alike way to do that in flyway? What is the best way to populate the database?
Actually there is a way, you can find more info on the official documentation's page
You need to add some spring boot properties too:
spring.flyway.enabled=true
spring.flyway.locations=classpath:/db/migration
spring.flyway.schemas=public
Properties details here
In my case, a use Repetables scripts by my needs but take care with the prefixes
Flyway is a direct competitor of liquidbase, so if you need to track the status of migrations, manage distributed migration (many instances of the same service start simultaneously, and only one instance should actually execute a migration), check upon startup which migration should be applied and execute only relevant migrations, and all other benefits that you have previously expected from "migration management system", then you should use Flyway rather than managing SQLs directly.
Spring boot has integrations with both Flyway and Liquidbase, so you can place your migrations in the "resources" folder, define a couple of properties and spring boot will run Flyway automatically.
For example, here you can find a tutorial of Flyway integration with spring boot.
Since flyway's migrations are SQL files- you can place there whatever you want (even plSQL I believe), it will even manage transaction per migration guaranteeing that the migration "atomicity" (all or nothing, no partial migration).
So the straightforward approach would be creating a SQL file file with 150 inserts and running it via flyway in spring or even via maven depending on your actual setup.
If you want more fine-grained control and the SQL is not flexible enough, its possible to implement Migration in Java Code. See Official Flyway Documentation
I have a spring-boot application with PostgreSQL. Some of the tables are created using models and other tables have to be created prior to the start of the application or while starting the application. That can be done by running an SQL file while startup.
But DB tend to change over time, we may have to alter some of the tables, add some new tables without disturbing the existing data in the tables, etc.
Is there a way to add new SQL files, and run only the SQL files which was not run in spring-boot application each time when we rebuild and rerun? And don't run any of the SQL files while start-up if everything were already executed?
For your scenario liquibase is the best solution. You can merge liquibase on your spring boot application.
Ex: https://javadeveloperzone.com/spring-boot/spring-boot-liquibase-example/
You can use flyway. It allows you to have versioned sql scripts:
flywaydb.org
flywaydb spring boot plugin
Examples:
Spring Boot Database Migrations with Flyway
Incrementally changing your db with java and flyway
I'm working on a spring boot application that uses Camunda
I've just bumped our Camunda version by a couple of minor versions, and I'm running incremental SQL scripts with Flyway to update the DB to support the new camunda version (v1.02--7.8-7.9.sql and v1.03--7.9>7.10.sql)
This migration works fine on existing databases that are currently on Camunda 7.8, but it fails when I try to run the application on a clean database.
One of the scripts tries to add a column:
ALTER TABLE ACT_ID_USER
ADD LOCK_EXP_TIME_ timestamp;
But, if the application is running on a clean database, it will complain that the column is already present - presumably because the database has just been initialised with 9.10, so running a 9.8 migration after the fact doesn't work
I'm wondering if there's a way for Flyway to not run those migrations, if the database is empty on startup?
Or, is my problem the way I'm using Flyway? Should flyway be getting executed before Camunda initialises the Database?
I was using flyway through the CL to migrate my production DB (mySql), while I was using a fixed SQL query to create the DB, tables, etc. in my unit tests, using H2. I'd like now to better integrate flyway and create/delete DB after each unit test.
I have a DB factory and inside its build method I'm using the following code:
flyway.setLocations("filesystem:sql/migrations/common","filesystem:sql/migrations/h2");
flyway.setSchemas("MYSERVER");
flyway.setDataSource(
p.getProperty(DB_URL.getName()),
p.getProperty(USERNAME.getName()),
p.getProperty(PASSWORD.getName()));
flyway.setInitOnMigrate(true);
flyway.migrate();
The migrations seems to apply correctly, as I can see my SQL code from the flyway logs. But when I start using the DB, immediately afterwards, I get TABLE NOT FOUND errors. I'm using h2 as in memory DB with the following URL to initialize my client:
jdbc:h2:mem:MYSERVER;MVCC=true
Do you have any idea on what I might be making wrong?
The problem is your JDBC url.
MYSERVER is the name of the database, not the schema.
The easiest thing to do is to let flyway use the same url, and not set a schema. This way you'll find everything in the public schema of the MYSERVER database.
Having the same issue - after digging around, it appears that the H2 database is being recreated between migration scripts somehow.
You can confirm this happens by putting a SELECT * FROM migrations in the first two migrations - it will succeed in the first and fail in the second.
According to the H2 Documentation, specifying a database name and ;DB_CLOSE_DELAY=-1 option should suffice to allow concurrent and subsequent accesses to the in-memory database, however this is not the case.
Upgrading Flyway from 3.1 to 3.2.1 has fixed the problem.