I have a SpringBoot application with in memory H2 database and Spring Data JPA.
I need to configure a #Scheduled job that drops and recreates the schema and loads it with fresh data from a file.
How can I programmatically recreate the schema in my application?
You can use database version control tool like eg Liquibase to create and maintain database schema definition as well as initial data. Than, you will be able to easily invoke database migration including drop of whole schema during applicaiton runtime. IT has some integration with Spring Boot already.
Keep in mind, that you will have to lock database access in order to execute migration - DDL is not transactional, so database will be of no use anyway during the migration process and you app can yeld many errors during that time.
If locking is not an option - you should be able to create another instance or at least separate schema in running instance, run migration against it and if everything is done, "switch" peristence context to use brand new schema (and probably remove the old one)
Related
I am trying to create a small Spring Boot setup with a h2 database.
Now I have a weird problem, which I can't solve.
If I don't create an data.sql for initial data, the app starts fine and creates my entity tables.
If I create an data.sql for initial data and keep the existing table from previous step, everything works fine.
If I create an data.sql for initial data and remove my existing h2 file, I get the error that it can't import the data, because the table is missing.
How do I tell Spring to create my tables before importing the initial data?
This is covered in the release notes for Spring Boot 2.5:
By default, data.sql scripts are now run before Hibernate is initialized. This aligns the behavior of basic script-based initialization with that of Flyway and Liquibase. If you want to use data.sql to populate a schema created by Hibernate, set spring.jpa.defer-datasource-initialization to true. While mixing database initialization technologies is not recommended, this will also allow you to use a schema.sql script to build upon a Hibernate-created schema before it’s populated via data.sql.
I'm trying to populate my database with around 150 different values (one for each row).
So far, I've found two different ways to implement the inserts, but none of them seems to be the best way to do it.
Flyway + Postgres: One of them is to create a migration file and make use of the COPY command from postgres but to do so, I need to give superuser permissions to the user and that doesn't seem to be a good choice.
Spring boot: place a data.sql file in the classpath with a lot of inserts. If I'm not wrong I would have to write 150 insert into... statements.
In previous projects, I have used liquibase and it has a loadData command which is very convenient to do what is says it does. You just give the file, table name and that's it. You end up with your csv file values in your table rows.
Is there an alike way to do that in flyway? What is the best way to populate the database?
Actually there is a way, you can find more info on the official documentation's page
You need to add some spring boot properties too:
spring.flyway.enabled=true
spring.flyway.locations=classpath:/db/migration
spring.flyway.schemas=public
Properties details here
In my case, a use Repetables scripts by my needs but take care with the prefixes
Flyway is a direct competitor of liquidbase, so if you need to track the status of migrations, manage distributed migration (many instances of the same service start simultaneously, and only one instance should actually execute a migration), check upon startup which migration should be applied and execute only relevant migrations, and all other benefits that you have previously expected from "migration management system", then you should use Flyway rather than managing SQLs directly.
Spring boot has integrations with both Flyway and Liquidbase, so you can place your migrations in the "resources" folder, define a couple of properties and spring boot will run Flyway automatically.
For example, here you can find a tutorial of Flyway integration with spring boot.
Since flyway's migrations are SQL files- you can place there whatever you want (even plSQL I believe), it will even manage transaction per migration guaranteeing that the migration "atomicity" (all or nothing, no partial migration).
So the straightforward approach would be creating a SQL file file with 150 inserts and running it via flyway in spring or even via maven depending on your actual setup.
If you want more fine-grained control and the SQL is not flexible enough, its possible to implement Migration in Java Code. See Official Flyway Documentation
I have a spring-boot application with PostgreSQL. Some of the tables are created using models and other tables have to be created prior to the start of the application or while starting the application. That can be done by running an SQL file while startup.
But DB tend to change over time, we may have to alter some of the tables, add some new tables without disturbing the existing data in the tables, etc.
Is there a way to add new SQL files, and run only the SQL files which was not run in spring-boot application each time when we rebuild and rerun? And don't run any of the SQL files while start-up if everything were already executed?
For your scenario liquibase is the best solution. You can merge liquibase on your spring boot application.
Ex: https://javadeveloperzone.com/spring-boot/spring-boot-liquibase-example/
You can use flyway. It allows you to have versioned sql scripts:
flywaydb.org
flywaydb spring boot plugin
Examples:
Spring Boot Database Migrations with Flyway
Incrementally changing your db with java and flyway
For the database schema management with spring data/hibernate, setting spring.jpa.hibernate.ddl-auto option doesn't look like a cleaner approach.
Bcoz
1) We are forced to put the credentials of a user that has permission to create and delete in the application.properties.
2) In production, relaying on spring.jpa.hibernate.ddl-auto option could lead to dataloss, if not managed carefully.
So what is the best way out there to handle the database schema management automatically in a spring boot/spring data app?
Any pointers would help.
Thanks
If you want to track each change state of database then you can use flyway
See this link how to maintain database versioning in spring-boot
In production, you should ideally set spring.jpa.hibernate.ddl-auto property as none so that no schema changes are allowed.
For the database schema management with spring data/hibernate, I would suggest you go for Liquibase, it basically is an open source database-independent library for tracking, managing and applying database schema changes.
Every change to Schema is added as a changeset using property file in Liquibase , this is for the new changes.
In order to migrate the existing database structure into Liquibase, it provides you with commands to automatically generate Changesets by reading the current database.
So using this you can generate database schema, add constraints, load data.
More info at : https://www.liquibase.org/ , Why and when Liquibase?
I have an application pointing to a mysql database.
I have been trying to use DBUnit to set up my tests environments, which works fine.
The problem is that when configuring DBUnit I pointed it to the SAME mysql database. So when DBUnit is executed, it takes the specified dataset.xml and overrides the information from my original database. which makes sense because there is where I am pointing it to.
The question is, am I supposed to create a new database only for tests so my DBUnit can point to it? If so, how would I manage the structure synchronization between my original database and the one for tests?
Thanks in advance.
am I supposed to create a new database only for tests so my DBUnit can point to it?
It is a better approach to do so as it eliminates multiple problems.
how would I manage the structure synchronization between my original database and the one for tests?
You don't mention tech in your persistence stack, such as Hibernate, Spring/Spring Boot/Spring Data/Flyway/LiquiBase/etc. to suggest more of how to implement this. In general, run DDL in the schema at tests run startup (either from managed DDL from something like Flyway or auto-generation from Hibernate).
Additionally, my preferred and typical testing approach is with:
An in-memory/embedded database for its speed, such as Apache Derby, automatically started just before launching tests.
Create tables in schema using Hibernate DDL gen from annotated entities.
No existing rows in any tables; happens automatically with an embedded database and a clean build when storing any of its files in a subdirectory of the build output dir.
dbUnit configured with DatabaseOperation.CLEAN_INSERT [0].
Minimal dbUnit data for each test.