I'm trying to populate my database with around 150 different values (one for each row).
So far, I've found two different ways to implement the inserts, but none of them seems to be the best way to do it.
Flyway + Postgres: One of them is to create a migration file and make use of the COPY command from postgres but to do so, I need to give superuser permissions to the user and that doesn't seem to be a good choice.
Spring boot: place a data.sql file in the classpath with a lot of inserts. If I'm not wrong I would have to write 150 insert into... statements.
In previous projects, I have used liquibase and it has a loadData command which is very convenient to do what is says it does. You just give the file, table name and that's it. You end up with your csv file values in your table rows.
Is there an alike way to do that in flyway? What is the best way to populate the database?
Actually there is a way, you can find more info on the official documentation's page
You need to add some spring boot properties too:
spring.flyway.enabled=true
spring.flyway.locations=classpath:/db/migration
spring.flyway.schemas=public
Properties details here
In my case, a use Repetables scripts by my needs but take care with the prefixes
Flyway is a direct competitor of liquidbase, so if you need to track the status of migrations, manage distributed migration (many instances of the same service start simultaneously, and only one instance should actually execute a migration), check upon startup which migration should be applied and execute only relevant migrations, and all other benefits that you have previously expected from "migration management system", then you should use Flyway rather than managing SQLs directly.
Spring boot has integrations with both Flyway and Liquidbase, so you can place your migrations in the "resources" folder, define a couple of properties and spring boot will run Flyway automatically.
For example, here you can find a tutorial of Flyway integration with spring boot.
Since flyway's migrations are SQL files- you can place there whatever you want (even plSQL I believe), it will even manage transaction per migration guaranteeing that the migration "atomicity" (all or nothing, no partial migration).
So the straightforward approach would be creating a SQL file file with 150 inserts and running it via flyway in spring or even via maven depending on your actual setup.
If you want more fine-grained control and the SQL is not flexible enough, its possible to implement Migration in Java Code. See Official Flyway Documentation
Related
I have a SpringBoot application with in memory H2 database and Spring Data JPA.
I need to configure a #Scheduled job that drops and recreates the schema and loads it with fresh data from a file.
How can I programmatically recreate the schema in my application?
You can use database version control tool like eg Liquibase to create and maintain database schema definition as well as initial data. Than, you will be able to easily invoke database migration including drop of whole schema during applicaiton runtime. IT has some integration with Spring Boot already.
Keep in mind, that you will have to lock database access in order to execute migration - DDL is not transactional, so database will be of no use anyway during the migration process and you app can yeld many errors during that time.
If locking is not an option - you should be able to create another instance or at least separate schema in running instance, run migration against it and if everything is done, "switch" peristence context to use brand new schema (and probably remove the old one)
I have a situation where I have two SpringBoot microservices which share the same database schema. The schema is maintained by a liquibase changelog file. One service reads from the database, and the other service is responsible for writing to the database.
Right the Writing Service owns the liquibase changelog file, which means the Writing service owns the schema. And the way I validate the Reading Service is to deploy the Writing service first into a test environment followed by the Reading Service, and then execute end-to-end tests against the Reading Service.
Is there a way for both services (two separate apps, two separate repos) to share the liquibase changelog file? I feel this is similar to a contract test as the changelog file will be the contract for both services, but wasn't sure if there was something provided by Liquibase, Spring, Pact, etc that supported this idea.
Thanks for your time!
I think it won't count as a legit answer, but two solutions come to mind:
since your second service reads from the database, I suppose you have a full set of entities there, and entities are supposed to match your database schema. And since you're using Spring, I suppose you can add spring.jpa.hibernate.ddl-auto=validate to the application.properties file. This will validate your database schema against the entities.
You can create a separate library which will contain all the changeLogs. After that you can include this library to both services, so liquibase will validate that all changeSets are executed during application's deployment. But you should make sure that all your changeSets have preConditions, so your deployment won't fail and there won't be any duplicates in the DB schema.
I would like to use liquibase in my spring boot app. My requirement is that I have a dummy schema which is populated with tables every time I change the entity classes. This is done by hibernate's ddl create. There are many identical schemas to the dummy schema with data. I want those schemas to be compared with the dummy schema on update and be synced without affecting my data. How can I achieve this? I could not find a tutorial anywhere. If there is one please do give me the link.
I think this tutorial explains what your are looking for
baeldung maven liquibase plugin
In section 5.3 is a description on how you can get a changlog file with differences between two databases.
How can I insert data to a table on Spring Boot application start? My application is generated by JHipster. What I need to check is that if that particular data already exist in that table or not. If it doesn't I should add it.
If your application was generated by JHipster, then it should already include and be properly configured for Liquibase.
You have a few options:
You can use Liquibase's insert change to insert data.
You can put your data in a CSV file and use the loadData change to load it. Look for the loadData tag in 00000000000000_initial_schema.xml for examples.
You can use the sql change to run native SQL directly.
All three options can be combined with preconditions to make sure the data doesn't exist before trying to insert it. If you need the changeset to run every time you boot your application, you can use the runAlways="true" attribute (docs for that are on this page).
You can create a function in your service class which checks if the data already exists or not and if not then save that data.
You can implement an ApplicationRunner and use a repository to do whatever you need to do.
If the ApplicationRunner is a spring bean it is run on application startup.
For more sophisticated requirements I would try to rely on a tool like flyway that has good integration with spring boot
I have an application pointing to a mysql database.
I have been trying to use DBUnit to set up my tests environments, which works fine.
The problem is that when configuring DBUnit I pointed it to the SAME mysql database. So when DBUnit is executed, it takes the specified dataset.xml and overrides the information from my original database. which makes sense because there is where I am pointing it to.
The question is, am I supposed to create a new database only for tests so my DBUnit can point to it? If so, how would I manage the structure synchronization between my original database and the one for tests?
Thanks in advance.
am I supposed to create a new database only for tests so my DBUnit can point to it?
It is a better approach to do so as it eliminates multiple problems.
how would I manage the structure synchronization between my original database and the one for tests?
You don't mention tech in your persistence stack, such as Hibernate, Spring/Spring Boot/Spring Data/Flyway/LiquiBase/etc. to suggest more of how to implement this. In general, run DDL in the schema at tests run startup (either from managed DDL from something like Flyway or auto-generation from Hibernate).
Additionally, my preferred and typical testing approach is with:
An in-memory/embedded database for its speed, such as Apache Derby, automatically started just before launching tests.
Create tables in schema using Hibernate DDL gen from annotated entities.
No existing rows in any tables; happens automatically with an embedded database and a clean build when storing any of its files in a subdirectory of the build output dir.
dbUnit configured with DatabaseOperation.CLEAN_INSERT [0].
Minimal dbUnit data for each test.