For my upcoming microservice deployments I intend to use spring-boot and Flyway as the database migration tool. For security reasons I need to draw a line between the DDL schema creation parts (create, alter, drop table etc.) and accessing the DML data (select, insert, update, delete) stored inside the Oracle db.
I was thinking of using two different datasources, the first used solely for Flyway operations and the second for the real application.
I havent't found a best practise implementation to solve this problem yet. Any advice or link would be very helpful.
Related
I have a use case where I need to create exact same postgresql database in two different regions. Everything is same in these two databases i.e same schema and same tables and same data.
I have a use to achieve distributed transaction. So if a request land in region-a and write to region-a database to let's say Person table, then exact same record must be either written in Person table in both these database or if there is any error, write attempt should be rolled back.
I am trying to figure out if I can attach two different datasources with same Person Entity and CRUD repository in spring so the respoistory.save() method can write to Person table in both the databases.
So far, I have come across AbstractRoutingDataSource but that is for achieving multi tenancy in the databases. Other solutions are found are slightly different where use case is to write different records in different database (mostly sharding based on various data points).
Does spring provide any out of the box solution so I can achieve transactional write to same table in two different databases.
Does spring provide any out of the box solution so I can achieve transactional write to same table in two different databases.
Depends on your definition of "out of the box" - it doesn't itself implement distributed transactions, but does have support for using libraries that do. It is however relatively complicated to get everything working correctly, and requires additional components to be carefully configured in your runtime environment.
Spring Boot 2.x documentation on distributed transactions is here: https://docs.spring.io/spring-boot/docs/2.7.x/reference/htmlsingle/#io.jta
The Spring Boot 3.x documentation is here: https://docs.spring.io/spring-boot/docs/current/reference/html/io.html#io.jta but it's also worth noting that for 3.x, the Spring Boot team have changed direction and decided that integrated support should be provided by the relevant JTA provider (cf. https://github.com/spring-projects/spring-boot/issues/28589 ), and so there's projects like https://github.com/snowdrop/narayana-spring-boot
I have a SpringBoot application with in memory H2 database and Spring Data JPA.
I need to configure a #Scheduled job that drops and recreates the schema and loads it with fresh data from a file.
How can I programmatically recreate the schema in my application?
You can use database version control tool like eg Liquibase to create and maintain database schema definition as well as initial data. Than, you will be able to easily invoke database migration including drop of whole schema during applicaiton runtime. IT has some integration with Spring Boot already.
Keep in mind, that you will have to lock database access in order to execute migration - DDL is not transactional, so database will be of no use anyway during the migration process and you app can yeld many errors during that time.
If locking is not an option - you should be able to create another instance or at least separate schema in running instance, run migration against it and if everything is done, "switch" peristence context to use brand new schema (and probably remove the old one)
I'm trying to populate my database with around 150 different values (one for each row).
So far, I've found two different ways to implement the inserts, but none of them seems to be the best way to do it.
Flyway + Postgres: One of them is to create a migration file and make use of the COPY command from postgres but to do so, I need to give superuser permissions to the user and that doesn't seem to be a good choice.
Spring boot: place a data.sql file in the classpath with a lot of inserts. If I'm not wrong I would have to write 150 insert into... statements.
In previous projects, I have used liquibase and it has a loadData command which is very convenient to do what is says it does. You just give the file, table name and that's it. You end up with your csv file values in your table rows.
Is there an alike way to do that in flyway? What is the best way to populate the database?
Actually there is a way, you can find more info on the official documentation's page
You need to add some spring boot properties too:
spring.flyway.enabled=true
spring.flyway.locations=classpath:/db/migration
spring.flyway.schemas=public
Properties details here
In my case, a use Repetables scripts by my needs but take care with the prefixes
Flyway is a direct competitor of liquidbase, so if you need to track the status of migrations, manage distributed migration (many instances of the same service start simultaneously, and only one instance should actually execute a migration), check upon startup which migration should be applied and execute only relevant migrations, and all other benefits that you have previously expected from "migration management system", then you should use Flyway rather than managing SQLs directly.
Spring boot has integrations with both Flyway and Liquidbase, so you can place your migrations in the "resources" folder, define a couple of properties and spring boot will run Flyway automatically.
For example, here you can find a tutorial of Flyway integration with spring boot.
Since flyway's migrations are SQL files- you can place there whatever you want (even plSQL I believe), it will even manage transaction per migration guaranteeing that the migration "atomicity" (all or nothing, no partial migration).
So the straightforward approach would be creating a SQL file file with 150 inserts and running it via flyway in spring or even via maven depending on your actual setup.
If you want more fine-grained control and the SQL is not flexible enough, its possible to implement Migration in Java Code. See Official Flyway Documentation
I'm really struggling to understand how I can use flyway to manage an oracle database.
The database in question has 3 schemas.
If I stipulate the 3 schemas in the flyway plugin definition in my gradle file, how do I manage the creation of the users themselves, and the tablespaces they use?
Any tips or suggestions would be more than welcome.
Creating a DDL for a baseline for flyway
If You would create the DB with some limited super-user account, You could use it to create users and tablespaces, having a separate Flyway migration instance. That is:
Flyway migration by super-user from sql/db-configuration/ using dedicated flyway.conf
creates users, tablespaces, and all other configuration
Flyway migration by application-user from sql/app-code/ using dedicated flyway.conf
regular application code
I was working on my Spring boot app project and noticed that, sometimes there is a connection time out error to my Database on another server(SQL Server).
This happens specially when I try to do some script migration with FlyWay but it works after several tries.
Then I noticed that I didn't specify spring.jpa.hibernate.ddl-auto in my properties file. I did some research and found that it is recommended to add
spring.jpa.hibernate.ddl-auto= create-drop in development.
And change it to: spring.jpa.hibernate.ddl-auto= none in production.
But I didn't actually understand how does it really work and how does hibernate generate database schema using create-drop or none value. Can you please explain technically how does it really work, and what are recommendations for using this property in development and on a production server.
Thank you
For the record, the spring.jpa.hibernate.ddl-auto property is Spring Data JPA specific and is their way to specify a value that will eventually be passed to Hibernate under the property it knows, hibernate.hbm2ddl.auto.
The values create, create-drop, validate, and update basically influence how the schema tool management will manipulate the database schema at startup.
For example, the update operation will query the JDBC driver's API to get the database metadata and then Hibernate compares the object model it creates based on reading your annotated classes or HBM XML mappings and will attempt to adjust the schema on-the-fly.
The update operation for example will attempt to add new columns, constraints, etc but will never remove a column or constraint that may have existed previously but no longer does as part of the object model from a prior run.
Typically in test case scenarios, you'll likely use create-drop so that you create your schema, your test case adds some mock data, you run your tests, and then during the test case cleanup, the schema objects are dropped, leaving an empty database.
In development, it's often common to see developers use update to automatically modify the schema to add new additions upon restart. But again understand, this does not remove a column or constraint that may exist from previous executions that is no longer necessary.
In production, it's often highly recommended you use none or simply don't specify this property. That is because it's common practice for DBAs to review migration scripts for database changes, particularly if your database is shared across multiple services and applications.
In Spring/Spring-Boot, SQL database can be initialized in different ways depending on what your stack is.
JPA has features for DDL generation, and these can be set up to run on startup against the database. This is controlled through two external properties:
spring.jpa.generate-ddl (boolean) switches the feature on and off and is vendor independent.
spring.jpa.hibernate.ddl-auto (enum) is a Hibernate feature that controls the behavior in a more fine-grained way. See below for more detail.
Hibernate property values are: create, update, create-drop, validate and none:
create – Hibernate first drops existing tables, then creates new tables
update – the object model created based on the mappings (annotations or XML) is compared with the existing schema, and then Hibernate updates the schema according to the diff. It never deletes the existing tables or columns even if they are no more required by the application
create-drop – similar to create, with the addition that Hibernate will drop the database after all operations are completed. Typically used for unit testing
validate – Hibernate only validates whether the tables and columns exist, otherwise it throws an exception
none – this value effectively turns off the DDL generation
Spring Boot internally defaults this parameter value to create-drop if no schema manager has been detected, otherwise none for all other cases.
"spring.jpa.hibernate.ddl-auto= create-drop" means that when the server is run, the database(table) instance is created. And whenever the server stops, the database table instance is droped.
Also depending on spring.jpa.hibernate.ddl-auto the DML files feature is enabled
DDL and DML
It is worth to understand the difference between them.
Data Definition Language(DDL) - related to database schema creating
Data Manipulation Language(DML) - related to importing data
Basically there are 3 types of database schema creating(DDL) and importing data(DML):
Using Hibernate
Using Spring JDBC SQL scripts
Using high level tools like Flyway/Liquibase
This topic covers Hibernate and it's DDL (first option), but it is worth to mention Hibernate DML files feature that enabled if spring.jpa.hibernate.ddl-auto is create or create-drop
That means import.sql in the root of the classpath will be executed on startup by Hibernate. This can be useful for demos and for testing if you are careful, but probably not something you want to be on the classpath in production. It is a Hibernate feature (nothing to do with Spring).
Also here is a table that explains spring.jpa.hibernate.ddl-auto and whether the import.sql can be used depending on spring.jpa.hibernate.ddl-auto value specified:
spring.jpa.hibernate.ddl-auto
Create schema from entities
import.sql
create
true
true
update
update schema from entities
false
create-drop
true
true
validate
false
false
none
false
false
Also some extra information about different types of DDL amd DML can be found in Spring docs
For the Propertie of JPA/Hibernate
spring.jpa.hibernate.ddl-auto value should be create, update, create-drop not other then it will give an exception, where the correct meaning for these value -
Create : when the server will start all entity will be newly created
Update : when the server will start container will find which entities are update and which all are newly created the same thing will happen inside database as well old table will update as per the entity and newly table will created
Create-drop: when the server will start then auto all entity will crete and when the server will stop all the entities will auto remove from database
none : it means database ddl will not impact from back-end application Note: Production environment always set with none value