I'm building a Spring Boot app that watches a folder in a file system. When a new file shows up, the file should be parsed and saved to a database. The file will be processed line by line and each line will be saved as a new record. There is a scenario when I should FIRST delete all records from the table and then start to insert them. I was recommended to use Spring Integration. Is this a good fit for this? If so, what is the best way to achieve the scenario with "delete all"?
Add a bean that implements SmartLifecycle then; in the start() method use a JdbcTemplate.execute() to truncate the table.
Related
I have a working Spring Boot application which embeds a Spring Batch Job. The job is not run on a schedule, instead we kick it with an endpoint. It is working as it should. The basics of the batch are
Kick the endpoint to start the job
Reader reads from input file
Processor reads from oracle database using jpa repository and simple spring datasource config
Writer writes to output file
However there are new requirements:
The schema of the repository database is from here on unknown on application startup. The tables are the same, it is just an unknown schema. This fact is out of our control and you might think it is stupid but there are reasons for it and this cant be changed. This means that with current functionality we need to reconfigure the datasource when we know the new schema name, and restart the application. This is a job that we will run for a number of times when migrating from one system to another, so it has a limited lifecycle and we just need a "quick fix" to be able to use it without rewriting the whole app. So what I would like to do is:
Send the schema name as a query param to the application, put it in job parameters and then - get a new datasource when the processor reads from the repository. Would this be doable at all using Spring Batch? Any help appreciated!
My application is using spring batch , in the first run the tables of the batch were created in the database , but in the other attempts to run it is trying to recreate the existing batch tables which causes me errors .
Is there any way to prevent spring batch from recreating these tables ?
Spring Batch does not create any table, you should be talking about Spring Boot. There is a way to prevent boot from creating the tables using spring.batch.initialize-schema=never, but there is no way to prevent re-creating them. Ideally, the scripts should include "if does not exist" clause (there is an open issue for that here: https://github.com/spring-projects/spring-batch/issues/1855), but this is not the case.
My spring container is started with the property auto-ddl=update mode and the store that I am using postgreSQl gets updated as per the changes in my container every time I starts the container.
For example if I add an attribute to my jpa entity, my container will update my data base with new column corresponding to my new attribute.
My requirement is that I need this data base updates to be flushed into a file so that I can use it at a later point of time.
I am aware of the hibernate property javax.persistence.schema-generation.scripts.action which defines which scripts the persistence provider shall create.(none,create,drop-and-create,drop).
The above property will not give me a patch script but it gives full ddl scripts.
The thing is that the spring container along with hibernate is able to update my data base upon container start (I am able to see the alter scripts in my log file when show-sql=true is provided.) Hence this should be possible.
How can I generate the patch scripts ?
I'm trying to populate my database with around 150 different values (one for each row).
So far, I've found two different ways to implement the inserts, but none of them seems to be the best way to do it.
Flyway + Postgres: One of them is to create a migration file and make use of the COPY command from postgres but to do so, I need to give superuser permissions to the user and that doesn't seem to be a good choice.
Spring boot: place a data.sql file in the classpath with a lot of inserts. If I'm not wrong I would have to write 150 insert into... statements.
In previous projects, I have used liquibase and it has a loadData command which is very convenient to do what is says it does. You just give the file, table name and that's it. You end up with your csv file values in your table rows.
Is there an alike way to do that in flyway? What is the best way to populate the database?
Actually there is a way, you can find more info on the official documentation's page
You need to add some spring boot properties too:
spring.flyway.enabled=true
spring.flyway.locations=classpath:/db/migration
spring.flyway.schemas=public
Properties details here
In my case, a use Repetables scripts by my needs but take care with the prefixes
Flyway is a direct competitor of liquidbase, so if you need to track the status of migrations, manage distributed migration (many instances of the same service start simultaneously, and only one instance should actually execute a migration), check upon startup which migration should be applied and execute only relevant migrations, and all other benefits that you have previously expected from "migration management system", then you should use Flyway rather than managing SQLs directly.
Spring boot has integrations with both Flyway and Liquidbase, so you can place your migrations in the "resources" folder, define a couple of properties and spring boot will run Flyway automatically.
For example, here you can find a tutorial of Flyway integration with spring boot.
Since flyway's migrations are SQL files- you can place there whatever you want (even plSQL I believe), it will even manage transaction per migration guaranteeing that the migration "atomicity" (all or nothing, no partial migration).
So the straightforward approach would be creating a SQL file file with 150 inserts and running it via flyway in spring or even via maven depending on your actual setup.
If you want more fine-grained control and the SQL is not flexible enough, its possible to implement Migration in Java Code. See Official Flyway Documentation
How can I insert data to a table on Spring Boot application start? My application is generated by JHipster. What I need to check is that if that particular data already exist in that table or not. If it doesn't I should add it.
If your application was generated by JHipster, then it should already include and be properly configured for Liquibase.
You have a few options:
You can use Liquibase's insert change to insert data.
You can put your data in a CSV file and use the loadData change to load it. Look for the loadData tag in 00000000000000_initial_schema.xml for examples.
You can use the sql change to run native SQL directly.
All three options can be combined with preconditions to make sure the data doesn't exist before trying to insert it. If you need the changeset to run every time you boot your application, you can use the runAlways="true" attribute (docs for that are on this page).
You can create a function in your service class which checks if the data already exists or not and if not then save that data.
You can implement an ApplicationRunner and use a repository to do whatever you need to do.
If the ApplicationRunner is a spring bean it is run on application startup.
For more sophisticated requirements I would try to rely on a tool like flyway that has good integration with spring boot