I have a requirement to update database records of particular table in bulk size. is there any way to update record instead of inserting new one through spring batch ? or any other way ?
Related
I am working with a Spring boot project and Oracle Database. My task is importing data from a csv file. The file includes 10000 rows; each row is equal to a row in a table and includes three attributes which are sCode, isDeleted (isDeleted=1 -> delete, isDeleted=0 -> update or insert). I am using batch size of 1000 to insert, update or delete. Each batch, I do the following steps:
Use JPA findAllBySCodeInAndDepartmentId to file all rows that exist in the table (departmentId is the department of current user, (sCode, departmentId) is unique)
Use Map to store data from the first step's result (key is sCode)
Filter out rows in batch that have isDeleted = 1 and exist in the Map to delete (I use JPA deleteAll)
Filter out rows that isDeleted = 0 and exist in the Map to update (I use JPA saveAll)
Left rows in batch which were not updated or deleted (not in the Map) will be inserted into the table (I use JPA saveAll)
It takes around 5 minutes to finish importing 10000 rows
How can I shrink the amount of time to around 1 minute?
i m using spring data saveAll to save 3500 records in an Oracle database but it execute very slowly, is there a way to do bulk insert or any other fast way
noteRepository.saveAll(noteEntityList);//<- this one is slow for 3000 records
thanks in advance
By default, saveAll does not create batch, the batch processing needs to be enabled.
You need to set below properties to enable batch processing
spring.jpa.properties.hibernate.jdbc.batch_size=100
spring.jpa.properties.hibernate.order_inserts=true (if inserts)
OR
spring.jpa.properties.hibernate.order_updates=true (if updates)
First property collects the transaction in batch and second property collects the statements grouped by entity.
Check this thread for more details
How to do bulk (multi row) inserts with JpaRepository?
Also, if you want to do batch inserts, make sure that if your table has an auto-incremented column (say as a PK), that its set up as a Sequence (not Identity) and that the allocationSize (Java) and increment_by value (DB Sequence) are set to the batch size you are trying to persist. Don't set those values to one, else insert will still be slow as JPA will need to keep going back to the DB to get the next value from the sequence.
I'm looking at a spring boot application that is used to copy data from temp to permanent table based on last updated date. It copies only if last updated date is greater than desired date, so not all records are copied over. Currently the table has around 300K+ records and the process with spring JPA is taking over 2 hours (for all of them) and is not at all feasible. The goal is to bring it down to under 15 mins maximum. I'm trying to see how much difference using JDBCtemplate would bring in. Would a pl/sql script be a better option? Also wanted to see if there are better options out there. Appreciate your time.
Using oracle database at the moment but postgresql migration is on the cards.
Thanks!
You can do this operation with a straight SQL query (which will work on Oracle or PostgreSQL). Assuming your temp_table has the same columns as the permanent table, the last updated date column is called last_updated and you wanted to copy all records updated since 2020-05-03 you could write a query like:
INSERT INTO perm_table
SELECT *
FROM temp_table
WHERE last_updated > TO_DATE('2020-05-03', 'YYYY-MM-DD')
In your app you would pass '2020-05-03' via a placeholder either directly or via JdbcTemplate.
While development I was in need to create sqlite data base using dataset.
First create empty database file in sqlite. Then using dataset create table and dump data.
Is it possible using some dataset method?
I have done one thing
I have used one query
select * from sqlite_master where type='table'
That will give me schema of sqllite file from where I want to copy data.
On other end I will execute query return by above query.
That will help to create table dynamically.
On by one I used dataadapter to fill data and assigned dataset datatable data and used
Dataadapter. Update method.
I'm new to both Oracle and Informatica.
Currently working on a small task where I need to select all records from the source table, filter the results to get only records where field1='Y' and finally insert new rows into the target table that contains only src.field2 and src.field3 values.
These 2 fields are used for the PK and for the Index of the target table.
So i get an error in Informatica:
"ORA-26002: Table has index defined upon it"
I rather not dropping the index? is there a work around?
I've tried alter index to "unusable" but I got the same error.
Please advice.
Thanks.
Try to use Normal load mode instead of Bulk. You can set in session properties for the target.