I want to read a txt file which has oracle insert statements in java program and execute as a batch, If any of the record gets failed in the batch, How do i get to know the particular insert statement only got failed.
for Ex.if i have 1000 records in file and 659 record gets failed in insertion,
how can i get to know the 659 record got failed in executebatch update.Is there any other process to do this please let me know.
There are two ways to handle the above situation either you check-in Java code or write the procedure
Java Code: this will either load everything or roll back the whole batch.
updateResult = preparedStatement.executeBatch();
checkSuccessfulUpdates(updateResult);
public static void checkSuccessfulUpdates(int[] updateCounts) throws Exception {
for (int i :updateCounts) {
if (i == Statement.EXECUTE_FAILED) {
throw new Exception("Not all rows were updated. Rolling back entire batch ");
}
}
}
Write a SQL procedure
Send the entire data in custom object to procedure, load it 1 by 1 by record, handle the exception within the loop, on error load the failed record in a temporary table.
Related
I am creating a Spring Batch process (Spring Boot 2) that reads a file and writes it to a Database. It processes it one record at a time. Read from file, process it, and write (or update) to the Database.
If a record for the same ID exists in the DB, the process has to update the end date of the existing record in DB, and create a new record with new start date. Below is the code:
public class Processor implements ItemProcessor<CelebVO, CelebVO> {
#Autowired
EndorseTableRepository endorseTableRepository;
#Override
#Transactional
public CelebVO process(CelebVO celebVO) {
CelebEndorsement celebEndorsement = endorseTableRepository.findAllByCelebIDAndBrandID(celebVO.getCelebID(),celebVO.getBrandID());
if (celebEndorsement == null) {
CelebEndorsement newEndorsement = new CelebEndorsement(celebVO);
endorseTableRepository.save(newEndorsement);
} else {
celebEndorsement.setEndDate(celebVO.getEffDt.minusDays(1));
endorseTableRepository.save(celebEndorsement);
// create a new row with new start date
CelebEndorsement newEndorsement = new CelebEndorsement(celebVO);
newEndorsement.setStartDate(celebVO.getEffDt());
endorseTableRepository.save(newEndorsement);
}
return celebVO;
}
}
Below is the input txt file (CelebVO):
CelebID BrandID EffDt
J Lo Pepsi 2021-01-05
J Lo Pepsi 2021-05-30
Now, lets suppose we are starting with an empty EndorseTable. When the process picks up the file and reads the records, it will see there are no records for CelebID 'J Lo'. So it will insert a row to the DB.
Now, the process reads the second row and process it. It should see that there is already a record in the table for J Lo. So it should put an endDate to that records and then create a new record.
After this file is processed we should see two records in the table.
But that is not what is happening. Though I do a repository.save() for the first record, it is still not commited to the table. So when the process reads the second row, it doesn't find any rows in the table. It ends up writing only one record to the table.
I tried a repository.saveAndFlush(). That doesn't help.
My chunk size is 1
I tried removing #Transactional. But that breaks the code. So I kept it there.
The chunk-oriented processing model of Spring Batch commits a transaction per chunk, not per record. So in your case, if the insert and the update happen to be in the same chunk, the processor won't see the change of the previous record as the transaction is not committed yet at that point.
Adding #Transactional on your processor's method is incorrect, because the processor will already be executed within the scope of a transaction driven by Spring Batch. What you are trying to do would work if you set the commit interval to 1, but this would impact the performance of your step.
I had to modify the Entity class. I replaced
#ManyToOne(cascade = CascadeType.ALL)
with
#ManyToOne(cascade = {CascadeType.MERGE, CascadeType.DETACH})
and it worked.
I have a simple kafka consumer that collects events and based on the data in them inserts or updates a record in the database - table has a unique ID constraint on ID column and also in the entity field.
Everything works fine when the table is pre-populated and inserts happen every now and then. However when i truncate the table and send a couple thousand events with limited number of ID (i was doing 50 unique ID within 3k events) then events are processed simultaneously and the save() method randomly fails with Unique constraint violation exception. I debugged it and the outcome is pretty simple.
event1={id = 1 ... //somedata} gets picked up, service method saveOrUpdateRecord() looks for the record by ID=1, finds none, inserts a new record.
event2={id = 1 ... //somedata} gets picked up almost at the same time, service method saveOrUpdateRecord() looks for the record by ID=1, finds none (previous one is mid-inserting), tries to insert and fails with constraint violation exception - should find this record and merge it with the input from the event based on my conditions.
How can i get the saveOrUpdateRecord() to run only when the previous one was fully executed to prevent such behaviour? I really dont want to slow kafka consumer down with poll size etc, i just want my service to execute one transaction at a time.
The service method:
public void saveOrUpdateRecord(Object input) {
Object output = repository.findById(input.getId));
if (output == null) {
repository.save(input);
} else {
mergeRecord(input, output);
repository.save(output);
}
}
Will #Transactional annotaion on method do the job?
Make your service thread safe.
Use this:
public synchronized void saveOrUpdateRecord(Object input) {
Object output = repository.findById(input.getId));
if (output == null) {
repository.save(input);
} else {
mergeRecord(input, output);
repository.save(output);
}
}
I need to execute a procedure multiple times with the requestbody. For now, i am using for loop to do that, but it is not a good way. For example, when the one request is failed, catching that request is a problem. See for the example usage.
Implementation of stored procedure with for-loop;
**Controller**
public void runSP(
#RequestBody List<IdNoteModel> idNotes
){
getService().runSP(idNotes);
}
**Service**
public void runSP(List<IdNoteModel> idNotes){
for (IdNoteModel idNote : idNotes){
getRepository().runSP(idNote);
}
}
**Repository**
#Query(nativeQuery = true, value = "EXECUTE PROCEDURE SP_RUN_ID_NOTE(:id, :note)")
void runSP(Long id, String note);
Is there a better way to run stored procedures multiple times? or do you have an idea to catch the failed requests?
You can use one stored procedure in which you can use the cursor to execute the procedure (SP_RUN_ID_NOTE) multiple times. In that stored procedure, you can also use rollback for something to fail. In your above case, you can store request body input in the table and then call procedure while taking input row from that table.
You can use #Retryable for this purpose and also you can catch exceptions and log requests with #Recover
This question already has answers here:
Unable to edit db entries using EFCore, EntityState.Modified: "Database operation expected to affect 1 row(s) but actually affected 0 row(s)."
(17 answers)
Closed 2 years ago.
I am using Sql Server on Linux with EC Core (2.1.0-preview1-final)
I am trying to update some data coming from a web Api service (PUT) request. The data (ticket) is passed correctly and gets deserialised into an object (ticket).
When I try to update, I use the following code:
public Ticket UpdateTicket(Ticket ticket)
{
using (var ctx = new SupportTicketContext(_connectionString))
{
ctx.Entry(ticket).State = EntityState.Modified;
ctx.SaveChanges(); // <== **BLOWS UP HERE**
var result = ctx.Tickets
.First(t => t.TicketId == ticket.TicketId);
return result;
}
}
The code throws the following error:
Database operation expected to affect 1 row(s) but actually affected 0 row(s). Data
may have been modified or deleted since entities were loaded
I am able to Insert and fetch from the database, no problem.
If I restart Visual Studio, the error occurs usually on the second time I try to update ANY ticket (i.e. any other ticketId - it seems to be on the second and subsequent requests).
The updates are unpredictably successful! Sometimes I can update another ticket and it goes through (even on the 3rd or susequent request)
I have tried a number of modifications to the code, including
ctx.Attach(ticket);
but this does not help.
How do I get it to update the database successfully?
Any ideas on how to debug this? Logging seems to be difficult to set up.
Any ideas greatly appreciated.
There were two errors. First, I tried to call the Update() method without specifying a DbSet:
_applicationDbContext.Update(user);
Correct way:
_applicationDbContext.Users.Update(user);
The second error was trying to update a model that without first pulling it from the database:
public bool UpdateUser(IdentityUser model)
{
_applicationDbContext.Users.Update(model);
_applicationDbContext.SaveChanges();
return true;
}
Nope.
I first needed to retrieve it from the database, then update it:
public bool UpdateUser(IdentityUser model)
{
var user = _applicationDbContext.Users.FirstOrDefault(u => u.Id == model.Id);
user.PhoneNumberConfirmed = model.PhoneNumberConfirmed;
_applicationDbContext.Users.Update(user);
_applicationDbContext.SaveChanges();
return true;
}
Had such problem too, but the reason was a trigger I was using.
Changed the trigger type from "INSTEAD OF INSERT" to "AFTER INSERT" and instead of modifying and inserting the data I was allowing my command to insert the data to the table and then updated it using the trigger.
I created jsp web apps to perform 4 millions data insert. The insertion process using loop from another table, so i did select query -> loop -> insert to other table, but everytime i run this, the page load slowly an ended with timeout. So not all data was successfully inserted
I had tried to use bul data insert, it didnt hep me.
btw this is the sample code :
pstatement = connection.prepareStatement(insertquery);
pstatement.setString(1, request.getParameter("promo"));
while (rset.next()) {
pstatement.setString(1, rset.getString(1));
pstatement.setString(2, request.getParameter("promo"));
pstatement.addBatch();
out.print(rset.getString(1) + " Added<br>");
if (++countbatch % batchSize == 0) {
pstatement.executeBatch();
}
}
pstatement.executeBatch();
Another try using select trick, it also didnt help
String insertquery = "INSERT INTO datapin (msisdn,nama_promo) SELECT msisdnlist.msisdn AS msisdn, ? AS nama_promo FROM msisdnlist ";
pstatement = connection.prepareStatement(insertquery);
pstatement.setString(1, request.getParameter("promo"));
pstatement.executeQuery();
Anyone have better idea ?? Thanks :)
Is it required that the jsp page waits for the 4M inserts to complete?
I think waiting for a long lasting operation is always tricky in a web app.
You might start a thread in the request handler that will do the inserts async.
The request handler could return a token.
Then this token could be used by a second JSP page to periodically check the status of the thread.