crudrepository save takes too long to update an entity - spring

I've got a controller that first selects all data with status = 'CREATED' transferType = 'SOME_TYPE' and DATE_TIME between x and y, and then put all the data in the List<TransferEntity>
then i am going through each element in the list and updating status to 'CHECKED'
if (listOfTransfers.isNotEmpty()){
for(element in listOfTransfers){
element.status = "CHECKED"
repos.transfers.save(element)
}
}
entity itself is pretty straight forward with no relations to other tables
#Entity
#Table( name = "TRANSFERS")
class TransferEntity(
#Id
#Column(name = "Identifier", nullable = false)
var Identifier: String? = null,
#Column(name = "TRANS_DATE_TIME")
var transDateTime: LocalDateTime? = null,
#Column(name = "TRANS_TYPE", nullable = true, length = 255)
var transType: String? = null,
#Column(name = "STATUS")
var status: String = ""
)
i tried to experiment with indexes (oracle)
`CREATE INDEX TRANS_INDEX_1 ON TRANSFERS(STATUS)`
`CREATE INDEX TRANS_INDEX_2 ON TRANSFERS(TRANS_DATE_TIME)`
`CREATE INDEX TRANS_INDEX_3 ON TRANSFERS(TRANS_TYPE)`
or created them as one index
CREATE INDEX TRANS_INDEX_4 ON TRANSFERS(STATUS,TRANS_DATE_TIME,TRANS_TYPE)
but it wasnt a big difference
UPDATE
witn TRANS_INDEX_1 2 and 3 - 3192 elements were updateind in 5 minutes 30 sec
with TRANS_INDEX_4 - 3192 elements were updated in 5 minutes 30 sec
maybe there are different approaches to mass update elements inside the list or perhaps indexes are completely wrong and i dont understand them as much as i want it to.
UPDATE 2
technically saveAll() method works much faster but still I think there should be a room for improvement
saveAll() - 3192 elements were saved under 3minutes 21seconds
save() 3192 elements were save under 5minutes 30 seconds

You call save() each time you update an element. 1000 elements will create 1000 query calls to the database, you repeat too many calls to your DB and that's why your function is slow.
Instead, you could use saveAll() after you updated all the elements
as suggested below, we also have to config the batch_size properly to really do the trick
Indexes won't help in this situation since they benefit the select operation more than update or insert

Since you set the same value to all the elements of your list, you can make a batch update query :
Query q = entityManager.createQuery("update TransferEntity t set t.status = :value where t in (:list)");
q.setParameter("value", "CHECKED");
q.setParamter("list", listOfTransfers);
q.execute();
If you use ORACLE as backend be aware that in clause is limited to 1000 elements. Therefore you might have to split your list in buckets of 1000 elements and loop on this query for each bucket.

Related

Find newest Revisions of Entities where timestamp is less than x

I've made a painting to illustrate my desired result:
painting of the result
I have a max timestamp that i pass to my backend and i want to return the newest versions of the Entities that have a timestamp that is less than the passed one.
The passed timestamp in my example is 180. I've kept the numbers low so that it is easier to visualize.
This is a combination of a horizontal and a vertical Query and i'm not sure how i can achieve that.
Any help would be appreciated!
It can be achieved with the following:
public List<Book> getBooksByMaxTimestamp(Date date) {
String formattedDate = simpleDateFormat.format(date);
System.out.println("Less than " + formattedDate);
AuditQuery maxRevisionNumberQuery = auditReader.createQuery().forRevisionsOfEntity(Book.class, true, false);
maxRevisionNumberQuery.add(AuditEntity.revisionProperty("timestamp").le(date.getTime()));
maxRevisionNumberQuery.addProjection(AuditEntity.revisionNumber().max());
int maxRevisionNumber = (int) maxRevisionNumberQuery.getSingleResult();
AuditQuery auditQuery = auditReader.createQuery().forEntitiesAtRevision(Book.class, maxRevisionNumber);
return auditQuery.getResultList();
}
First we get the max revision number that is in that range and after that we use that number to get the entities that we want.

Sorting in Java | How to get value from other field of entity if the current field is null while sorting entity list in java?

Context :
Consider following entity in spring boot project,
#Entity
public class Transaction{
#Id
private Integer tId;
private Integer amount;
private LocalDate invoiceDate;
private LocalDate tDate;
}
A method which creates object of transaction, doesn't set tDate.
There is an independent method which sets tDate.
So, in the database some entries don't have 'tDate'.
Problem :
Now, I want to show all the entries in database sorted using tDate but, for those entries which does not contain tDate it should consider invoiceDate.
Sample Database entries
t_id
amount
invoice_date
t_date
1
1200
2/3/2022
4/3/2022
2
2434
5/3/2022
6/3/2022
3
234
3/3/2022
NULL
Sample expected Output
[[tId = 3, amount = 234, invoiceDate = 3/3/2022, tDate = NULL]
[tId = 1, amount = 1200, invoiceDate = 2/3/2022, tDate = 4/3/2022]
[tId = 2, amount = 2434, invoiceDate = 5/3/2022, tDate = 6/3/2022]]
Note : Highlighted dates above are the dates considered for sorting.
What I tried
I tried to use the combination of nullsLast(Comparator.naturalOrder()) and thenComparing() methods of Comparator.comparing() but it doesn't give the expected output. It sorts the entries withtDate and entries without tDate separately.
Thank you in advance for any help!!
Also, I'm using MongoRepository in repository layer, so I can use the Sort object as well.
Thank you!

Unable to create a constant value - only primitive types or Enumeration types allowed

I have seen some questions related to this Exception here but none made me understand the root cause of the problem. So here we have one more...
var testquery =
((from le in context.LoanEMIs.Include("LoanPmnt")
join lp in context.LoanPmnts on le.Id equals lp.LoanEMIId
where lp.PmntDtTm < date && lp.IsPaid == false
&& le.IsActive == true && lp.Amount > 0
select new ObjGetAllPendingPmntDetails
{
Id = lp.Id,
Table = "LoanEMI",
loanEMIId = lp.LoanEMIId,
Name = le.AcHead,
Ref = SqlFunctions.StringConvert((double)le.FreqId),
PmntDtTm = lp.PmntDtTm,
Amount = lp.Amount,
IsDiscard = lp.IsDiscarded,
DiscardRemarks = lp.DiscardRemarks
}).DefaultIfEmpty(ObjNull));
List<ObjGetAllPendingPmntDetails> test = testquery.ToList();
This query gives the following Exception Message -
Unable to create a constant value of type CashVitae.ObjGetAllPendingPmntDetails. Only primitive types or enumeration types are supported in this context.
I got this Exception after I added the SQL function statement to convert le.FreqId which is a byte to a string as ToString() is not recognized in the LINQ Expression Store.
ObjGetAllPendingPmntDetails is a partial class in my model which is added as it is used too many times in the code to bind data to tables.
It has both IDs as long, 'Amount' as decimal, PmntDtTm as Datetime,IsDiscard as bool and remaining all are string including 'Ref'.
I get no results as currently no data satisfies the condition. While trying to handle null, I added DefaultIfEmpty(ObjNull) and ObjNull has all properties initialized as follows.
ObjGetAllPendingPmntDetails ObjNull = new ObjGetAllPendingPmntDetails()
{ Id = 0, Table = "-", loanEMIId = 0, Name = "-", Ref = "-",
PmntDtTm = Convert.ToDateTime("01-01-1900"),
Amount = 0, IsDiscard = false, DiscardRemarks = "" };
I need this query to work fine as it has Union() called on it with 5 other queries. All returning the same ObjGetAllPendingPmntDetails columns. But there is some problem as this query has no data satisfying the conditions and the Exception Shared Above.
Any suggestions are appreciated as I am unable to understand the root cause of the problem.
#AndrewCoonce is right, the .DefaultIfEmpty(ObjNull) is the culprit here. Entity Framework turns DefaultIfEmpty into something like...
CASE WHEN ([Project1].[C1] IS NULL) THEN #param ELSE [Project1].[Value] END AS [C1]
...but there's no way to coerce an instance of ObjGetAllPendingPmntDetails into something that can take the place of #param, so you get an exception.
If you move the DefaultIfEmpty call to after the ToList it should work correctly (although you'll need to call ToList again after that if you really want a concrete list instance).

restrict objects based on count of sub list in LINQ

I am well and truly stuck for some reason. I have a bunch of XML which via linq I have adjusted to fit into my DTO objects, this works fine, but I need an additional filter that only returns the Room Types that have rooms that have full availability for a period.
Now my original query to setup the DTO Works fine, but I would like to add something that only returns the rooms that have rates available for the entire periods, so say you want to book 10 days, you should only get the room types back that have the full 10 days available. My original query is the following:
var items = (
from rt in data.Descendants("RoomType")
select new RoomType
{
name = rt.Descendants("RoomDescription").Descendants("Text").SingleOrDefault().Value,
rooms = (
from r in rt.Descendants("Room")
select new Room
{
Name = r.Attribute("id").Value,
rates = (
from rr in r.Descendants("RoomRate")
where DateTime.Parse(rr.Attribute("EffectiveDate").Value) >= startDate
where DateTime.Parse(rr.Attribute("EffectiveDate").Value) <= endDate
select new RoomRate
{
EffectiveDate = DateTime.Parse(rr.Attribute("EffectiveDate").Value)
})
})
});
if it is at all possible to have the restriction in this query that would be amazing, but I couldn't see how to do it.
When I tried to create another query off the back of this one I didn't know how I could query the count of Rooms.RoomRates from the RoomType object to return. I tried
var result = items.Where(i => i.rooms.Where(r => r.rates.Count() == 10));
but that gives me an exception where it can't convert IEnumerable to bool, .Any() compiles but returns everything (as probably expected).
Does anyone know what I am doing wrong here?
EDIT: ** this is how it is pulling the data out at the moment
Room Type: Single
Room 1 (Days Available 10)
Room 2 (Days Available 10)
Room Type: Twin
Room 3 (Days Available 10)
Room 4 (Days Available 4)
what I am trying to do is exclude Room 4 from returning as it doesn't meet the days criteria
so what I should get back is:
Room Type: Single
Room 1 (Days Available 10)
Room 2 (Days Available 10)
Room Type: Twin
Room 3 (Days Available 10)
If you only want Rooms, you can just flatten the collection, then filter it:
IEnumerable<Room> rooms = items.SelectMany(i => i.Rooms)
.Where(r => r.rates.Count() == 10)
If you want RoomTypes, you'll need to create new RoomType objects with filtered Rooms:
var types = items.Select(i =>
new RoomType {
name = i.name,
rooms = i.rooms.Where(r => r.rates.Count() == 10)
}
);

Max sequence from a view containing multiple record using Linq lambda

I've been at this for a while. I have a data set that has a reoccurring key and a sequence similar to this:
id status sequence
1 open 1
1 processing 2
2 open 1
2 processing 2
2 closed 3
a new row is added for each 'action' that happens, so the various ids can have variable sequences. I need to get the Max sequence number for each id, but I still need to return the complete record.
I want to end up with sequence 2 for id 1, and sequence 3 for id 2.
I can't seem to get this to work without selecting the distinct ids, then looping through the results, ordering the values and then adding the first item to another list, but that's so slow.
var ids = this.ObjectContext.TNTP_FILE_MONITORING.Select(i => i.FILE_EVENT_ID).Distinct();
List<TNTP_FILE_MONITORING> vals = new List<TNTP_FILE_MONITORING>();
foreach (var item in items)
{
vals.Add(this.ObjectContext.TNTP_FILE_MONITORING.Where(mfe => ids.Contains(mfe.FILE_EVENT_ID)).OrderByDescending(mfe => mfe.FILE_EVENT_SEQ).First<TNTP_FILE_MONITORING>());
}
There must be a better way!
Here's what worked for me:
var ts = new[] { new T(1,1), new T(1,2), new T(2,1), new T(2,2), new T(2,3) };
var q =
from t in ts
group t by t.ID into g
let max = g.Max(x => x.Seq)
select g.FirstOrDefault(t1 => t1.Seq == max);
(Just need to apply that to your datatable, but the query stays about the same)
Note that with your current method, because you are iterating over all records, you also get all records from the datastore. By using a query like this, you allow for translation into a query against the datastore, which is not only faster, but also only returns only the results you need (assuming you are using Entity Framework or Linq2SQL).

Resources