I am working on a Spring-MVC application using Hibernate as ORM and PostgreSQL database, in which I am saving in a
database row some values(obvious). With that information, I am also
saving the TimeStamp(java.sql.TimeStamp) when the entry was made. For
some reasons I want to remove entries which are older than 5 minutes.
How is it possible for me to give an HQL query with Timestamp, something like if timestamp>oldTimestamp then delete that row. I have something like this till now :
#Override
#Scheduled(fixedRate = 100000)
public void removeStaleLocks() {
session = this.sessionFactory.getCurrentSession();
//Timestamp timestamp = // current timestamp;
Query query = session.createQuery("delete from NoteLock as nl where nl.timeStamp=:timeStamp");
query.setParameter("timeStamp",timeStamp);
query.executeUpdate();
session.flush();
}
What I would like to do is pass the query a parameter as use this as current timestamp denoting this is the currentTime, and delete all notes which are more than 5 minutes old. Any help would be nice. Thanks a lot.
long now = System.currentTimeMillis();
long nowMinus5Minutes = now - (5L * 60L * 1000L);
Timestamp nowMinus5MinutesAsTimestamp = new Timestamp(nowMinus5Minutes);
Query query = session.createQuery("delete from NoteLock as nl where nl.timeStamp < :limit");
query.setParameter("limit", nowMinus5MinutesAsTimestamp);
query.executeUpdate();
Related
The code
int offset = 0;
int bulkSize = eodFileConfig.getBulkSize(); // sample of 10k
setDateTimeFormat();
//Get total record from Temp table
long max = extQRMerchantTrxHistService.getTotalRecords();
do {
log.debug("[execute] start to write to actual table");
// bulk size represent how many items in a page, offset is the page
Page<ExtensionQRMerchantTrxHistEntity> records = extQRMerchantTrxHistService.findRecordsWithPagination(offset, bulkSize);
List<TransactionHistoryExtEntity> transactions = new ArrayList<>();
for (ExtensionQRMerchantTrxHistEntity tempEntity : records) {
log.debug("Record: {} ", tempEntity);
Date a = new Date();
//Query from T_TRXN_DETAIL_EXT
Date dateTime = sf.parse(tempEntity.getTransactionDate());
List<TransactionHistoryExtEntity> histories =
transactionHistoryInquiryService.retrieveHistoryBasedOnRefNoDateAmt(
tempEntity.getTransactionRefNo(), dateTime, tempEntity.getTransactionAmount());
Date b = new Date();
System.out.println("Query Time: " + Math.abs(a.getTime() - b.getTime()));
Date c = new Date();
TransactionHistoryExtEntity transaction;
if (histories.isEmpty()) {
//Insert record
transaction = setTransactionHistory(Boolean.TRUE, tempEntity, null);
} else {
//Update record
transaction = setTransactionHistory(Boolean.FALSE, tempEntity, histories.get(0));
}
Date d = new Date();
System.out.println("Query Time: " + Math.abs(c.getTime() - d.getTime()));
Date e = new Date();
//transactionHistoryExtRepository.saveAndFlush(transaction);
Date f = new Date();
System.out.println("Query Time: " + Math.abs(e.getTime() - f.getTime()));
//Add to list
transactions.add(transaction);
}
//Save & Update all records
transactionHistoryExtRepository.saveAll(transactions);
offset++;
} while ((long) offset * bulkSize < max);
The query
List<TransactionHistoryExtEntity> findTopByReferenceNumberAndTransactionDateAndAmountOrderByTransactionDateDesc(
String referenceNo, Date transactionDate, BigDecimal amount);
I am a bit new with this spring boot stuff. I am trying to insert/update 50k-100k of records into the table. Running 10k seems fast enough however as the size increases the time it takes for the query part in the inner for loop increases. For a record of 50k with 10k bulksize, the first 10k took about 80 seconds to complete (the entire inner iteration), followed by 472 seconds for the next 10k while the last 10k took 1k+ seconds to process. Can anyone explain what is causing the issue? Also, I cleared my table before executing this, meaning it is empty when I run this.
As a result, I tried different bulksize such as 30k and 50k. At 30k bulksize, which means it will be 2 outer loops, it completes everything at about 38mins, 50k bulksize at about 33 mins whereas 10k at an hour for 50k records. But this would defeat the purpose of having pagination at the start.
So I noticed that the avg time to query spiked after saveAll method. I am not too sure if it is due to the saveAll or the size of the records. Without the inner loop query part, the entire process just takes about 1 mins to complete.
Does anyone have any idea regarding the issue and how I can increase the performance?
I am working on one spring boot application. Here I have 100,000 records that are inserted into db by different process. and its inserting one by one. I can't do batch insert.
So in starting some of the task performing well and not taking too much time ,but application process some and database is growing slowly - 2, insert time is increasing.
How can I speed up the process or avoid to get it slow?
The quickest way for inserts would be to use a prepared Statement.
Inject the jdbcTemplate and use its batchUpdate method and set the batch size. It's lightning fast.
If you think you cannot use the batch insert, which is hard for me to understand, then set the batch size to 1.
However, the most optimal batch size is certainly larger than that and depends on the insert statement. You have to experiment a bit with it.
Here an example for you with a class called LogEntry. Substitute class, table, columns and attributes by your class, table, columns and attributes and place it into your repository implementation.
Also make sure you set the application properties as mentioned here https://stackoverflow.com/a/62414315/12918872
Regarding the id Generator either set a sequence id generator (also shown in that link) or like in my case, just generate it on your own by asking for the maxId of your table at the beginning and then counting up.
#Autowired
private JdbcTemplate jdbcTemplate;
public void saveAllPreparedStatement2(List<LogEntry> logEntries) {
int batchSize = 2000;
int loops = logEntries.size() / batchSize;
for (int j = 0; j <= loops; j++) {
final int x = j;
jdbcTemplate.batchUpdate("INSERT INTO public.logentries(\r\n"
+ " id, col1, col2, col3, col4, col5, col6)\r\n"
+ " VALUES (?, ?, ?, ?, ?, ?, ?);\r\n", new BatchPreparedStatementSetter() {
public void setValues(PreparedStatement ps, int i) throws SQLException {
int counter = x * batchSize + i;
if (counter < logEntries.size()) {
LogEntry logEntry = logEntries.get(counter);
ps.setLong(1, (long) logEntry.getId());
ps.setString(2, (String) logEntry.getAttr1());
ps.setInt(3, (int) logEntry.getAttr2());
ps.setObject(4, logEntry.getAttr3(), Types.INTEGER);
ps.setLong(5, (long) logEntry.getAttr4());
ps.setString(6, (String) logEntry.getAttr5());
ps.setObject(7, logEntry.getAttr6(), Types.VARCHAR);
}
}
public int getBatchSize() {
if (x * batchSize == (logEntries.size() / batchSize) * batchSize) {
return logEntries.size() - (x * batchSize);
}
return batchSize;
}
});
}
}
Some advices for you :
It is not normal if you say the inserting time is getting increasing if more records are inserted. From my experience , most probably it is due to some logic bugs in your program such that you are processing more unnecessary data when you are inserting more records. So please revise your inserting logic first.
Hibernate cannot batch insert entities if the entity are using IDENTITY to generate is ID . You have to change it to use SEQUENCE to generate the ID with the pooled or pooled-lo algorithm.
Make sure you enable the JDBC batching feature in the hibernate configuration
If you are using PostgreSQL , you can add reWriteBatchedInserts=true in the JDBC connection string which will provide 2-3x performance gain.
Make sure each transaction will insert a batch of entities and then commit but not each transaction only insert one entity.
For more details about points (2), (3) and (4) , you can refer to my previous answers at this.
Is it possible to generate arbitrary where condtions SQL query through Jdbc template:
example:
If i pass value for 1 parameter (only name) : search by name
"select * from address where shopname = ?";
If i pass value for 2 parameter (name and city) - search by shopname and city:
"select * from address where shopname = ? and city = ?";
I have mupliple search fields. 7 fields. If user enters any combination. i have search only based on parameter. How to dynamically pass the parameters to the sql. Need snippet/Example how to achieve this.
What you want is some sort of criteria building api, which Hibernate has. Unfortunately, I don't think Spring's JdbcTemplate has any such facility. Others will correct me if I'm wrong...
Though as some guys already suggested that Hibernate is the best way of doing this, but still i think you can try this approach-
String sql = "select * from address where 1 = 1";
if(shopname != null)
sql += "and shopname = :shopname";
if(city!= null)
sql += "and city = :city";
and so on..and use NamedParameterJdbcTemplate
Spring Data and Hibernate have that kind of functionality. Though it might not be worth dragging in such big framework for your app.
You can try to check out SimpleJdbcInsert
http://docs.spring.io/spring/docs/current/spring-framework-reference/html/jdbc.html
Edit:
Alternatively you can try to fix it in SQL with checking on empty, but if you have lots of data to go through, this technique will slow down your request.
"select * from address
where (shopname = ? or shopname = null)
and (city = ? or city = null)";
If Scala is an option to you, the query could be constructed with something like this:
case class Search(shopname:String, city:String = None) {
def sql = "select * from address where shopname = '"+shopname+"'" + city.map(" and city = '"+
_ +"'").getOrElse("")
}
Example usage:
Search("lloh").sql
Search("lloh", Some("Austin")).sql
Is it possible to use JPQL for getting random rows? For example in SQL Server I would use:
select * from myTable where columnName = 4 order by newid()
Thanks,
Rod
This is what I use. I first get the number of rows for the entity and I then limit the results of the fetch query to a random row. This involves two queries, so if this is a problem for you you might want to watch native queries. If not here is the code I use:
public <T> T randomEntity(EntityManager em, Class<T> clazz) {
Query countQuery = em.createQuery("select count(id) from "+clazz.getName());
long count = (Long)countQuery.getSingleResult();
Random random = new Random();
int number = random.nextInt((int)count);
Query selectQuery = em.createQuery("from "+clazz.getName());
selectQuery.setFirstResult(number);
selectQuery.setMaxResults(1);
return (T)selectQuery.getSingleResult();
}
As of today (April 9th 2010), JPQL does not support random ordering
ScalarQuery<int> query = new ScalarQuery<int>(typeof(Role),
"select count(role.RoleId) from Role as role");
return query.Execute();
It fails with the invalidcast exception but succeeds when count is replaced with max.
Edit: Some databases will return long for count queries. For example SQL Server.
ScalarQuery<long> query = new ScalarQuery<long>(typeof(Role),
"select count(r) from Role r");
return query.Execute();
What database are you using? Could be that count does not return an int.
Your could also try using http://api.castleproject.org/html/T_Castle_ActiveRecord_Queries_CountQuery.htm
Not exactly the answer to the question, but a recommendation: if you want to avoid the hassle of having to issue a query at all yourself, then just use ActiveRecordMediator<T>.Count() (which has overloads that take criteria / filter strings if you want a conditional count) and all return int against all databases.
Based on testing the answers given to date, the following worked for me (including a where clause):
// Option 1
int result = ActiveRecordMediator<Post>.Count("BlogId = ?", blogId);
// Option 2
CountQuery query = new CountQuery(typeof(Post), "BlogId = ?", blogId);
int result = ActiveRecordMediator.ExecuteQuery(query);
// Option 3
ScalarQuery<long> query= new ScalarQuery<long>(typeof(Post),
"SELECT COUNT(*) FROM Post WHERE BlogId = ?", blogId);
long result = query.Execute();