I am creating a batch program which will pick records having particular status say 'x' from Oracle Database and do some processing and update status to 'y' once processing is successful.
However, there would be multiple instances of same batch running and I have to make sure that no two batches pick same records in 'x' status. I am trying to achieve this by using 'Select ... for Update' in Spring data.
While testing same I can see that the lock is not restricting read in other sessions. I cannot post original code but Pseudo code for same is present below. Any reference to Source using #Lock & #Transactional will be helpful.
++ I am able to see correct query (Select For update) being written in trace logs, but still lock is not acquired.
---Starting Pseudo code---
#Controller
class {
#PutMapping
#Transactional
methodCall(){
City city = emMethod(Long id)
//My print statement that data has been selected
sleep thread for 15 sec
//Print statement after sleep
update Method call here
}
}
---End of Controller class
#Repository
Class Repos{
#Autowired
EntityManager em;
public City emMethod(Long id){
Query q = em.CreateQuery("Select C from City C where id =:id");
q.setParameter("id",id);
**q.setLockMode(LockModeType.PESSIMISTIC_WRITE)**;
List list = q.getResultList();
return (City) list.get(0);
}
#Modifying
myUpdateMethod(City city){//update body here};
}
}
Related
When i want to modify the database via JPQL i have to mark the query as Transactional and Modiyfing. If i do so, the return type of the method representing the query has to be either void or int(representing the number of edited rows i think). Why are only the two return types allowed? If i do a HTTP-PUT request and update the object with an own JPQL query, i would like to return the updated object again. Whats the best way to do it if the return type of the query has to be void or int? Do i have to do a seperate query/request again which selects the object after it was updated?
EDIT:
Thats how i call the query:
if (inactivityListDTO.getProjectIds().size() > 0) {
projectRepository.updateProjectsIsArchivedByProjectIds(inactivityListDTO.getProjectIds(), inactivityListDTO.getIsArchived());
}
Thats the query:
#Transactional
#Modifying
#Query("UPDATE Project project SET project.isArchived = :isArchived,
project.archivedDate = current_date " +
"WHERE project.id IN :ids")
void updateProjectsIsArchivedByProjectIds(#Param("ids") List<Long> ids, #Param("isArchived") boolean isArchived);
Because it finally boils down to execute a standard UPDATE SQL in the DB , and the UPDATE in standard SQL only returns the number of records being updated and does not return a result set.
And yes , if you need get a record 's value after update , you have to query it again. Alternatively , you should consider using a JPA way to update a record , which first query the object , then update it by changing its state . Something like below (Assume you are using spring #Transactional to manage the transactional boundary):
#Transactional
public void changeEmployeeSalary(Integer employeeId , Integer salary){
Employee employee = entityManager.find(Employee.class , employeeId);
employee.setSalary(salary);
}
In this way , you do not need to query the record again after it is updated and you also do not need to manually write a UPDATE SQL.
During code optimization I found few areas where I was using findOne() within for loop –
public List<User> validateUsers(List<String> userIds) {
List<User> validUsers = new ArrayList<>();
for ( String userId : userIds) {
User user = userRepository.findOne(userId); //Network hit :: expensive call
//Perform validations
...
//Add valid users to validUsers list
...
}
return validUsers;
}
Above method takes long time if I pass huge list of users to validate. [for 300 users around 5 sec.]
Then I changed above method to use findAll() and perform validations on result collection -
public List<User> validateUsers(List<String> userIds) {
List<User> validUsers = new ArrayList<>();
Iterable<User> itr = userRepository.findAll(userIds); //Only one Network hit
for ( User user : itr) {
//Perform validations
...
//Add valid users to validUsers list
...
}
return validUsers;
}
Now for 300 users, results coming in 100 ms.
Question is: Is there any side effects of using findAll() considering the underlying structure of Cassandra? Also I am using CrudRepository. Should I use CassandraRepository?
Following are the parameters to think of when you are attempting this.
How big is the users table, if you are using findAll.
Partition keys for the user table
As Cassandra queries are faster with the primary key fields, findOne might perform better with the large amount of data.
However, can you try
List<T> findAllById(Iterable<ID> ids);
from org.springframework.data.cassandra.repository.CassandraRepository
I'm working on a process that checks and updates data from Oracle database. I'm using hibernate and spring framework in my application.
The application reads a csv file, processes the content, then persiste entities :
public class Main() {
Input input = ReadCSV(path);
EntityList resultList = Process.process(input);
WriteResult.write(resultList);
...
}
// Process class that loops over input
public class Process{
public EntityList process(Input input) :
EntityList results = ...;
...
for(Line line : input.readLine()){
results.add(ProcessLine.process(line))
...
}
return results;
}
// retrieving and updating entities
Class ProcessLine {
#Autowired
DomaineRepository domaineRepository;
#Autowired
CompanyDomaineService companydomaineService
#Transactional
public MyEntity process(Line line){
// getcompanyByXX is CrudRepository method with #Query that returns an entity object
MyEntity companyToAttach = domaineRepository.getCompanyByCode(line.getCode());
MyEntity companyToDetach = domaineRepository.getCompanyBySiret(line.getSiret());
if(companyToDetach == null || companyToAttach == null){
throw new CustomException("Custom Exception");
}
// AttachCompany retrieves some entity relationEntity, then removes companyToDetach and adds CompanyToAttach. this updates relationEntity.company attribute.
companydomaineService.attachCompany(companyToAttach, companyToDetach);
return companyToAttach;
}
}
public class WriteResult{
#Autowired
DomaineRepository domaineRepository;
#Transactional
public void write(EntityList results) {
for (MyEntity result : results){
domaineRepository.save(result)
}
}
}
The application works well on files with few lines, but when i try to process large files (200 000 lines), the performance slows drastically, and i get a SQL timeout.
I suspect cache issues, but i'm wondering if saving all the entities at the end of the processing isn't a bad practice ?
The problem is your for loop which is doing individual saves on the result and thus does single inserts slowing it down. Hibernate and spring support batch inserts and should be done when ever possible.
something like domaineRepository.saveAll(results)
Since you are processing lot of data it might be better to do things in batches so instead of getting one company to attach you should get a list of companies to attach processes those then get a list of companies to detach and process those
public EntityList process(Input input) :
EntityList results;
List<Code> companiesToAdd = new ArrayList<>();
List<Siret> companiesToRemove = new ArrayList<>();
for(Line line : input.readLine()){
companiesToAdd.add(line.getCode());
companiesToRemove.add(line.getSiret());
...
}
results = process(companiesToAdd, companiesToRemove);
return results;
}
public MyEntity process(List<Code> companiesToAdd, List<Siret> companiesToRemove) {
List<MyEntity> attachList = domaineRepository.getCompanyByCodeIn(companiesToAdd);
List<MyEntity> detachList = domaineRepository.getCompanyBySiretIn(companiesToRemove);
if (attachList.isEmpty() || detachList.isEmpty()) {
throw new CustomException("Custom Exception");
}
companydomaineService.attachCompany(attachList, detachList);
return attachList;
}
The above code is just sudo code to point you in the right direction, will need to work out what works for you.
For every line you read you are doing 2 read operations here
MyEntity companyToAttach = domaineRepository.getCompanyByCode(line.getCode());
MyEntity companyToDetach = domaineRepository.getCompanyBySiret(line.getSiret());
You can read more than one line and us the in query and then process that list of companies
I'm developing a Spring web application whose persistence layer consists in Spring Roo generated JPA entities, with Hibernate as persistence provider and MySql as underlying DB.
Among my entities I have a class Detection with a tstamp java.util.Date field generated in Roo as follows:
entity jpa --class ~.data.Detection
...
field date --fieldName tstamp --type java.util.Date
...
finder add findDetectionsByTstampBetween
(the finder method was of course chosen after executing finder list)
In my controller code, at a point I invoke:
List<Detection> detections = Detection.findDetectionsByTstampBetween(from, to).getResultList();
Where from and to are two valid java.util.Date(s). When testing sample data though (after ensuring that for a given choice of from, to the returned list shouldn't be empty), I got an empty list and investigated the reasons.
I found in tomcat logs that Hibernate was generating the following SQL:
Hibernate: select detection0_.id as id1_3_, ...etc..., detection0_.tstamp as tstamp4_3_ from detection detection0_ where detection0_.tstamp>=?
I would expect the where clause should contain a trailing "AND detection0_.tstamp<=?", checking the other date range limit. I took a look at the generated Detection.findDetectionsByTstampBetween(Date minTstamp, Date maxTstamp) method in Detection_Roo_Finder.aj and actually the "AND" is present in the invocation to createQuery.
public static TypedQuery<Detection> Detection.findDetectionsByTstampBetween(Date minTstamp, Date maxTstamp) {
if (minTstamp == null) throw new IllegalArgumentException("The minTstamp argument is required");
if (maxTstamp == null) throw new IllegalArgumentException("The maxTstamp argument is required");
EntityManager em = Detection.entityManager();
TypedQuery<Detection> q = em.createQuery("SELECT o FROM Detection AS o WHERE o.tstamp BETWEEN :minTstamp AND :maxTstamp", Detection.class);
q.setParameter("minTstamp", minTstamp);
q.setParameter("maxTstamp", maxTstamp);
return q;
}
Any idea what could cause the problem?
I've finally found the solution to the riddle and, as it turned out, the issue had nothing to do with JPA.
The problem was that the call to the persistence layer was inserted inside a Rest service controller with the following mapping:
#ResponseBody
#RequestMapping(value="/detections", method=RequestMethod.GET, params="from, to" )
public Object getDetectionsInRange(
#RequestParam(required=true) #DateTimeFormat(pattern="yyyy-MM-dd HH:mm") final Date from,
#RequestParam(required=true) #DateTimeFormat(pattern="yyyy-MM-dd HH:mm") final Date to
)
{
...
List<Detection> detections = Detection.findDetectionsByTstampBetween(from, to).getResultList();
...
}
The error was in the definition of the params= argument in #RequestMapping, the correct format being as follows:
#RequestMapping(value="/detections", method=RequestMethod.GET, params={"from", "to"} )
This error caused another version of the controller method for /detections. In this second version I called a different finder method, which appeared to generate the wrong SQL in Hibernate.
#ResponseBody
#RequestMapping(value="/detections", method=RequestMethod.GET )
public Object getDetections(
#RequestParam(required=false, defaultValue="0") int days,
#RequestParam(required=false, defaultValue="0") int hours,
#RequestParam(required=false, defaultValue="0") int minutes
)
{
...
List<Detection> detections = Detection.findDetectionsByTstampGreaterThanEquals( ... ).getResultList();
...
}
How to select a record from a table using WHERE clause and comparing instances (patient)
public History findHistory(Patient patient) { History model=null;
Session sesion=util.HibernateUtil.getSessionFactory().getCurrentSession();
String sql="FROM History h WHERE h.patients=" + patient;
try{
sesion.beginTransaction();
model=(History) sesion.createQuery(sql).uniqueResult();
sesion.beginTransaction().commit();
}
catch(Exception e){
sesion.beginTransaction().rollback();
}
return model;
}
That throws a queryException #1562
e.queryString="FROM entidad.Historia h WHERE h.pacientes=entidad.Paciente#3ad3a221"
e.detailMessage="unexpected char: '#'"
The problem with your code is that concatenating patient like you do will just append patient.toString(), which in your case is the default implementation (i.e. classname#hashcode) and it is no use for Hibernate to find out which data to retrieve in the DB.
You need to bind the parameter, first:
String sql = "FROM History h WHERE h.patients = :patient";
Then
model = (History) sesion.createQuery(sql)
.setParameter("patient", patient)
.uniqueResult();
Edit:
SQLGrammarException: could not execute query can occurs for various reason. Try to run the generated query in SqlDeveloper (or any other tool) and see what your DB says. In your case, the last part and .=? cause the error. The cross join is Harming too. I suspect your mapping is incomplete and Hibernate can't find how to join History and Patient. Try to add something like this in History entity:
#ManyToOne
#JoinColumn(name = "patient")
private Patient patient;