I am trying to fetch List from table FlexiBooking and then add this to another list i.e. to move to another table and delete those entries fetched from this table. I have used HibernateTemplate() object, since the project is being done using Spring Framework. But I am getting exception that trying to delete Detatched object. What is the problem?
Below is my code:
#Override
public void moveToNormalBooking(User user,
int no_of_seats) {
String queryString = "FROM FlexiBooking";
HibernateTemplate hibernateTemplate = getHibernateTemplate();
hibernateTemplate.setMaxResults(no_of_seats);
List<FlexiBooking> flexiBookingsTobeMoved = hibernateTemplate.find(queryString);
List<FlightBooking> flightBookings = new ArrayList<FlightBooking>();
int i =0;
while(i < flexiBookingsTobeMoved.size()) {
FlightBooking flightbooking = new FlightBooking();
flightbooking.setCostPerTicket(flexiBookingsTobeMoved.get(i).getTotalCost());
flightbooking.setDateOfJourney(flexiBookingsTobeMoved.get(i).getScheduledFlight().getScheduledFlightDate());
Booking booking = new Booking();
booking.setBooker(flexiBookingsTobeMoved.get(i).getUser());
booking.setBookingDate(flexiBookingsTobeMoved.get(i).getBookingDate());
booking.setBookingReferenceNo(flexiBookingsTobeMoved.get(i).getBookingReferenceNumber());
booking.setCancelled(false);
flightbooking.setBooking(booking);
flightbooking.setFlightRoute(flexiBookingsTobeMoved.get(i).getScheduledFlight());
flightbooking.setCouponCode(flexiBookingsTobeMoved.get(i).getCouponCode());
flightBookings.add(flightbooking);
i++;
}
hibernateTemplate.saveOrUpdateAll(flightBookings);
// hibernateTemplate.update(flexiBookingsTobeMoved);
hibernateTemplate.deleteAll(flexiBookingsTobeMoved);
}
Related
I am Using Spring batch withing spring boot.
reading a table from source and writing to destination.
i am populating three tables where the read-count for one of the table is less than Actual.
Say In source table there are 2656274 Rows but the read-count I am getting is readCount=2577203, writeCount=2577203
other tables having 8 lacks records and 10 k records working fine
public class DxDatabaseItemReader extends JdbcCursorItemReader<DxDataRead> {
public DxDatabaseItemReader(DxJobContext jobCntx) {
synchronized (this){
System.err.print("Into DxDatabaseItemReader");
String[] dsParam =jobCntx.srcDbName.split("#");
this.setDataSource(createOracleDataSource(dsParam[1],dsParam[2],dsParam[3]));
this.setSql(createFetchQuery(jobCntx.srcFieldNames,jobCntx.srcTableName));//"SELECT SOEID, FST_NAM, LST_NAM FROM REF_PRSNL_MSTR");
this.setFetchSize(0);
this.setRowMapper((ResultSet rs, int rowNum) -> {
Map<String, String> fieldNameMap = new HashMap<>();
for(int i=0;i<jobCntx.srcFieldNames.size();i++)
fieldNameMap.put(getSrcFieldName(jobCntx.srcFieldNames, i), rs.getString(jobCntx.srcFieldNames.get(i)));
DxDataRead dataRead = new DxDataRead();
dataRead.setFieldNameMap(fieldNameMap);
return dataRead;
});
}
}
Expected : it should read all the data and write
I am trying to develop a batch process using Spring Batch + Spring Boot (Java config), but I have a problem doing so. I have a software that has a database and a Java API, and I read records from there. The batch process should retrieve all the documents which expiration date is less than a certain date, update the date, and save them again in the same database.
My first approach was reading the records 100 by 100; so the ItemReader retrieve 100 records, I process them 1 by 1, and finally I write them again. In the reader, I put this code:
public class DocumentItemReader implements ItemReader<Document> {
public List<Document> documents = new ArrayList<>();
#Override
public Document read() throws Exception, UnexpectedInputException, ParseException, NonTransientResourceException {
if(documents.isEmpty()) {
getDocuments(); // This method retrieve 100 documents and store them in "documents" list.
if(documents.isEmpty()) return null;
}
Document doc = documents.get(0);
documents.remove(0);
return doc;
}
}
So, with this code, the reader reads from the database until no records are found. When the "getDocuments()" method doesn't retrieve any documents, the List is empty and the reader returns null (so the Job finish). Everything worked fine here.
However, the problem appears if I want to use several threads. In this case, I started using the Partitioner approach instead of Multi-threading. The reason of doing that is because I read from the same database, so if I repeat the full step with several threads, all of them will find the same records, and I cannot use pagination (see below).
Another problem is that database records are updated dynamically, so I cannot use pagination. For example, let's suppose I have 200 records, and all of them are going to expire soon, so the process is going to retrieve them. Now imagine I retrieve 10 with one thread, and before anything else, that thread process one and update it in the same database. The next thread cannot retrieve from 11 to 20 records, as the first record is not going to appear in the search (as it has been processed, its date has been updated, and then it doesn't match the query).
It is a little difficult to understand, and some things may sound strange, but in my project:
I am forced to use the same database to read and write.
I can have millions of documents, so I cannot read all the records at the same time. I need to read them 100 by 100, or 500 by 500.
I need to use several threads.
I cannot use pagination, as the query to the databse will retrieve different documents each time it is executed.
So, after hours thinking, I think the unique possible solution is to repeat the job until the query retrives no documents. Is this possible? I want to do something like the step does: Do something until null is returned - repeat the job until the query return zero records.
If this is not a good approach, I will appreciate other possible solutions.
Thank you.
Maybe you can add a partitioner to your step that will :
Select all the ids of the datas that needs to be updated (and other columns if needed)
Split them in x (x = gridSize parameter) partitions and write them in temporary file (1 by partition).
Register the filename to read in the executionContext
Then your reader is not reading from the database anymore but from the partitioned file.
Seem complicated but it's not that much, here is an example which handle millions of record using JDBC query but it can be easily transposed for your use case :
public class JdbcToFilePartitioner implements Partitioner {
/** number of records by database fetch */
private int fetchSize = 100;
/** working directory */
private File tmpDir;
/** limit the number of item to select */
private Long nbItemMax;
#Override
public Map<String, ExecutionContext> partition(final int gridSize) {
// Create contexts for each parttion
Map<String, ExecutionContext> executionsContexte = createExecutionsContext(gridSize);
// Fill partition with ids to handle
getIdsAndFillPartitionFiles(executionsContexte);
return executionsContexte;
}
/**
* #param gridSize number of partitions
* #return map of execution context, one for each partition
*/
private Map<String, ExecutionContext> createExecutionsContext(final int gridSize) {
final Map<String, ExecutionContext> map = new HashMap<>();
for (int partitionId = 0; partitionId < gridSize; partitionId++) {
map.put(String.valueOf(partitionId), createContext(partitionId));
}
return map;
}
/**
* #param partitionId id of the partition to create context
* #return created executionContext
*/
private ExecutionContext createContext(final int partitionId) {
final ExecutionContext context = new ExecutionContext();
String fileName = tmpDir + File.separator + "partition_" + partitionId + ".txt";
context.put(PartitionerConstantes.ID_GRID.getCode(), partitionId);
context.put(PartitionerConstantes.FILE_NAME.getCode(), fileName);
if (contextParameters != null) {
for (Entry<String, Object> entry : contextParameters.entrySet()) {
context.put(entry.getKey(), entry.getValue());
}
}
return context;
}
private void getIdsAndFillPartitionFiles(final Map<String, ExecutionContext> executionsContexte) {
List<BufferedWriter> fileWriters = new ArrayList<>();
try {
// BufferedWriter for each partition
for (int i = 0; i < executionsContexte.size(); i++) {
BufferedWriter bufferedWriter = new BufferedWriter(new FileWriter(executionsContexte.get(String.valueOf(i)).getString(
PartitionerConstantes.FILE_NAME.getCode())));
fileWriters.add(bufferedWriter);
}
// Fetching the datas
ScrollableResults results = runQuery();
// Get the result and fill the files
int currentPartition = 0;
int nbWriting = 0;
while (results.next()) {
fileWriters.get(currentPartition).write(results.get(0).toString());
fileWriters.get(currentPartition).newLine();
currentPartition++;
nbWriting++;
// If we already write on all partitions, we start again
if (currentPartition >= executionsContexte.size()) {
currentPartition = 0;
}
// If we reach the max item to read we stop
if (nbItemMax != null && nbItemMax != 0 && nbWriting >= nbItemMax) {
break;
}
}
// closing
results.close();
session.close();
for (BufferedWriter bufferedWriter : fileWriters) {
bufferedWriter.close();
}
} catch (IOException | SQLException e) {
throw new UnexpectedJobExecutionException("Error writing partition file", e);
}
}
private ScrollableResults runQuery() {
...
}
}
I'm working on a process that checks and updates data from Oracle database. I'm using hibernate and spring framework in my application.
The application reads a csv file, processes the content, then persiste entities :
public class Main() {
Input input = ReadCSV(path);
EntityList resultList = Process.process(input);
WriteResult.write(resultList);
...
}
// Process class that loops over input
public class Process{
public EntityList process(Input input) :
EntityList results = ...;
...
for(Line line : input.readLine()){
results.add(ProcessLine.process(line))
...
}
return results;
}
// retrieving and updating entities
Class ProcessLine {
#Autowired
DomaineRepository domaineRepository;
#Autowired
CompanyDomaineService companydomaineService
#Transactional
public MyEntity process(Line line){
// getcompanyByXX is CrudRepository method with #Query that returns an entity object
MyEntity companyToAttach = domaineRepository.getCompanyByCode(line.getCode());
MyEntity companyToDetach = domaineRepository.getCompanyBySiret(line.getSiret());
if(companyToDetach == null || companyToAttach == null){
throw new CustomException("Custom Exception");
}
// AttachCompany retrieves some entity relationEntity, then removes companyToDetach and adds CompanyToAttach. this updates relationEntity.company attribute.
companydomaineService.attachCompany(companyToAttach, companyToDetach);
return companyToAttach;
}
}
public class WriteResult{
#Autowired
DomaineRepository domaineRepository;
#Transactional
public void write(EntityList results) {
for (MyEntity result : results){
domaineRepository.save(result)
}
}
}
The application works well on files with few lines, but when i try to process large files (200 000 lines), the performance slows drastically, and i get a SQL timeout.
I suspect cache issues, but i'm wondering if saving all the entities at the end of the processing isn't a bad practice ?
The problem is your for loop which is doing individual saves on the result and thus does single inserts slowing it down. Hibernate and spring support batch inserts and should be done when ever possible.
something like domaineRepository.saveAll(results)
Since you are processing lot of data it might be better to do things in batches so instead of getting one company to attach you should get a list of companies to attach processes those then get a list of companies to detach and process those
public EntityList process(Input input) :
EntityList results;
List<Code> companiesToAdd = new ArrayList<>();
List<Siret> companiesToRemove = new ArrayList<>();
for(Line line : input.readLine()){
companiesToAdd.add(line.getCode());
companiesToRemove.add(line.getSiret());
...
}
results = process(companiesToAdd, companiesToRemove);
return results;
}
public MyEntity process(List<Code> companiesToAdd, List<Siret> companiesToRemove) {
List<MyEntity> attachList = domaineRepository.getCompanyByCodeIn(companiesToAdd);
List<MyEntity> detachList = domaineRepository.getCompanyBySiretIn(companiesToRemove);
if (attachList.isEmpty() || detachList.isEmpty()) {
throw new CustomException("Custom Exception");
}
companydomaineService.attachCompany(attachList, detachList);
return attachList;
}
The above code is just sudo code to point you in the right direction, will need to work out what works for you.
For every line you read you are doing 2 read operations here
MyEntity companyToAttach = domaineRepository.getCompanyByCode(line.getCode());
MyEntity companyToDetach = domaineRepository.getCompanyBySiret(line.getSiret());
You can read more than one line and us the in query and then process that list of companies
Note: I'm specifically not using Fluent NHibernate but am using 3.x's built-in mapping style. However, I am getting a blank recordset when I think I should be getting records returned.
I'm sure I'm doing something wrong and it's driving me up a wall. :)
Background / Setup
I have an Oracle 11g database for a product by IBM called Maximo
This product has a table called workorder which lists workorders; that table has a field called "wonum" which represents a unique work order number.
I have a "reporting" user which can access the table via the maximo schema
e.g. "select * from maximo.workorder"
I am using Oracle's Managed ODP.NET DLL to accomplish data tasks, and using it for the first time.
Things I've Tried
I created a basic console application to test this
I added the OracleManagedClientDriver.cs from the NHibernate.Driver on the master branch (it is not officially in the release I'm using).
I created a POCO called WorkorderBriefBrief, which only has a WorkorderNumber field.
I created a class map, WorkorderBriefBriefMap, which maps only that value as a read-only value.
I created a console application with console output to attempt to write the lines of work orders.
The session and transaction appear to open correct,
I tested a standard ODP.NET OracleConnection to my connection string
The Code
POCO: WorkorderBriefBrief.cs
namespace PEApps.Model.WorkorderQuery
{
public class WorkorderBriefBrief
{
public virtual string WorkorderNumber { get; set; }
}
}
Mapping: WorkorderBriefBriefMap.cs
using NHibernate.Mapping.ByCode;
using NHibernate.Mapping.ByCode.Conformist;
using PEApps.Model.WorkorderQuery;
namespace ConsoleTests
{
public class WorkorderBriefBriefMap : ClassMapping<WorkorderBriefBrief>
{
public WorkorderBriefBriefMap()
{
Schema("MAXIMO");
Table("WORKORDER");
Property(x=>x.WorkorderNumber, m =>
{
m.Access(Accessor.ReadOnly);
m.Column("WONUM");
});
}
}
}
Putting it Together: Program.cs
namespace ConsoleTests
{
class Program
{
static void Main(string[] args)
{
NHibernateProfiler.Initialize();
try
{
var cfg = new Configuration();
cfg
.DataBaseIntegration(db =>
{
db.ConnectionString = "[Redacted]";
db.Dialect<Oracle10gDialect>();
db.Driver<OracleManagedDataClientDriver>();
db.KeywordsAutoImport = Hbm2DDLKeyWords.AutoQuote;
db.BatchSize = 500;
db.LogSqlInConsole = true;
})
.AddAssembly(typeof(WorkorderBriefBriefMap).Assembly)
.SessionFactory().GenerateStatistics();
var factory = cfg.BuildSessionFactory();
List<WorkorderBriefBrief> query;
using (var session = factory.OpenSession())
{
Console.WriteLine("session opened");
Console.ReadLine();
using (var transaction = session.BeginTransaction())
{
Console.WriteLine("transaction opened");
Console.ReadLine();
query =
(from workorderbriefbrief in session.Query<WorkorderBriefBrief>() select workorderbriefbrief)
.ToList();
transaction.Commit();
Console.WriteLine("Transaction Committed");
}
}
Console.WriteLine("result length is {0}", query.Count);
Console.WriteLine("about to write WOs");
foreach (WorkorderBriefBrief wo in query)
{
Console.WriteLine("{0}", wo.WorkorderNumber);
}
Console.WriteLine("DONE!");
Console.ReadLine();
// Test a standard connection below
string constr = "[Redacted]";
OracleConnection con = new OracleConnection(constr);
con.Open();
Console.WriteLine("Connected to Oracle Database {0}, {1}", con.ServerVersion, con.DatabaseName.ToString());
con.Dispose();
Console.WriteLine("Press RETURN to exit.");
Console.ReadLine();
}
catch (Exception ex)
{
Console.WriteLine("Error : {0}", ex);
Console.ReadLine();
}
}
}
}
Thanks in advance for any help you can give!
Update
The following code (standard ADO.NET with OracleDataReader) works fine, returning the 16 workorder numbers that it should. To me, this points to my use of NHibernate more than the Oracle Managed ODP.NET. So I'm hoping it's just something stupid that I did above in the mapping or configuration.
// Test a standard connection below
string constr = "[Redacted]";
OracleConnection con = new Oracle.ManagedDataAccess.Client.OracleConnection(constr);
con.Open();
Console.WriteLine("Connected to Oracle Database {0}, {1}", con.ServerVersion, con.DatabaseName);
var cmd = new OracleCommand();
cmd.Connection = con;
cmd.CommandText = "select wonum from maximo.workorder where upper(reportedby) = 'MAXADMIN'";
cmd.CommandType = CommandType.Text;
Oracle.ManagedDataAccess.Client.OracleDataReader reader = cmd.ExecuteReader();
while (reader.Read())
{
Console.WriteLine(reader.GetString(0));
}
con.Dispose();
When configuring NHibernate, you need to tell it about your mappings.
I found the answer -- thanks to Oskar's initial suggestion, I realized it wasn't just that I hadn't added the assembly, I also needed to create a new mapper.
to do this, I added the following code to the configuration before building my session factory:
var mapper = new ModelMapper();
//define mappingType(s) -- could be an array; in my case it was just 1
var mappingType = typeof (WorkorderBriefBriefMap);
//use AddMappings instead if you're mapping an array
mapper.AddMapping(mappingType);
//add the compiled results of the mapper to the configuration
cfg.AddMapping(mapper.CompileMappingForAllExplicitlyAddedEntities());
var factory = cfg.BuildSessionFactory();
In my current project I have an entity which can be published to other systems. For keeping track on the publications the entity has a relation called "publications". I am using Eclipselink.
This entity bean also has a "PreUpdate" annotated method.
In order to be able to keep the other systems data up to date, I created an Aspect that is executed around the call to the PreUpdate method. Depending on which properties have changed, I need to remove some of the publications. Everything is working absolutely fine.
The problem I am having is that the portal-publishing component correctly sends delete commands and removes the publication from the entities "publications" list. I can even see in the changeset that JPA has noticed the "publications" property to have changed. After the transaction is flushed, the cached entity correctly doesn't have the deleted publications anymore. Unfortunately the database still does and when the system is restarted or the Entity is loaded from the DB again, the publication metadata is there again.
I tried allmost everything. I even managed to get the deleted instances from the JPA ChangeSet in the Aspect and tried to use the entityManager to manually delete them, but nothing actually worked. I seem to be unable to delete these relational entities. Currently I am thinking about using JDBC to delete them, but this would only be my last measure.
#Transactional
#Around("execution(* de.cware.services.truck.model.Truck.jpaPreUpdate(..))")
public Object truckPreUpdate(final ProceedingJoinPoint pjp) throws Throwable {
if (alreadyExecutingMarker.get() != Boolean.TRUE) {
alreadyExecutingMarker.set(Boolean.TRUE);
final Truck truck = (Truck) pjp.getTarget();
final JpaEntityManager jpaEntityManager = (JpaEntityManager) entityManager.getDelegate();
final UnitOfWorkChangeSet changeSet = jpaEntityManager.getUnitOfWork().getCurrentChanges();
final ObjectChangeSet objectChangeSet = changeSet.getObjectChangeSetForClone(truck);
if (log.isDebugEnabled()) {
log.debug("--------------------- Truck pre update check (" + truck.getId() + ") ---------------------");
}
////////////////////////////////////////////////////////////////////////////////////////////////////////////
// If the truck date has changed, revoke all publication copies.
////////////////////////////////////////////////////////////////////////////////////////////////////////////
final ChangeRecord truckFreeDate = objectChangeSet.getChangesForAttributeNamed("lkwFreiDatum");
if (truckFreeDate != null) {
if (log.isDebugEnabled()) {
log.debug("The date 'truckFreeDate' of truck with id '" + truck.getId() + "' has changed. " +
"Revoking all publications that are not marked as main applications");
}
for (final String portal : truck.getPublishedPortals()) {
if (log.isDebugEnabled()) {
log.debug("- Revoking publications of copies to portal: " + portal);
}
portalService.deleteCopies(truck, portal);
// Get any deleted portal references and use the entityManager to finally delete them.
changeSet = jpaEntityManager.getUnitOfWork().getCurrentChanges();
objectChangeSet = changeSet.getObjectChangeSetForClone(truck);
final ChangeRecord publicationChanges = objectChangeSet.getChangesForAttributeNamed("publications");
if (publicationChanges != null) {
if (publicationChanges instanceof CollectionChangeRecord) {
final CollectionChangeRecord collectionChanges =
(CollectionChangeRecord) publicationChanges;
#SuppressWarnings("unchecked")
final Collection<ObjectChangeSet> removedPublications =
(Collection<ObjectChangeSet>)
collectionChanges.getRemoveObjectList().values();
for (final ObjectChangeSet removedPublication : removedPublications) {
final TruckPublication publication = (TruckPublication) ((org.eclipse.persistence.internal.sessions.ObjectChangeSet) removedPublication).getUnitOfWorkClone();
entityManager.remove(publication);
}
}
}
}
}
}
}
Chris
The issue is that PreUpdate is raised during the commit process, when the set of changes have already been computed, and the set of objects to delete have already been computed.
Ideally you would perform something like this in your application logic, not through a persistence event.
You could try executing a DeleteObjectQuery directly from your event (instead of using em.remove()), this may work, but in general it would be better to perform this logic in your application.
jpaEntityManager.getUnitOfWork().deleteObject(object);
Also note that getCurrentChanges() computes the changes, in a PreUpdate event the changes are already computed, so you should be able to use getUnitOfWorkChangeSet().
The only solution I found was to create a new Method for performing the delete and forcing JPA to create a new Transaction. As by this I am losing the changeSet, I had to manually find out which publications were deleted. I then simply call that helper method and the publications are correctly deleted, but I find this solution extremely ugly.
#Transactional
#Around("execution(* de.cware.services.truck.model.Truck.jpaPreUpdate(..))")
public Object truckPreUpdate(final ProceedingJoinPoint pjp) throws Throwable {
if (alreadyExecutingMarker.get() != Boolean.TRUE) {
alreadyExecutingMarker.set(Boolean.TRUE);
final Truck truck = (Truck) pjp.getTarget();
final JpaEntityManager jpaEntityManager = (JpaEntityManager) entityManager.getDelegate();
final UnitOfWorkChangeSet changeSet = jpaEntityManager.getUnitOfWork().getCurrentChanges();
final ObjectChangeSet objectChangeSet = changeSet.getObjectChangeSetForClone(truck);
if (log.isDebugEnabled()) {
log.debug("--------------------- Truck pre update check (" + truck.getId() + ") ---------------------");
}
////////////////////////////////////////////////////////////////////////////////////////////////////////////
// If the truck date has changed, revoke all publication copies.
////////////////////////////////////////////////////////////////////////////////////////////////////////////
final ChangeRecord truckFreeDate = objectChangeSet.getChangesForAttributeNamed("lkwFreiDatum");
if (truckFreeDate != null) {
if (log.isDebugEnabled()) {
log.debug("The date 'truckFreeDate' of truck with id '" + truck.getId() + "' has changed. " +
"Revoking all publications that are not marked as main applications");
}
removeCopyPublications(truck);
}
}
#Transactional(propagation = Propagation.REQUIRES_NEW)
protected void removeCopyPublications(Truck truck) {
// Delete all not-main-publications.
for (final String portal : truck.getPublishedPortals()) {
if (log.isDebugEnabled()) {
log.debug("- Revoking publications of copies to portal: " + portal);
}
final Map<Integer, TruckPublication> oldPublications = new HashMap<Integer, TruckPublication>();
for(final TruckPublication publication : truck.getPublications(portal)) {
oldPublications.put(publication.getId(), publication);
}
portalService.deleteCopies(truck, portal);
for(final TruckPublication publication : truck.getPublications(portal)) {
oldPublications.remove(publication.getId());
}
for (TruckPublication removedPublication : oldPublications.values()) {
if(!entityManager.contains(removedPublication)) {
removedPublication = entityManager.merge(removedPublication);
}
entityManager.remove(removedPublication);
entityManager.flush();
}
}
}
Why doesn't my first version work?
I had a similar problem, I have a class and its children, when I remove the children from the parent they were deleted from the DB, then I attached new children using merge on the class Parent (CascadeType.ALL) using JPA/EclipseLink, then the children didn't create on the DB but in the persistency Motor (JPA). I fix this doing the following:
1- I set shared-cache-mode to NONE in the persistence.xml file
2- When I remove the children, I executed inmediatly this:
public void remove(T entity) {
getEntityManager().remove(getEntityManager().merge(entity));
getEntityManager().getEntityManagerFactory().getCache().evictAll();
}
And that's all. I hope this would help anyone else.
CHECK REFERENCE
http://wiki.eclipse.org/EclipseLink/Examples/JPA/Caching