I need to update thousands of records in the database but i would like to commit after a batch of 5000 records.
#Service
#Transactional (rollbackFor=Throwable.class)
public class AttributeProcessorServiceImpl extends DataLoader implements
AttributeProcessorService
{
.....
private final TransactionTemplate transTemplate;
private final JdbcTemplate jdbcTemplate;
#Autowired private PlatformTransactionManager platformTransactionManager;
#Autowired
public BlockAttributeProcessorServiceImpl(
TransactionTemplate transTemplate,
JdbcTemplate jdbcTemplate,
.....)
{
super();
this.transTemplate = transTemplate;
this.jdbcTemplate=jdbcTemplate;
.....
}
#Async
#Transactional (propagation=Propagation.NOT_SUPPORTED)
public void reloadAttrs()
{
loadAttrs();
updateAttrs();
}
private void loadAttrs()
{
...some data fetching and processing, finally call db update.
updateDbInBatches(rowcount, sql);
}
private void updateAttrs()
{
...some data fetching and processing, finally call db update.
updateDbInBatches(rowcount, sql);
}
private void updateDbInBatches(long rowcount, String sql)
{
DefaultTransactionDefinition def;
boolean hasMore=true;
Integer from;
Integer to = 0;
int batchSize=5000; //gets from property
while (hasMore)
{
from = to+1;
to = batchSize;
def = new DefaultTransactionDefinition();
def.setName("backCommitTx");
def.setPropagationBehavior(TransactionDefinition.PROPAGATION_REQUIRED);
TransactionStatus status = platformTransactionManager.getTransaction(def);
int rows = jdbcTemplate.update(sql,paramValues,paramTypes);
logger.debug("Loaded ["+rows+"] records.");
platformTransactionManager.commit(status);
if (to > rowcount)
{
hasMore=false;
logger.debug("All records ["+rowcount+"] updated.");
}
}
}
}
If I put a breakpoint after loadAttrs(), it shows it loaded bunch of records to the database and issued a commit(), but database does not reflect that commit, until after entire public method completes. How do i ensure data is indeed written to the database after each commit. commit neither gives any error as well.
I missed an important piece of information that solved the problem.
I had another public method which is what was called from outside.
public void reloadAttrs(TransDetail trans)
{
reloadAttrs();
}
Above method was infact using default Transaction Propagation as i did not mention it specifically. Since this was the first public method that was called, spring was ignoring transaction demarcation on next public (async) method that was called. I changed above signature to:
#Transactional (propagation=Propagation.NOT_SUPPORTED)
public void reloadAttrs(TransDetail trans)
{
reloadAttrs();
}
It then worked. I was able to see changes in the database after every commit.
Related
I am exporting a report in my code, I am using HibernateDAOSupport and the method is not annotated with #Transactional. So when the request comes from the UI then automatically Transaction is created and the report is exported in 2 mins.
But when I try to use thread, so I had to put annotation #Transactional, otherwise I get an error of LazyInitialization as no Transcaction is present.
So when I put #Transactional then the same report takes time 2.4 mins. This time keeps increasing and sometimes it take double of the time without #Transactional
I am not sure why it takes time when I put annotation #Transactional
The main class:
CommonDAO
public class CommonDAO extends HibernateDaoSupport
private HibernateTransactionManager txnManager;
public void setTxnManager(HibernateTransactionManager txnManager) {
this.txnManager = txnManager;
}
public List executeSQLQueryPaging(final String hql,
final Object[] params, final Integer[] pagingParam) throws ServiceException {
List results = null;
results = getHibernateTemplate().executeFind(new HibernateCallback() {
public Object doInHibernate(Session session) {
SQLQuery query = session.createSQLQuery(hql);
if (params != null) {
for (int i = 0; i < params.length; i++) {
query.setParameter(i, params[i]);
}
}
query.setFirstResult(pagingParam[0]);
query.setMaxResults(pagingParam[1]);
return query.list();
}
});
return results;
}
ModuleDAO extends CommonDAO
public List getReportData{
executeSQLQueryPaging();
...
return list}
Service
public List getReportData(){
.....
return ModuleDAO.getReportData();
}
If I put #Transactional at service layer then the performance detriorates, or else it is faster if executed from web.
I have a JPA entity with Lazy loaded collection on it. I do not need the collection every time.
#Entity(name = "Foo")
#Access(AccessType.FIELD)
#Table(name = "TEST", schema = "TEST")
public class Foo implements Serializable {
private static final long serialVersionUID = 1L;
#OneToMany(mappedBy="foo", targetEntity=Bar.class, fetch=FetchType.LAZY, cascade=CascadeType.ALL)
private List<Bar> bars;
}
#Entity(name = "Bar")
#Access(AccessType.FIELD)
#Table(name = "TEST", schema = "TEST")
public class Bar implements Serializable {
private static final long serialVersionUID = 1L;
#ManyToOne(targetEntity = Foo.class)
#JoinColumn(name = "FOO_ID", referencedColumnName = "ID")
private Foo foo;
}
I have a few methods on a service class that perform a lot of database interactions and at the end save a Foo entity to the database. I need this to happen for about a 100 items in a collection.
#Service
public class FooService {
#Autowired
private FooRepository fooRepository;
public void processAllFoos() {
fooRepository.findAll().forEach(foo -> {
processFoo(foo);
});
}
private void processFoo(Foo foo) {
foo.getBars().forEach(bar -> {
// Do a lot of time consuming stuff here that involves
// entities of other types and modify each bar object
});
fooRepository.save(foo);
}
}
processAllFoos gets called from a #RESTController whenever it gets a request.
However, I do not want processAllFoos to be wrapped in a single database transaction, because that locks up the entire Foo table till the business logic is executed for all Foos.
If I make the processFoo method #Transactional I get the LazyInitializationException which complains that the Hibernate session is non-existent. To make this work I need to make all methods in the call stack #Transactional so that the nested methods can join onto the calling method's transaction. But this locks the entire Foo table as mentioned above.
Adding a OpenSessionInViewFilter for the dispatcher servlet solves my problem but I've read that there are issues with performance and entity detaching/reattaching (which I do in other parts of the application) with this approach.
Is there a way I can do what I want to without using the OpenSessionInView approach? What other vulnerabilities am I adding by using this approach?
Spring/Hibernate 4.x
Based on the answer below, I was able to do the following:
#Service
public class FooService {
#Autowired
private FooRepository fooRepository;
#Autowired
private TransactionTemplate transactionTemplate;
public void processAllFoos() {
fooRepository.findAll().forEach(foo -> {
transactionTemplate.execute(new TransactionCallback<Object>() {
public Object doInTransaction(TransactionStatus status) {
try {
processFoo(foo);
status.flush();
} catch(Exception e) {
status.setRollbackOnly();
}
return null;
}
});
});
}
private void processBar(Foo foo) {
foo.getBars().foreEach(bar -> {
// Do a lot of time consuming stuff here that involves
// entities of other types and modify each bar object
});
fooRepository.save(foo);
}
}
OpenSessionInViewFilter commonly used to solve LazyInitialization problem in View layer (UI components or page templates), because View layer can't and must not manage transactions directly.
In your case another way to get all the Bar objects can be applied.
First You get all the Foo object ids instead to get fully objects.
Second Use Foo ids collection to iterate thru related Bar objects.
Third If you don't want one BIG transaction then you can use Spring Transaction template to manage transactions explicitly.
Your code example may look like this:
#Service
public class FooService {
#Autowired
private FooRepository fooRepository;
#Autowired
private BarRepository barRepository;
#Autowired
private TransactionTemplate transactionTemplate;
public void processAllFoos() {
final List < Long > fooIdList = transactionTemplate.execute(new TransactionCallback() {
public Object doInTransaction(TransactionStatus status) {
return fooRepository.findIdList();
}
});
transactionTemplate.execute(new TransactionCallback() {
public Object doInTransaction(TransactionStatus status) {
barRepository.findByFooIdList(fooIdList).forEach(bar - > {
processBar(bar);
});
return null;
}
});
}
private void processBar(Bar bar) {
// Do a lot of time consuming stuff here that involves
// entities of other types and modify each bar object
barRepository.save(bar);
}
}
Example below shows how to solve your task without some performance overheads. But you should understand that if Foo and Bar tables linked with foreign key constraint, then related record in Foo table may be blocked by RDBMS each time you update row in Bar table.
so i basically want to process a HTTP-Post with a Controller in Spring and send back a result for the User AND after that i want to make a database call.
So here is my example:
#Controller
public class AngebotController extends WebMvcConfigurerAdapter {
#Autowired
private DatabaseUtils dbUtil;
#Autowired
private MyMailSender mailSender;
#RequestMapping(value = REQUEST_PATH, method = RequestMethod.POST)
public String doPost(#Valid FormInput input, BindingResult bindingResult) {
// .. some input validations here
// after the validation is complete i will have accesss to a object, that i just created, just like the following
// Lets say that this object holds important values for the database query
final MyObject validatedInput = new MyObject();
// Start a new Thread to do the remaining work (the Database Call)
Thread t = new Thread() {
#Override
public void run() {
// so i am starting the database query with the information that i just validated above
// That Database Query will Return a List of Items based on the given MyObject
// This Query will take a long time and i dont want the user to wait, because this data is not nessessary for the user
List<Item> items = dbUtil.getStuffByInput(validatedInput);
for(Item i : items) {
// Now i just want to send some informations about the item via email, this part works
mailSender.sendMail("mail#mail.mail", i);
}
}
}.start();
return "viewname";
}
}
#Service
public class DatabaseUtils {
#Autowired
private ItemRepository repository;
public List<Item> getStuffByInput(MyObject o) {
List<Item> items = repository.findAllByMyObject(o);
// Doing some more stuff with those items here ..
return items;
}
}
// The Implementation will be generated by Spring
public interface ItemRepository extends CrudRepository<Item, Long> {
// will select all Items by comparing the myObject with each Item
// This also works like intended
public List<Item> findAllByMyObject(MyObject myObject);
}
So where is my Problem?
The only Problem i have is, that the Database Query will end throwing an Exception, because the Database Connection was closed (i guess by Spring)
The Exception: Exception in thread "Thread-6" org.hibernate.SessionException: Session is closed!
Any Help appreciated.
Thanks!
Model structure:
#MappedSuperclass
public class BaseModel<K extends Comparable> implements Serializable, Comparable<Object> {
private static final long serialVersionUID = 1L;
#Id
private K id;
#Version
private Integer version;
// getter/setter
}
#Entity
public class MyEntity extends BaseModel<String> {
// some fields and it's getter/setter
}
Record in my database for my_entity:
id: 1
version: 1
...
Below is my update method:
void update(String id, Integer currentVersion, ....) {
MyEntity myEntity = myRepository.findOne(id);
myEntity.setVersion(currentVersion);
// other assignments
myRepository.save(myEntity);
}
Below is the query being fired when this method is invoked.
update my_entity set version=?, x=?, y=?, ...
where id=? and version=?
I am expecting OptimisticLockException when currentVersion passed in above method is other than 1.
Can any body help me why I am not getting OptimisticLockException?
I am using spring-boot for my webmvc project.
Section 11.1.54 of the JPA specification notes that:
In general, fields or properties that are specified with the Version
annotation should not be updated by the application.
From experience, I can advise that some JPA providers (OpenJPA being one) actually throw an exception should you try to manually update the version field.
While not strictly an answer to your question, you can re-factor as below to ensure both portability between JPA providers and strict compliance with the JPA specification:
public void update(String id, Integer currentVersion) throws MyWrappedException {
MyEntity myEntity = myRepository.findOne(id);
if(currentVersion != myEntity.getVersion()){
throw new MyWrappedException();
}
myRepository.save(myEntity);
//still an issue here however: see below
}
Assuming your update(...) method is running in a transaction however you still have an issue with the above as section 3.4.5 of the JPA specification notes:
3.4.5 OptimisticLockException Provider implementations may defer writing to the database until the end of the transaction, when
consistent with the lock mode and flush mode settings in effect. In
this case, an optimistic lock check may not occur until commit time,
and the OptimisticLockException may be thrown in the "before
completion" phase of the commit. If the OptimisticLockException must
be caught or handled by the application, the flush method should be
used by the application to force the database writes to occur. This
will allow the application to catch and handle optimistic lock
exceptions.
Essentially then, 2 users can submit concurrent modifications for the same Entity. Both threads can pass the initial check however one will fail when the updates are flushed to the database which may be on transaction commit i.e. after your method has completed.
In order that you can catch and handle the OptimisticLock exception, your code should then look something like the below:
public void update(String id, Integer currentVersion) throws MyWrappedException {
MyEntity myEntity = myRepository.findOne(id);
if(currentVersion != myEntity.getVersion()){
throw new MyWrappedException();
}
myRepository.save(myEntity);
try{
myRepository.flush()
}
catch(OptimisticLockingFailureException ex){
throw new MyWrappedException();
}
}
Use EVICT before updating when using JPA. I did not get the #Version to work either. The property was increased but no exception was thrown when updating an object that had the wrong version-property.
The only thing I have got to work is to first EVICT the object and then save it. Then the HibernateOptimisticLockingException is thrown if the Version properties does not match.
Set the hibernates ShowSQL to 'true' to verify that the actual update sql ends with "where id=? and version=?". If the object is not evicted first, the update statement only has "where id=?", and that will (for obvious reasons) not work.
Optimistic hibernation lock works out of the box (You don't must put a version for Entity):
#Entity
#Table(name = "product")
public class Product {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
private Long quantity;
private Long likes;
#Version
private Long version;
public Product() {
}
//setter and getter
//equals and hashcode
repository
public interface ProductRepository extends JpaRepository<Product, Long> {}
service
#Service
public class ProductOptimisticLockingService {
private final ProductRepository productRepository;
public ProductOptimisticLockingService(ProductRepository productRepository) {
this.productRepository = productRepository;
}
#Transactional(readOnly = true)
public Product findById(Long id, String nameThread){
Product product =
productRepository
.findById(id)
.get();
System.out.printf(
"\n Select (%s) .... " +
"(id:) %d | (likes:) %d | (quantity:) %d | (version:) %d \n",
nameThread,
product.getId(),
product.getLikes(),
product.getQuantity(),
product.getVersion()
);
return product;
}
#Transactional(isolation = Isolation.READ_COMMITTED)
public void updateWithOptimisticLocking(Product product, String nameThread) {
try {
productRepository.save(product);
} catch (ObjectOptimisticLockingFailureException ex) {
System.out.printf(
"\n (%s) Another transaction is already working with a string with an ID: %d \n",
nameThread,
product.getId()
);
}
System.out.printf("\n--- Update has been performed (%s)---\n", nameThread);
}
}
test
#SpringBootTest
class ProductOptimisticLockingServiceTest {
#Autowired
private ProductOptimisticLockingService productService;
#Autowired
private ProductRepository productRepository;
#Test
void saveWithOptimisticLocking() {
/*ID may be - 1 or another. You must put the ID to pass in your methods. You must think how to write right your tests*/
Product product = new Product();
product.setLikes(7L);
product.setQuantity(5L);
productRepository.save(product);
ExecutorService executor = Executors.newFixedThreadPool(2);
Lock lockService = new ReentrantLock();
Runnable taskForAlice = makeTaskForAlice(lockService);
Runnable taskForBob = makeTaskForBob(lockService);
executor.submit(taskForAlice);
executor.submit(taskForBob);
executorServiceMethod(executor);
}
/*------ Alice-----*/
private Runnable makeTaskForAlice(Lock lockService){
return () -> {
System.out.println("Thread-1 - Alice");
Product product;
lockService.lock();
try{
product = productService
.findById(1L, "Thread-1 - Alice");
}finally {
lockService.unlock();
}
setPause(1000L); /*a pause is needed in order for the 2nd transaction to attempt
read the line from which the 1st transaction started working*/
lockService.lock();
try{
product.setQuantity(6L);
product.setLikes(7L);
update(product,"Thread-1 - Alice");
}finally {
lockService.unlock();
}
System.out.println("Thread-1 - Alice - end");
};
}
/*------ Bob-----*/
private Runnable makeTaskForBob(Lock lockService){
return () -> {
/*the pause makes it possible to start the transaction first
from Alice*/
setPause(50L);
System.out.println("Thread-2 - Bob");
Product product;
lockService.lock();
try{
product = findProduct("Thread-2 - Bob");
}finally {
lockService.unlock();
}
setPause(3000L); /*a pause is needed in order for the 1st transaction to update
the string that the 2nd transaction is trying to work with*/
lockService.lock();
try{
product.setQuantity(5L);
product.setLikes(10L);
update(product,"Thread-2 - Bob");
}finally {
lockService.unlock();
}
System.out.println("Thread-2 - Bob - end");
};
}
private void update(Product product, String nameThread){
productService
.updateWithOptimisticLocking(product, nameThread);
}
private Product findProduct(String nameThread){
return productService
.findById(1L, nameThread);
}
private void setPause(long timeOut){
try {
TimeUnit.MILLISECONDS.sleep(timeOut);
} catch (InterruptedException e) {
e.printStackTrace();
}
}
private void executorServiceMethod(ExecutorService executor){
try {
executor.awaitTermination(10L, TimeUnit.SECONDS);
} catch (InterruptedException e) {
e.printStackTrace();
}
executor.shutdown();
}
}
I am doing some tests to understand the behaviour of #Transactional in Spring 3. Though, it is not working as I would expect. If have one method with Propagation.REQUIRED calling another with Propagation.REQUIRES_NEW, will the second method be able to retrieve from the DB the data inserted by the first method?
EDITED:
I AM seeing uncommitted changed in a #Transaction, here is my (nasty looking) code.
#Service
public class FeedManager {
#Autowired
JdbcTemplate jdbcTemplate;
#Transactional(isolation = Isolation.READ_COMMITTED, propagation = Propagation.REQUIRED)
public boolean createFeed(Feed feed, boolean anonymizeIt) {
String query = "INSERT INTO feed (name, url, is_active) values (?, ?, ?)";
int rowsAffected = jdbcTemplate.update(query, feed.getName(), feed.getUrl(), feed.isActive());
boolean success = (rowsAffected == 1);
if (anonymizeIt) {
success = success && this.anonymizeFeedName(feed);
}
return success;
}
#Transactional(isolation = Isolation.READ_COMMITTED, propagation = Propagation.REQUIRES_NEW)
public boolean anonymizeFeedName(Feed feed) {
String query = "UPDATE feed set name = ? where name = ?";
int rowsAffected = jdbcTemplate.update(query, feed.getName() + (new Date()).toString(), feed.getName());
boolean success = (rowsAffected == 1);
return success;
}
}
#RunWith(SpringJUnit4ClassRunner.class)
#ContextConfiguration("classpath:mrpomario/springcore/jdbc/jdbc-testenv-config.xml")
public class TransactionalTest {
#Autowired
FeedManager feedManager;
Feed feed;
#Before
public void setup() {
feed = new Feed("RSS", "http://www.feedlink.com", true);
}
#Test
public void test_Create() {
assertTrue(feedManager.createFeed(feed, false));
}
#Test
public void test_Anonymize() {
assertTrue(feedManager.anonymizeFeedName(feed));
}
#Test
public void test_Create_And_Anonymize() {
Feed feedo = new Feed("AnotherRSS", "http://www.anotherfeedlink.com", true);
assertTrue(feedManager.createFeed(feedo, true));
}
}
It should not be able to see any changes made by the first method (as long as your isolation level is READ COMMITTED or above).
If you get different results, make sure that #Transactional actually takes effect. In particular, make sure that you don't call another #Transactional method of the same class - due to limitations of Spring proxy-based AOP model transactional aspect is applied only to calls that come from the outside of the class.
See also:
7.6.1 Understanding AOP proxies