In my Spring boot app I'm deleting and inserting a large amount of data into my MySQL db in a single transaction. Ideally, I want to only commit the results to my database at the end, so all or nothing. I'm running into issues where my deletions will be committed before my insertions, so during that period any calls to the db will return no data (not good). Is there a way to manually commit transaction?
My main logic is:
#Transactional
public void saveParents(List<Parent> parents) {
parentRepo.deleteAllInBatch();
parentRepo.resetAutoIncrement();
//I'm setting the id manually before hand
String sql = "INSERT INTO parent " +
"(id, name, address, number) " +
"VALUES ( ?, ?, ?, ?)";
jdbcTemplate.batchUpdate(sql, new BatchPreparedStatementSetter() {
#Override
public void setValues(PreparedStatement ps, int i) throws SQLException {
Parent parent = parents.get(i);
ps.setInt(1, parent.getId());
ps.setString(2, parent.getName());
ps.setString(3, parent.getAddress());
ps.setString(4, parent.getNumber());
}
#Override
public int getBatchSize() {
return parents.size();
}
});
}
ParentRepo
#Repository
#Transactional
public interface ParentRepo extends JpaRepository<Parent, Integer> {
#Modifying
#Query(
value = "alter table parent auto_increment = 1",
nativeQuery = true
)
void resetAutoIncrement();
}
EDIT:
So I changed
parentRepo.deleteAllInBatch();
parentRepo.resetAutoIncrement();
to
jdbcTemplate.update("DELETE FROM output_stream");
jdbcTemplate.update("alter table output_stream auto_increment = 1");
in order to try avoiding jpa's transaction but each operation seems to be committing separately no matter what I try. I have tried TransactionTemplate and implementing PlatformTransactionManager (seen here) but I can't seem to get these operations to commit together.
EDIT: It seems the issue I was having was with the alter table as it will always commit.
I'm running into issues where my deletions will be committed before my insertions, so during that period any calls to the db will return no data
Did you configure JPA and JDBC to share transactions?
If not, then you're basically using two different mechanisms to access the data (EntityManager and JdbcTempate), each of them maintaining a separate connection to the database. What likely happens is that only EntityManager joins the transaction created by #Transactional; the JdbcTemplate operation executes either without a transaction context (read: in AUTOCOMMIT mode) or in a separate transaction altogether.
See this question. It is a little old, but then again, using JPA and Jdbc together is not exactly a common use case. Also, have a look at the Javadoc for JpaTransactionManager.
Related
I need the ability to check if Entity belongs to the current ongoing transaction. I have a JPA entity cache as a thread-local cache. Almost always when there is single #Transaction there is no issue when a same thread calls multiple times to save on service and NOT nested #Trasaction(Propagation.REQUIRES_NEW) (meaning when you have nested transactions) on same thread it does not work. Is there any way to check if JPA Entity ( MyExpesiveEntity) belongs to the current ongoing transaction?
ThreadLocal<Map<Long,MyExpesiveEntity>> cache = new ThreadLocal<Map<Long,MyExpesiveEntity>>()
#Entity('MyExpesiveEntity')
class MyExpesiveEntity{
}
#Transaction
save(DTO save){
MyExpesiveEntity entity = cache.get(dto.getID());
if(entity == null){
entity = myExpesiveRespository.findById(dto.getID());
cache.get().put(entity.getID(),entity);
}
entity.setXXX(dto.getXXX())
// THIS always works but Sometime when the caller is using
// Propagation.REQUIRES_NEW.
// It does not work when you have nested transactions.It throws duplicated id.
myExpesiveRespository.saveAndFlush(entity);
}
It's Hibernate specific, but yes, this is possible
public boolean containsEntity(EntityManager em, Class<?> entityClass, Object id) {
SessionImplementor session = em.unwrap(SessionImplementor.class);
EntityKey entityKey = session.generateEntityKey((Serializable) id, session.getFactory().getEntityPersister(entityClass.getName()));
PersistenceContext pc = session.getPersistenceContext();
return pc.getEntity(entityKey) != null || pc.getProxy(entityKey) != null;
}
Since the entity is always existing you could simply update based on the ID. See Modifying
#Modifying
#Query("update MyExpesiveEntity expEnt set expEnt.xxxx = ?1 where expEnt.id = ?2")
void setXxxById(String xxxx, Integer id);
I am using spring transactions and hibernate to insert data into the oracle database table:
Here is the scenarion I am facing problem with:
I have two tables which have one to one mapping in hibernate. And I am inserting data in these two tables using below method calls.Transaction propagates from one method to another.So that insertion of data in both the tables happens in one transaction.
Problem: is that , while inserting data in second table, if an exception like "constraintvoilationexception---can not insert null into a particular column", is thrown,....then ideally the data should not be inserted in any of the tables i.e transaction should roll back,,,...But this is not happening ...when an exception is thrown while inserting data in the second table....records do get inserted in the first table, which ideally should not happen i.e whole transaction should be rolled back...
Can you please help,...where I am wrong while applying #Transactional , or there is some other reason for this ( may be from database side, not sure though)
#Transactional(readOnly = false, propagation = Propagation.REQUIRED)
public void methodA(){
// inserting data in table 1;
methodB();
}
#Transactional(readOnly = false, propagation = Propagation.REQUIRED)
public void methodB{
// inserting data in table 2;
}
Defines zero (0) or more exception classes, which must be subclasses of Throwable, indicating which exception types must cause a transaction rollback. Details Here
#Transactional(readOnly = false, propagation = Propagation.REQUIRED,rollbackFor = Exception.class)
public void methodA(){
try{
// inserting data in table 1;
methodB();
}
catch(Exception ex)
{
}
}
public void methodB{
// inserting data in table 2;
}
I'm performing an update via a method using Hibernate and the EntityManager.
This update method is called multiple times (within a loop).
It seems like when I execute it the first time, it locks the table and does not free it.
When trying to update the table via SQL Developer after having closed the application, I see the table is still locked because the update is hanging.
What do you see as a solution to this problem? If you need more information, let me know.
Class
#Repository
#Transactional(propagation = REQUIRES_NEW)
public class YirInfoRepository {
#Autowired
EntityManager entityManager;
#Transactional(propagation = REQUIRES_NEW)
public void setSent(String id) {
String query = "UPDATE_QUERY";
Query nativeQuery = entityManager.createNativeQuery(String.format(query, id));
nativeQuery.executeUpdate();
}
}
UPDATE
After having waited more than one hour, I launched the application again and it worked fine once but now again, it hangs.
UPDATE 2 -- I'll give a maximum bounty to whoever helps me solve this
On another place I use an application managed entity manager and it still gives me the same type of errors.
public void fillYirInfo() {
File inputFile = new File("path");
try (InputStream inputStream = new FileInputStream(inputFile);
BufferedReader bufferedReader = new BufferedReader(new InputStreamReader(inputStream))) {
bufferedReader.lines().skip(1).limit(20).forEach(line -> {
String[] data = line.split(",");
String rnr = data[0];
String linked = data[1];
String email = data.length > 2 ? data[2] : "";
String insuredId = insuredPeopleRepository.getInsuredIdFromNationalId(rnr);
int modifiedCounter = 0;
if (!isNullOrEmpty(insuredId)) {
EntityManager entityManager = emf.createEntityManager();
EntityTransaction transaction = entityManager.getTransaction();
Query nativeQuery = entityManager.createNativeQuery(
"QUERY"
);
transaction.begin();
nativeQuery.executeUpdate();
entityManager.flush();
transaction.commit();
entityManager.close();
}
System.out.println(modifiedCounter + " rows modified");
});
} catch (IOException e) {
e.printStackTrace();
}
}
Try without an update-query:
#Repository
#Transactional(propagation = REQUIRES_NEW)
public class YirInfoRepository {
#Autowired
EntityManager entityManager;
#Transactional(propagation = REQUIRES_NEW)
public void setSent(String id) {
//guessing your class name and method..
final YirInfo yirInfo = entityManager.find(YirInfo.class, id);
yirInfo.setSent();
}
}
Might not be as fast as a single update query, but it's possible to get it reasonably fast, unless the amount of data is huge. This is the preferred way of using Hibernate/JPA, instead of thinking in terms of single values and SQL queries, you work with entities/objects and (sometimes) HQL/JPQL queries.
You are using #Transactional annotation. This means you are using Spring Transaction. Then in your UPDATE 2 you are using transaction by yourself and managed by spring (I guess it's another project or class not managed by Spring).
In any case what I would do is to try to update your records in single spring transaction and I'd not use #Transactional in DAO layer but in service layer. Something like this:
Service layer:
#Service
public class YirInfoService {
#Autowired
YirInfoRepository dao;
#Transactional(propagation = REQUIRES_NEW)
public void setSent(List < String > ids) {
dao.setSents(ids);
}
}
DAO layer:
#Repository
public class YirInfoRepository {
#Autowired
EntityManager entityManager;
//Here you can update by using and IN statement or by doing a cycle
//Let's suppose a bulk operation
public void setSents(List < String > ids) {
String query = "UPDATE_QUERY";
for (int i = 0; i < ids.size(); i++) {
String id = ids.get(i);
Query nativeQuery = entityManager.createNativeQuery(String.format(query, id));
nativeQuery.executeUpdate();
if (i % 20 == 0) {
entityManager.flush();
entityManager.clear();
}
}
}
}
The first thing you have to understand is that for the first example, you are using a native query to update rows in the DB. In this case you are completely skipping Hibernate to do anything for you.
In your second example, you have the same thing, you are updating via an update query. You don't need to flush the entity manager as it's only necessary for transferring the pending changes made to your entity objects within that entity manager.
Plus I don't know how your example works as you are autowiring the entity manager and not using the #PersistenceContext annotation. Make sure you use this one properly because you might have misconfigured the application. Also there is no need to manually create the entity manager when using Spring as it looks in the second example. Just use #PersistenceContext to get an entity manager in your app.
You are also mixing up transaction management. In the first example, it's enough if you put the #Transactional annotation to either of your method or to the class.
For the other example, you are doing manual transaction management which makes no sense in this case. If you are using Spring, you can simply rely on declarative transaction management.
The first thing I'd check here is to integrate datasource-proxy into your connection management and log out how your statements are executed. With this info, you can make sure that the query is sent to the DB side and the DB is executing it very slowly, or you are having a network issue between your app and db.
If you find out that the query is sent properly to the DB, you want to analyze your query, because most probably it's just executed very slowly and needs some optimizations. For this, you can use the Explain plan feature, to find out how your execution plan looks like and then make it faster.
So I am trying to configure Hibernate for batch processing purposes. I have built a sample application and configured according to Hibernates docs.
But after configuring Hibernate to log out the SQL, it looks like it is not performing a batch insert at all but merely individual inserts. Am I reading this log wrong?
So I have the following properties configured in my Spring Boot app.
spring.jpa.properties.hibernate.jdbc.batch_size=10
spring.jpa.properties.hibernate.show_sql=true
spring.jpa.properties.hibernate.format_sql=true
Here is my very basic Batch Writer..
#Transactional
public class BatchWriter {
private EntityManager entityManager;
private int batchSize;
public BatchWriter(EntityManager entityManager, int batchSize) {
this.entityManager = entityManager;
this.batchSize = batchSize;
}
public void writeBatchOfCustomers(int numOf) {
for(long i = 0; i <= numOf; i++) {
Customer customer = new Customer(i);
entityManager.persist(customer);
if ( i % batchSize == 0 ) { //20, same as the JDBC batch size
//flush a batch of inserts and release memory:
entityManager.flush();
entityManager.clear();
}
}
}
}
Now I am running this to insert 20 Customers for example and in the hibernate log I am seeing the following 20 times:
Hibernate:
insert
into
customer
(first_name, last_name, id)
values
(?, ?, ?)
What am I missing here?
It is currently using Spring Boot auto configuration with H2 database. I will however be looking to use it with Spring Batch and an Oracle db eventually, which will be inserting around 30k objects with about 35 attributes.
Any help appreciated.
Thanks,
So it appears that the hibernate SQL logging is rather misleading (in my opinion).
My configuration was in fact batch processing.
I added a logger with level DEBUG for this class:
org.hibernate.engine.jdbc.batch.internal.BatchingBatch
as currently named in Hibernate version 5.0.12. (Think it is named something else previously).
In this class you can see the it is in fact batch processing.
I'm using SimpleJdbcDaoSupport object to access DB resources. I have a query which is frequently executed against the database to locate a record with a specific key. for some reason after executing the same query several times I start to get an empty result even though the record exists in the database.
Any ideas what can cause this behavior?
daoSupport.getJdbcTemplate().query(this.getConsumerTokenQueryStatement(),params, this.rowMapper);
public static class TokenServicesRowMapper implements RowMapper {
public Object mapRow(ResultSet rs, int rowNum) throws SQLException {
DefaultLobHandler lobHandler = new DefaultLobHandler();
return lobHandler.getBlobAsBytes(rs, 1);
}
}
If this is not related to your code one reason can be the fact that another transaction is doing something (like an update) to the row you search and due do the isolation between transactions you cannot see your row. One transaction can change but not commit your row yet while in the same time the other one is searching for it but as it can only see committed rows it does not see your row.