I am working in Spring Boot framework.
I have a working way to run batches of "insert ignore" but i don't fully understand how/why it works.
I am using the Persistable interface to have inserts be done in batches.
In addition i'm using SQLInsert to run insert ignore instead of insert.
The code below demonstrates it:
#Entity
#SQLInsert(sql = "insert ignore into my_table (created_at, data, id) VALUES (?, ?, ?)")
public class MyTable implements Persistable<String> {
#Id
#Column
private String id;
#Column
private String data;
#Column
private Timestamp createdAt;
#Id
#Column
private String data;
public String getId() {
calculateId();
return id;
}
#Override
public boolean isNew() {
return true;
}
}
===============
#Repository
public interface MyTableRepository extends JpaRepository<MyTable, String> {
#Transactional
#Modifying(clearAutomatically = true, flushAutomatically = true)
<S extends MyTable> List<S> saveAll(Iterable<S> entities);
}
In debug, and on the DB side, i see that the save is indeed done in batches.
The query shown in debug mode is of this sort:
["insert ignore into my_table (created_at, data, id) VALUES (?, ?, ?)"], Params:[(2022-06-08 17:44:35.041,data1,id1),(2022-06-08 17:44:35.042,data2,id2),(2022-06-08 17:44:35.042,data3,id3)]
I see that there are 3 new records created in the DB.
The question how does it work if the number of "?" is smaller than the numbers of parameters passed?
If you would try it directly on the DB you would get an error
Related
I have this entity with these fields and this primary key:
#Getter
#Setter
#Entity
#Builder
#NoArgsConstructor
#AllArgsConstructor
#Table(name = "MY_TABLE", schema = "MY_SCHEME")
public class MyEntity{
#Id
#Column(name = "ID")
private String id;
#Column(name = "NAME")
private String name;
#Column(name = "DESCRIPTION")
private String description;
}
I'm experiencing some undesirable behavior.If I try to insert the exact same data, I was expecting a primary key violation because it already exists, but I'm finding that what it's doing is an upsert. I am extending my repository from JpaRepository and using the save method:
#Repository
public interface MyJpaRepository extends JpaRepository<MyEntity, String> {
}
In my service:
...
this.repository.save(myEntity);
...
The database is Oracle, and if I launch the insert manually in SQL developer, the data violation occurs.
That could be happening?
Based on source code of save:
public <S extends T> S save(S entity) {
Assert.notNull(entity, "Entity must not be null.");
if (entityInformation.isNew(entity)) { //it checks if entity is new based on id. i.e. insert operation.
em.persist(entity);
return entityx
} else {
return em.merge(entity); // just merging i.e. in your case doing upsert.
}
}
So currently save method works as expected; but if you really want to avoid upsert behaviour you might want to try using existsById; something like below:
if(!repository.existsById(..)){
repository.save(..);
}
For H2 database in my spring boot api test there is an insertion script that looks something like:
INSERT INTO STORES_TEMPLATE (ID, COUNTRY, TEMPLATE_NAME_1, STORES_1) VALUES(STORES_TEMPLATE_ID_SEQ.NEXTVAL,'country', 'template_name', 'stores');
INSERT INTO STORES_TEMPLATE (ID, COUNTRY, TEMPLATE_NAME_2, STORES_2) VALUES(STORES_TEMPLATE_ID_SEQ.NEXTVAL,'country', 'template_name', 'stores');
INSERT INTO STORES_TEMPLATE (ID, COUNTRY, TEMPLATE_NAME_3, STORES_3) VALUES(STORES_TEMPLATE_ID_SEQ.NEXTVAL,'country', 'template_name', 'stores');
The entity:
#Entity
#Getter
#Setter
#NoArgsConstructor
#Table(name = "STORES_TEMPLATE")
public class StoresTemplate {
#Id
#GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "STORES_TEMPLATE_ID_SEQ")
#SequenceGenerator(name = "STORES_TEMPLATE_ID_SEQ", sequenceName = "STORES_TEMPLATE_ID_SEQ", allocationSize = 1)
private Long id;
#Enumerated(EnumType.STRING)
private CountryEnum country;
private String templateName;
#Lob
private String stores;
public void setStores(List<String> stores) {
this.stores = String.join(",", stores);
}
#JsonIgnore
public List<String> getStoresAsList() {
return Arrays.stream(stores.split(","))
.distinct()
.collect(Collectors.toList());
}
}
On production oracle database everything works just fine, but on my test environment with h2 database when I try to save new entity the sequence generator gives me ids that are duplicated with the ids of the predefined by the sql migration data. What can I do to fix this issue?
What did the trick was to exclude the insertion script for the tests, but is there another possible solution?
I am trying to run some code just before Updating or Saving. I have in my entity:
#Data
#EqualsAndHashCode(callSuper=false)
#Table(name="file_management", uniqueConstraints = { #UniqueConstraint(columnNames = {"name"})})
#Entity
public class FileManagement {
#Id
#GeneratedValue(strategy= GenerationType.IDENTITY)
private long id;
#Column(name="name")
private String name;
#Column(name = "SEARCH_STRING", length = 1000)
#Getter #Setter
private String searchString;
#PreUpdate
#PrePersist
void updateSearchString() {
final String fullSearchString = StringUtils.join(Arrays.asList(
name),
" ");
this.searchString = StringUtils.substring(fullSearchString, 0, 999);
}
}
I have in my FileManagementRepository:
#Transactional
#Modifying
#Query("UPDATE FileManagement SET name = :name WHERE id = :id")
public void updateFile(long id, String name);
#Transactional
#Modifying
#Query(value = "INSERT INTO file_management (name, filename, created_by, create_date, is_active, is_deleted) VALUES (:name, :filename, :createdBy, :createdDate, :isActive, :isDeleted)", nativeQuery = true)
public void createFileWithFilename(String name, String filename, String createdBy, Date createdDate, boolean isActive, boolean isDeleted);
and in my FileManagementService.java
public void updateFileWithFilename(String id, String name) {
fileManagementRepository.updateFile(Long.parseLong(id), name);
}
public boolean createFileWithFilename(String name, String filename, String createdBy, Date createdDate, boolean isActive, boolean isDeleted) {
fileManagementRepository.createFileWithFilename(name, filename, createdBy, createdDate, isActive, isDeleted);
return true;
}
But the problem is the updateSearchString() method is not called when I update or insert a new row. The search_string column is (null)
Please help. Thanks.
I am guessing that entities that are inserted or updated using native SQL or JPQL in this way will by pass the persistent context and persistence context will not manage them. However , #PreUpdate and #PrePersist only work for the entities that are managed by persistence context , so your #PreUpdate and #PrePersist will not execute for them.
I think you should insert and update the entities in a more JPA way which ensure persistence context will manage them:
#Service
public class FileManagementService{
#Autowired
private FileManagementRepository fileManagementRepository;
#Transactional
public void updateFileWithFilename(String id, String name) {
Optional<FileManagement> file= fileManagementRepository.findById(id);
if(file.isPresent()){
file.get().setName(name);
}else{
throw new RuntimeException("Record does not exist");
}
}
#Transactional
public void createFileWithFilename(String name, String filename, String createdBy, Date createdDate, boolean isActive, boolean isDeleted) {
FileManagement file= new FileManagement(name,fileName,........);
fileManagementRepository.save(file);
}
}
I have the following oracle table..
create table post (id number(10,0) not null, text varchar2(255 char), title varchar2(255 char), update_date date, version number(10,0), primary key (id));
My entity looks something like this..
#Entity
#EntityListeners(AuditingEntityListener.class)
public class Post {
#Id Integer id;
String title;
String text;
#LastModifiedDate #Temporal(TemporalType.TIMESTAMP) Date updateDate;
#Version Integer version;
....
}
The repository is simply this...
public interface PostRepository extends JpaRepository<Post, Integer>{
}
The following test fails...
#RunWith(SpringRunner.class)
#SpringBootTest
public class PostRepositoryTest {
#Autowired protected JdbcTemplate jdbcTemplate;
#Autowired PostRepository postRepo;
private static final Integer TEST_POST_ID = -1;
private static final String TEST_TEXT = "This is the text.";
#Before
public void setup() {
String insertPostSql = "insert into POST (ID, TITLE, TEXT, UPDATE_DATE, VERSION) values (?, ?, ?, ?, ?)";
jdbcTemplate.update(insertPostSql, new Object[]{TEST_POST_ID, "Title 1.", TEST_TEXT, new Date(), 0});
}
#After
public void teardown() {
String deletePostSql = "delete from POST where ID = ?";
jdbcTemplate.update(deletePostSql, new Object[]{TEST_POST_ID});
}
#Test
public void testUpdateA() throws Exception {
Post post = postRepo.findById(TEST_POST_ID).get();
post.setTitle(TEST_TEXT+" Amendment.");
Post updatedPost = postRepo.save(post);
assertNotEquals(post.getVersion(), updatedPost.getVersion());
Post updatedPost2 = postRepo.save(updatedPost);
assertEquals(updatedPost.getVersion(), updatedPost2.getVersion());
}
}
The second update to the entity is persisted to the database even though nothing has changed. If I change the type of the update date column in the database to TIMESTAMP, the test succeeds. Unfortunately I can't do this, is there something else I can do as the date in the returned instance from the save call is not equal to the date in the database according to JPA dirty check.
Thanks in advance.
To resolve this I ended up implementing my own #PreUpdate method instead of using #LastModifiedDate and within this method I set the update date to an instance of timestamp and set the nano seconds to 0.
#PreUpdate
public void onPreUpdate(Post post){
Timestamp ts = new Timestamp(System.currentTimeMillis()); ts.setNanos(0);
post.setUpdateDate(ts);
}
I have a list of 4 million generated entities that I want to move into table. The entity have field with type LocalDateTime:
#Data
#AllArgsConstructor
#Entity
#Table(name = "invoices")
public class Invoice {
#Id
#GeneratedValue(strategy=GenerationType.IDENTITY)
Long id;
#Column(name = "exact_iss_time")
private LocalDateTime exactIssueTime;
#Column(name = "final_iss_time")
private LocalDateTime finalIssueTime;
#Column(name = "issuer")
private String issuer;
#Column(name = "groupid")
private Integer groupID;
protected Invoice() {
}
}
As it is a big number of entities I want to do it optimally - which I gues is with NamedParameterJdbcTemplate.batchUpdate() like this:
public int[] bulkSaveInvoices(List<Invoice> invoices){
String insertSQL = "INSERT INTO invoices VALUES (:id, :exactIssueTime, :finalIssueTime, :issuer, :groupID)";
SqlParameterSource[] sqlParams = SqlParameterSourceUtils.createBatch(invoices.toArray());
int[] insertCounts = namedParameterJdbcTemplate.batchUpdate(insertSQL, sqlParams);
return insertCounts;
}
However I keep getting error:
org.springframework.jdbc.BadSqlGrammarException: PreparedStatementCallback; bad SQL grammar [INSERT INTO invoices VALUES (?, ?, ?, ?, ?)]; nested exception is org.postgresql.util.PSQLException: Can't infer the SQL type to use for an instance of java.time.LocalDateTime. Use setObject() with an explicit Types value to specify the type to use.
Am I doing this right - if so how to fix this LocalDateTime issue in this case?
If this is the wrong way than what is the most optimal way to INSERT the 4 million generated test entities into the table with Spring-boot?
database?
JPA 2.1 was released before Java 8 and therefore doesn’t support the new Date and Time API.
You can try add convertor, for more check How to persist LocalDate and LocalDateTime with JPA
#Converter(autoApply = true)
public class LocalDateTimeAttributeConverter implements AttributeConverter<LocalDateTime, Timestamp> {
#Override
public Timestamp convertToDatabaseColumn(LocalDateTime locDateTime) {
return (locDateTime == null ? null : Timestamp.valueOf(locDateTime));
}
#Override
public LocalDateTime convertToEntityAttribute(Timestamp sqlTimestamp) {
return (sqlTimestamp == null ? null : sqlTimestamp.toLocalDateTime());
}
}
Or you can try not jdbc approach, but use custom JPA batch