Update millions of database row using Spring Data JPA - spring-boot

I'm wondering on which is the best for this
Here is my Entity:
#Entity
#Data
public class User {
#GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
private String firstName;
private String middleName;
private String lastName;
private Status status;
private String statusRemarks;
}
Option 1 Direct bulk update in UserRepository:
Does this will affect my database performance if the users to be update will reach to millions in number?
public interface UserRepository extends CrudRepository<User, Long> {
#Query("UPDATE u FROM User u set u.status=:status, u.statusRemarks=:statusRemarks where u.status in :statuses")
void bulkUpdateByStatuses(Status status,String statusRemarks,Status... statuses);
}
Option 2 will fetch the users by status and will update it one by one like this:
I'm pretty sure this will affect the performance of the MS due to the memory usage
public void bulkUpdateUserByStatuses(final UserBulkUpdateDto userbulkUpdateDto){
List<User> toUpdateUsers = userRepository.findByStatuses(userbulkUpdateDto.getStatuses())
for(final User user: toUpdateUsers){
user.setStatus(userbulkUpdateDto.getNewStatus());
user.setStatusRemarks(userbulkUpdateDto.getStatusRemarks());
userRepository.save(user);
}
}
}

Absolutely option 1.
Both options read the affected data, modify it and write it back.
The option 2 on top of that reads and writes the data across the network and turns it into java objects.
So it will be slower and probably even put a higher load on the database.
If the load still becomes to high for the database you should consider breaking the update in smaller ones, for example by limiting to a range of ids at a time.

Related

Bulk data to find exists or not : Spring Data JPA

I get an Post request that would give me a List<PersonApi> Objects
class PersonApi {
private String name;
private String age;
private String pincode ;
}
And I have an Entity Object named Person
#Entity
#Table(name = "person_master")
public class Person{
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
Long id;
#Column(name = "name")
String name;
#Column(name = "age")
String age;
#Column(name = "pincode ")
String pincode ;
}
My record from Post request would look something like this (pseudocode representation of the data below)
[
"Arun","33","09876gh"
"James","34","8765468"
]
I need to do a bulk-validation using Spring JPA.. Give the List<PersonApi> and get a True or False based on the condition that all the entries in the PersonApi objects list should be there in the database.
How to do this ?
The selected answer is not a right one. (not always right)
You are selecting the whole database to check for existence. Unless your use case is very special, i.e. table is very small, this will kill the performance.
The proper way may start from issuing repository.existsById(id) for each Person, if you never delete the persons, you can even apply some caching on top of it.
exists
Pseudo Code:
List<PersonApi> personsApiList = ...; //from request
List<Person> result = personRepository.findAll();
in your service class you can access your repository to fetch all database entities and check if your list of personapi's is completeley available.
boolean allEntriesExist = result.stream().allMatch(person -> personsApiList.contains(createPersonApiFromPerson(person)));
public PersonApi createPersonApiFromPerson(Person person){
return new PersonApi(person.getName(), person.getAge(), person.getPincode());
}

Sorting on #Transient column in Spring Data Rest via PagingAndSortingRepository

Our application uses PagingAndSortingRepository to serve our REST API. This works great, but we ran into a specific edge case that we can't seem to solve:
We have a alphanumeric field that has to be sortable (e.g. SOMETHING-123). One possible solution was to use something like a regex inside the database query's order by. This was ruled out, as we wanted to stay database independant. Thus we split up the column into two columns.
So before we had an Entity with 1 String field:
#Entity
public class TestingEntity {
#Id
#GeneratedValue
private long id;
private String alphanumeric
}
And now we have an Entity with 2 additional fields and the old field made #Transient which is filled at #PostLoad:
#Entity
public class Testing {
#Id
#GeneratedValue
private long id;
#Transient
public String alphanumeric;
#PostLoad
public void postLoad(){
this.alphanumeric = this.alphabetic + "-" + this.numeric;
}
public void setAlphanumeric(String alphanumeric) {
int index = alphanumeric.indexOf("-");
this.alphabetic = alphanumeric.substring(0, index);
this.numeric = Long.parseLong(alphanumeric.substring(index + 1));
}
#JsonIgnore
private String alphabetic;
#JsonIgnore
private Long numeric;
}
This is working great and the additional fields do not get exposed. However the sorting on the field "alphanumeric" does obviously not work anymore. The simplest solution would be to make this request:
localhost:8080/api/testingEntity?sort=alphanumeric,asc
and internally rewrite it to the working request:
localhost:8080/api/testingEntity?sort=alphabetic,asc&sort=numeric,asc
What is the best way to tackle this issue?

Data base pessimistic locks with Spring data JPA (Hibernate under the hood)

I need some help with pessimistic entity locking. I'm using PostgreSQL and Spring data JPA (hibernate 5 under the hood) in my application. So, I want to show a task I've faced with.
I have some user accounts with money:
#lombok.Data //Used to generate getters and setters
#Entity
class AccountEntity {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
private Integer id;
private Long balance;
}
And payments, that allows to transfer money from one account to another
#lombok.Data
#Entity
class PaymentEntity {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
private Integer id;
private Long amount;
#OneToOne
#JoinColumn(name="account_from_id")
private AccountEntity accountFrom;
#OneToOne
#JoinColumn(name="account_to_id")
private AccountEntity accountTo;
}
Repositories for both:
interface AccountRepository extends JpaRepository<AccountEntity, Integer> {}
interface PaymentRepository extends JpaRepository<PaymentEntity, Integer> {}
And a service where money transfer performed:
interface PaymentService {
PaymentEntity create(PaymentEntity paymentEntity);
}
#Service
class PaymentServiceImpl implements PaymentService {
#Autowired
private AccountRepository accountRepository;
#Autowired
private PaymentRepository paymentRepository;
#Transactional
#Override
public PaymentEntity create(PaymentEntity payment) {
AccountEntity from = accountRepository.getOne(payment.getAccountFrom().getId()); //want to lock account here
AccountEntity to = accountRepository.getOne(payment.getAccountTo().getId()); //want to lock account here
Long newFromBalance = from.getBalance() - payment.getAmount();
Long newToBalance = to.getBalance() + payment.getAmount();
if (newFromBalance < 0)
throw new RuntimeException("Not enough money for payment");
from.setBalance(newFromBalance);
to.setBalance(newToBalance);
PaymentEntity result = paymentRepository.save(payment); //create payment
accountRepository.save(from); //update account
accountRepository.save(to); //update account
return result; //want to unlock both accounts here
}
}
So, as you can see from code, I want to lock two accounts, involved to the money transfer at the beginning of the transaction (which will guarantee, that they both can't be changed before the transaction commits, and each new payment will get updated balance of the accounts). At the end of the transaction I want to unlock them both.
I've read documentation, and as I and as I understood, for do that I need to update my AccountRepository:
interface AccountRepository extends JpaRepository<AccountEntity, Integer> {
#Lock(LockModeType.PESSIMISTIC_WRITE)
AccountEntity save(AccountEntity account);
}
The code above must lock from and to accounts at the beginning of the transaction and automatically unlock them at the end. But I still have some questions:
In this case data base locks will be used (not the application one on the java side), because I use multiple instances of the application, that works with the same database. Isn't it?
Annotation #Lock(LockModeType.PESSIMISTIC_WRITE) can be applied only to repository method (not to the service one). Isn't it?
#Lock will lock all entities in a transaction, for which select was performed (from and to accounts in my case). Isn't it?
All locked entities will be unlocked automatically when the transaction commits or rollbacks. Isn't it?
Each payment, which will use account, locked with pessimistic write lock will wait until account will be updated and lock will be released. Isn't it?
I will be very thankful if you provide links to official documentation or just some site, where I can discover this topic by myself.

Spring data mongodb #DBRef list

I am trying to have a list in a model using #DBRef but I can't get it to work.
This is my User model:
#Data
#Document
public class User {
#Id
#JsonSerialize(using = ToStringSerializer.class)
private ObjectId id;
#Indexed(unique = true)
#NotBlank
private String email;
#NotBlank
private String name;
#NotBlank
private String password;
#DBRef
private List<Server> servers;
}
Server model:
#Data
#Document
public class Server {
#Id
#JsonSerialize(using = ToStringSerializer.class)
private ObjectId id;
#NotBlank
private String name;
#NotBlank
private String host;
}
The structure is very simple, every user can have multiple servers. But when I add servers to the user the server is created, but the servers array contains one null entry("servers" : [ null ]). So the server isn't added to the user. This is how I create a server and add it to an user:
#PostMapping
public Mono create(#Valid #RequestBody Server server, Mono<Authentication> authentication) {
return this.serverRepository.save(server).then(authentication.flatMap(value -> {
User user = (User) value.getDetails();
user.getServers().add(server);
return userRepository.save(user);
})).map(value -> server);
}
So I simply create and save a server, add the server the user and then save the user. But it doesn't work. I keep having an array with one null entry.
I've seen this page: http://www.baeldung.com/cascading-with-dbref-and-lifecycle-events-in-spring-data-mongodb. But it is for saving the child document, not for linking it. Also it is for a single document, not for an array or list.
Why is my list not being saved correctly?
All my libraries are coming from spring boot version 2.0.0.M6.
UPDATE
When removing #DBRef from the user's servers property the servers are getting saved, but they of course get double created, in the server collection and in every user.servers. So the error has something to do with references.
After some googling I found the answer...
https://jira.spring.io/browse/DATAMONGO-1583
https://jira.spring.io/browse/DATAMONGO-1584
Reactive mongo doesn't support this.
Actually there is a way to resolve DbRefs without to using the blocking driver. Yes - the references are resolved in a blocking fashion, but does not require a second connection. In order to achieve this we have to write our own DbRefResolver: NbDbRefResolver.java. In the provided resolver there is a flag: RESOLVE_DB_REFS_BY_ID_ONLY. If is switched on will not going to resolve the DbRefs from the database, but instead will resolve them to fake objects with id only. It is up to implementation to fill the references later in non-blocking fashion.
If the flag RESOLVE_DB_REFS_BY_ID_ONLY is set to false it will eagerly resolve the references by using the non-blocking driver, but will block the execution until the references are resolved.
Here is how to register the DbRefResolver in the app: DbConfig.kt
Files attached are provided here: https://jira.spring.io/browse/DATAMONGO-1584
Me did it like that for roles :
#Unwrapped(onEmpty = Unwrapped.OnEmpty.USE_NULL)
private Collection<Role> roles;
you can check the doc (2021) here : https://spring.io/blog/2021/04/20/what-s-new-in-spring-data-2021-0

I need help for persisting into oracle database

There is a problem about generating id while persisting into database.
I added the following code to my jpa entity file, however I'm getting 0 for personid.
#Id
#Column(unique=true, nullable=false, precision=10, name="PERSONID")
#SequenceGenerator(name="appUsersSeq", sequenceName="SEQ_PERSON", allocationSize=1)
#GeneratedValue(strategy=GenerationType.SEQUENCE, generator = "appUsersSeq")
private long personid;
EjbService:
#Stateless
public class EjbService implements EjbServiceRemote {
#PersistenceContext(name = "Project1245")
private EntityManager em;
#Override
public void addTperson(Tperson tp) {
em.persist(tp);
}
}
0 is default value for long type. The id will be set after invoking select query for the related sequence, which commonly is executed when you persist the entity. Are you persisting the entity? In case yes, post the database sequence definition to check it.

Resources