How to create a pagination with the output of the number of duplicates? - spring

I have a db that stores an archive of messages, my goal is to get the username and the number of messages that match a certain pattern, then page it all out in the browser. The table has just under 2 million rows and will keep getting bigger. The problem is that the output of the number of messages is incorrect and it corresponds (as I understand it) to the size #PageableDefault. I currently use Pageable.unpaged() to work around the problem. Is there a way to load a limited number of users per page, but all the posts associated with them? This would help speed up the loading of the final table.
Archive Entity:
#Entity
#AllArgsConstructor
#NoArgsConstructor
public class Archive {
private String username;
private String txt;
private long id;
private Timestamp createdAt;
#Basic
#Column(name = "username", nullable = false, length = 191)
public String getUsername() {
return username;
}
public void setUsername(String username) {
this.username = username;
}
#Basic
#Column(name = "txt", nullable = true, length = -1)
public String getTxt() {
return txt;
}
public void setTxt(String txt) {
this.txt = txt;
}
#Id
#GeneratedValue(strategy = GenerationType.AUTO)
#Basic
#Column(name = "id", nullable = false)
public long getId() {
return id;
}
public void setId(long id) {
this.id = id;
}
#Basic
#Column(name = "created_at", nullable = false)
#DateTimeFormat(pattern = "yyyy-MM-dd HH:mm:ss")
public Timestamp getCreatedAt() {
return createdAt;
}
public void setCreatedAt(Timestamp createdAt) {
this.createdAt = createdAt;
}
Repo:
Page<Archive> findByUsernameContainsAndCreatedAtGreaterThanEqualAndCreatedAtLessThanEqualAndTxt(String username, Timestamp createdAt, Timestamp createdAt2, String txt, Pageable pageable);
Service:
public Page<Report> getOrgData(String username, Timestamp from, Timestamp to, Status status, #PageableDefault(size = 15) Pageable pageable) {
pageable = Pageable.unpaged();
String statusString = getStatusString(status);
Page<Archive> find = repo.findByUsernameContainsAndCreatedAtGreaterThanEqualAndCreatedAtLessThanEqualAndTxt(
username,
from,
to,
statusString,
pageable
);
return getReport(find);
Formation of the final table:
public Page<Report> getReport(Page<Archive> archive) {
LinkedHashMap<String, Long> resultMap = new LinkedHashMap<>();
List<Report> report = new ArrayList<>();
archive.forEach(a -> resultMap.put(a.getUsername(), resultMap.getOrDefault(a.getUsername(), 0L) + 1L));
resultMap.forEach((key, value) -> report.add(new Report(key, value)));
return new PageImpl<>(report);
}

Related

JPA: How can I read particular fields of an Entity?

I use Spring JPA ( Hibernate ) and have bunch of entities which are mapped onto tables.
When I use an entity to write I need many fields in it (see an example below). But when I read, I wanna sometimes read only particular fields like first/last name. How can I perform it using Spring data JPA ? ( because due to CrudRepository nature it returns the whole entity)
#Entity
#Table(name="PERSON")
#AttributeOverride(name = "id", column = #Column(name = "ID_PERSON"))
public class Person extends BaseEntity implements Serializable {
private static final long serialVersionUID = 1L;
#Column(name="LAST_NAME", length = 100, nullable = false)
private String lastName;
#Column(name="FIRST_NAME", length = 50, nullable = false)
private String firstName;
#Column(name="MIDDLE_NAME", length = 50)
private String middleName;
#Column(name="BIRTHDAY", nullable = false)
#Temporal(value = TemporalType.DATE)
private Date birthday;
#ManyToOne(fetch = FetchType.LAZY, optional = true)
#JoinColumn(name = "ID_SEX")
private Sex sex;
public Person() {
super();
}
public String getLastName() {
return lastName;
}
public void setLastName(String lastName) {
this.lastName = lastName;
}
public String getFirstName() {
return firstName;
}
public void setFirstName(String firstName) {
this.firstName = firstName;
}
public String getMiddleName() {
return middleName;
}
public void setMiddleName(String middleName) {
this.middleName = middleName;
}
public Date getBirthday() {
return birthday;
}
public void setBirthday(Date birthday) {
this.birthday = birthday;
}
public Sex getSex() {
return sex;
}
public void setSex(Sex sex) {
this.sex = sex;
}
}
There are various possibilities.
With Spring Data JPA you can use projection (that's the name when you only select certain fields/columns of an entity/table).
You can return List of Object[] or a DTO or an Interface.
For example with interface it looks like this:
interface NamesOnly {
String getFirstname();
String getLastname();
}
interface PersonRepository extends Repository<Person, UUID> {
Collection<NamesOnly> findByLastname(String lastname);
}
As you can see the return value most not be of type Person.
Please check out the documentation:
https://docs.spring.io/spring-data/jpa/docs/current/reference/html/#projections
I was faced with a similar issue and I resorted to this:
Let's say you have your entity FooEntity related to repository FooRepository
To only get certain fields, let's say firstName and lastName using key I had to create a custom query in the FooRepository
In this manner
#Query("select new FooEntity(f.firstName, f.lastName) from FooEntity f where f.key = :key")
Optional<FooEntity> findCustomByKey(#Param("key") BigInteger key);
You also have to ensure that the FooEntity has the constructor accepting the values that you only want to be set or returned in this manner:
public FooEntity(String firstName, String lastName){
// Ensure the constructor is not called with null values
notNull(firstName, "Method called with null parameter (firstName)");
notNull(lastName, "Method called with null parameter (lastName)");
this.firstName = firstName;
this.lastName = lastName;
}
Please the full code below:
public class FooEntity implements Serializable {
#Id
#Column(name = "key")
private BigInteger key;
#Column(name = "first_name")
private String firstName;
#Column(name = "last_name")
private String lastName;
#Column(name = "birth_date")
private Date birthDate;
#Column(name = "hash")
private String hash;
public FooEntity(String firstName, String lastName){
// Ensure the constructor is not called with null values
notNull(firstName, "Method called with null parameter (firstName)");
notNull(lastName, "Method called with null parameter (lastName)");
this.firstName = firstName;
this.lastName = lastName;
}
// Getters and Setters
}
public interface FooRepository extends JpaRepository<FooEntity, BigInteger>{
#Query("select new FooEntity(f.firstName, f.lastName) from FooEntity f where f.key = :key")
Optional<FooEntity> findCustomById(#Param("key") BigInteger key); // This one only returns two set fields firstName and LastName and the rest as nulls
Optional<FooEntity> findById(BigInteger key) // This one returns all the fields
}

Null Foreign Key (Springboot, Hibernate, Postman)

I am using Springboot with Hibernate and I would like to save a new “post” using a POST request to my database. One thing that I would like to highlight is that I am using the dependency “spring-boot-starter-data-rest”.
Schema of the database (MySQL):
Class User:
#Entity
#Table(name="user")
public class User implements Serializable {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
#Column(name="id", nullable = false)
public int id;
#OneToMany(mappedBy = "user_id_fk")
public Set<Post> posts;
#Column(name="email")
private String email;
#Column(name="username")
private String username;
#Column(name="password")
private String password;
#Column(name="first_name")
private String firstName;
#Column(name="last_name")
private String lastName;
#Column(name="create_time")
protected Date createTime;
#Column(name="type")
private String accountType;
public User() {
this.createTime = new java.util.Date();
}
public User(String email, String username, String password, String firstName, String lastName, Date createTime, String accountType) {
this.email = email;
this.username = username;
this.password = password;
this.firstName = firstName;
this.lastName = lastName;
this.createTime = createTime;
this.accountType = accountType;
this.createTime = new java.util.Date();
}
public User(int id, String email, String username, String password, String firstName, String lastName, Date createTime, String accountType) {
this.id = id;
this.email = email;
this.username = username;
this.password = password;
this.firstName = firstName;
this.lastName = lastName;
this.createTime = createTime;
this.accountType = accountType;
this.createTime = new java.util.Date();
}
Plus the Getters & Setters & toString().
Class Post:
#Entity
#Table(name="post")
public class Post implements Serializable {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
#Column(name="id")
public int id;
#ManyToOne(optional = false)
#JoinColumn(name = "user_id_fk", nullable = false)
public User user_id_fk;
#Column(name="comment")
private String comment;
#Column(name="likes")
private int likes;
#Column(name="dislike")
private int dislike;
#Column(name="create_time")
protected Date createTime;
public Post() {
this.createTime = new java.util.Date();
}
public Post(String comment, int likes, int dislike, User user_id_fk) {
this.user_id_fk = user_id_fk;
this.comment = comment;
this.likes = likes;
this.dislike = dislike;
this.createTime = new java.util.Date();
}
public Post(int id, User user_id_fk, String comment, int likes, int dislike) {
this.id = id;
this.user_id_fk = user_id_fk;
this.comment = comment;
this.likes = likes;
this.dislike = dislike;
this.createTime = new java.util.Date();
}
Plus the Getters & Setters & toString().
Post request (I'm using Postman to send the request):
{
"comment" : "This is a comment",
"likes" : 123,
"dislike" : 1,
"user_id_fk" :
[
{
"id" : 1
}
]
}
In the request at the "user_id_fk" I tried with [ {"id" : 1 } ] and with { "id" : 1 } but the result was the same.
Issue:
When I am executing exactly the same code from my controller everything works are excepted. Bear in mind that I am using the dependency “spring-boot-starter-data-rest”.
Also, when I am executing the code without the “optional = false” and “nullable = false” is inserting the data into the database but the “user_id_fk” is null :(.
The error that I am getting:
not-null property references a null or transient value : com.citizen.citizen.entity.Post.user_id_fk;
nested exception is org.hibernate.PropertyValueException: not-null property references a null or transient value : com.citizen.citizen.entity.Post.user_id_fk]
That means that the foreign key ("user_id_fk") is null but should not be null.
Any help will be greatly appreciated.
I just remove the dependency "spring-boot-starter-data-rest" and I solved the issue by creating my custom rest and everything works. Kisses!
According to this article, you should make user_id_fk nullable and then:
Send POST to create User
Send second POST to create Post
Send PUT to create a relation between the two.
This article states the same.
And the documentation only mentions handling associations via association links.

Spring Boot test #After not deleting rows before next test

I have the following #Before and #After in my Spring Boot integration tests:
#Before
public void setup() {
this.mockMvc = MockMvcBuilders.webAppContextSetup(this.wac)
.addFilter(springSecurityFilterChain).build();
user = userRepository.save(
new User("Joe", "Bloggs", "joe#example.com", "joe", passwordEncoder.encode("secret")));
currency = currencyRepository.save(
new Currency("GBP", "£%01.2f"));
fund = fundRepository.save(
new Fund("Nationwide", (double) 100, currency));
}
#After
public void teardown() {
userRepository.delete(user);
currencyRepository.delete(currency);
fundRepository.delete(fund);
}
However, it doesn't seem that currencies are being deleted after each test and my tests are failing in error:
...
[ERROR] testGetFunds_whenNoToken_thenUnauthorized(biz.martyn.budget.FundsControllerTest) Time elapsed: 3.268 s <<< ERROR!
org.springframework.orm.jpa.JpaObjectRetrievalFailureException: Unable to find biz.martyn.budget.currency.Currency with id 437; nested exception is javax.persistence.EntityNotFoundException: Unable to find biz.martyn.budget.currency.Currency with id 437
Caused by: javax.persistence.EntityNotFoundException: Unable to find biz.martyn.budget.currency.Currency with id 437
...
After, if I query the test database, I see that rows haven't been deleted:
mysql> select * from currencies;
+----+---------------------+---------------------+---------------+------+---------------------+
| id | created_at | deleted_at | format | name | updated_at |
+----+---------------------+---------------------+---------------+------+---------------------+
...
| 437 | 2020-01-02 13:51:24 | 2020-01-02 13:51:23 | £%01.2f | GBP | 2020-01-02 13:51:24 |
...
+----+---------------------+---------------------+---------------+------+---------------------+
5 rows in set (0.00 sec)
There should only be one unique entry for name but I guess as delete is not happening it is pulling duplicates for "GBP". My repository for currencies:
Currency.java
#Entity(name = "currencies")
#SQLDelete(sql = "UPDATE currencies SET deleted_at = now() WHERE id = ?")
#Where(clause = "deleted_at is null")
public class Currency {
#Id
#GeneratedValue(strategy = GenerationType.AUTO)
protected Integer id;
#Column(unique = true, nullable = false)
private String name;
#Column(nullable = false)
private String format;
#Column(name = "created_at", updatable = false)
#CreationTimestamp
protected LocalDateTime createdAt;
#Column(name = "updated_at")
#UpdateTimestamp
protected LocalDateTime updatedAt;
#Column(name = "deleted_at")
protected LocalDateTime deletedAt;
protected Currency() {}
public Currency(String name, String format) {
this.name = name;
this.format = format;
}
public Integer getId() {
return id;
}
public String getName() {
return name;
}
public void setName(final String name) {
this.name = name;
}
public String getFormat() {
return format;
}
public void setFormat(final String format) {
this.format = format;
}
}
User.java
#Entity(name = "users")
public class User implements Serializable {
/**
*
*/
private static final long serialVersionUID = -8507204786382662588L;
#Id
#GeneratedValue(strategy = GenerationType.AUTO)
private Long id;
#Column(nullable = false)
private String firstName;
#Column(nullable = false)
private String surname;
#Column(nullable = false, unique = true)
private String email;
#Column(nullable = false, unique = true)
private String username;
#Column(nullable = false)
#JsonIgnore
private String password;
#OneToMany(cascade = CascadeType.ALL, orphanRemoval = true)
#JoinColumn(name="user_id")
#JsonIgnore
private List<Fund> funds;
protected User() {}
public User(String firstName, String surname, String email, String username, String password) {
this.firstName = firstName;
this.surname = surname;
this.email = email;
this.username = username;
this.password = password;
}
public Long getId() {
return id;
}
public String getUsername() {
return username;
}
public void setUsername(String username) {
this.username = username;
}
public String getPassword() {
return password;
}
public void setPassword(String password) {
this.password = password;
}
// standard getters and setters
public String getFirstName() {
return firstName;
}
public void setFirstName(String firstName) {
this.firstName = firstName;
}
public String getSurname() {
return surname;
}
public void setSurname(String surname) {
this.surname = surname;
}
public String getEmail() {
return email;
}
public void setEmail(String email) {
this.email = email;
}
public List<Fund> getFunds() {
return funds;
}
public void addFund(Fund fund) {
funds.add(fund);
fund.setUser(this);
}
public void removeFund(Fund fund) {
funds.remove(fund);
fund.setUser(null);
}
// public Fund getFund(int id) {
// fundRepository.findByIdAndUserId(id)
// .orElseThrow(() -> new EntityNotFoundException("Fund ID not found: "+id));
// }
}
The delete method takes an object of Currency. Your Currency object has an ID which is auto-generated.
When you pass the same object that you passed to save to delete you did not set the ID that's the reason the delete operation never actually deletes the data you wanted to delete.
You can either use the object that is returned from save method or get a hold of the genearted-id and use deleteById method.
Here is an example for delete using the object.
#Before
public void setup() {
user = repository.save(user);
}
#After
public void tearDown() {
repository.delete(user);
}
Alternatively, you can use the same object to get the generated-id and use deleteById method.
If you take a look at SimpleJpaRepository which provides an implementation for JPA, you will find that if your ID is null then they are treated as a new entity and are never deleted.
/*
* (non-Javadoc)
* #see org.springframework.data.repository.CrudRepository#delete(java.lang.Object)
*/
#Override
#Transactional
#SuppressWarnings("unchecked")
public void delete(T entity) {
Assert.notNull(entity, "Entity must not be null!");
if (entityInformation.isNew(entity)) {
return;
}
// Other stuff
}

Multi column search using Specifications Spring Data Jpa within associated entity?

I am taking this question Perform multi column search on Date, Integer and String Data type fields of Single Table? and This method must return a result of type Specification<Employee> in Java 8 further ahead.
Actually I wanted to search within association entity as well as a part of global search. Will that be possible using JPA 2 Specifications API ?
I've Employee and Department #OneToMany bi-directional relationship.
Employee.java
#Data
#Builder
#AllArgsConstructor
#NoArgsConstructor
#Entity
public class Employee implements Serializable {
private static final long serialVersionUID = 1L;
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
#Column(name = "EMPLOYEE_ID")
private Long employeeId;
#Column(name = "FIRST_NAME")
private String firstName;
#Column(name = "LAST_NAME")
private String lastName;
#Column(name = "EMAIL_ID")
private String email;
#Column(name = "STATUS")
private String status;
#Column(name = "BIRTH_DATE")
private LocalDate birthDate;
#Column(name = "PROJECT_ASSOCIATION")
private Integer projectAssociation;
#Column(name = "GOAL_COUNT")
private Integer goalCnt;
#ManyToOne(fetch = FetchType.LAZY)
#JoinColumn(name = "DEPT_ID", nullable = false)
#JsonIgnore
private Department department;
}
Department.java
#Data
#Builder
#AllArgsConstructor
#NoArgsConstructor
#Entity
public class Department implements Serializable {
private static final long serialVersionUID = 1L;
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
#Column(name = "DEPT_ID")
private Long departmentId;
#Column(name = "DEPT_NAME")
private String departmentName;
#Column(name = "DEPT_CODE")
private String departmentCode;
#OneToMany(fetch = FetchType.LAZY, mappedBy = "department")
#JsonIgnore
private Set<Employee> employees;
}
and I saved Data like below.
MyPaginationApplication.java
#SpringBootApplication
public class MyPaginationApplication implements CommandLineRunner {
public static void main(String[] args) {
SpringApplication.run(MyPaginationApplication.class, args);
}
#Autowired
private EmployeeRepository employeeRepository;
#Autowired
private DepartmentRepository departmentRepository;
#Override
public void run(String... args) throws Exception {
saveData();
}
private void saveData() {
Department department1 = Department.builder()
.departmentCode("AD")
.departmentName("Boot Depart")
.build();
departmentRepository.save(department1);
Employee employee = Employee.builder().firstName("John").lastName("Doe").email("john.doe#gmail.com")
.birthDate(LocalDate.now())
.goalCnt(1)
.projectAssociation(2)
.department(department1)
.build();
Employee employee2 = Employee.builder().firstName("Neha").lastName("Narkhede").email("neha.narkhede#gmail.com")
.birthDate(LocalDate.now())
.projectAssociation(4)
.department(department1)
.goalCnt(2)
.build();
Employee employee3 = Employee.builder().firstName("John").lastName("Kerr").email("john.kerr#gmail.com")
.birthDate(LocalDate.now())
.projectAssociation(5)
.department(department1)
.goalCnt(4)
.build();
employeeRepository.saveAll(Arrays.asList(employee, employee2, employee3));
}
}
EmployeeController.java
#GetMapping("/employees/{searchValue}")
public ResponseEntity<List<Employee>> findEmployees(#PathVariable("searchValue") String searchValue) {
List<Employee> employees = employeeService.searchGlobally(searchValue);
return new ResponseEntity<>(employees, HttpStatus.OK);
}
EmployeeSpecification.java
public class EmployeeSpecification {
public static Specification<Employee> textInAllColumns(Object value) {
return (root, query, builder) -> builder.or(root.getModel().getDeclaredSingularAttributes().stream()
.filter(attr -> attr.getJavaType().equals(value.getClass()))
.map(attr -> map(value, root, builder, attr))
.toArray(Predicate[]::new));
}
private static Object map(Object value, Root<?> root, CriteriaBuilder builder, SingularAttribute<?, ?> a) {
switch (value.getClass().getSimpleName()) {
case "String":
return builder.like(root.get(a.getName()), getString((String) value));
case "Integer":
return builder.equal(root.get(a.getName()), value);
case "LocalDate":
return builder.equal(root.get(a.getName()), value);//date mapping
default:
return null;
}
}
private static String getString(String text) {
if (!text.contains("%")) {
text = "%" + text + "%";
}
return text;
}
}
When I hit the /employees/{searchValue}, I want searching to be happened in Department Table along with Employee table (may be using Joins something like that). Is that possible ? If yes, how can we do that ?
Or:
Will this be good approach to put like here? Got reference from Using #Query
#Query("SELECT t FROM Todo t WHERE " +
"LOWER(t.title) LIKE LOWER(CONCAT('%',:searchTerm, '%')) OR " +
"LOWER(t.description) LIKE LOWER(CONCAT('%',:searchTerm, '%'))")
List<Todo> findBySearchTerm(#Param("searchTerm") String searchTerm);
Any pointers?
If you take a look at my post actually I have a solution for join
#Override
public Specification<User> getFilter(UserListRequest request) {
return (root, query, cb) -> {
query.distinct(true); //Important because of the join in the addressAttribute specifications
return where(
where(firstNameContains(request.search))
.or(lastNameContains(request.search))
.or(emailContains(request.search))
)
.and(streetContains(request.street))
.and(cityContains(request.city))
.toPredicate(root, query, cb);
};
}
private Specification<User> firstNameContains(String firstName) {
return userAttributeContains("firstName", firstName);
}
private Specification<User> lastNameContains(String lastName) {
return userAttributeContains("lastName", lastName);
}
private Specification<User> emailContains(String email) {
return userAttributeContains("email", email);
}
private Specification<User> userAttributeContains(String attribute, String value) {
return (root, query, cb) -> {
if(value == null) {
return null;
}
return cb.like(
cb.lower(root.get(attribute)),
containsLowerCase(value)
);
};
}
private Specification<User> cityContains(String city) {
return addressAttributeContains("city", city);
}
private Specification<User> streetContains(String street) {
return addressAttributeContains("street", street);
}
private Specification<User> addressAttributeContains(String attribute, String value) {
return (root, query, cb) -> {
if(value == null) {
return null;
}
ListJoin<User, Address> addresses = root.joinList("addresses", JoinType.INNER);
return cb.like(
cb.lower(addresses.get(attribute)),
containsLowerCase(value)
);
};
}
private String containsLowerCase(String searchField) {
return "%" + searchField.toLowerCase() + "%";
}
Here you can see how I search the users by their address columns (city and street).
EDIT: Also you cannot use the #Query annotation that much dinamically (you van insert parameter values dinamically, but not parameters. That's where Specificaion is handy)
EDIT2: I know this is not the 2.x.x Spring version, but 1.5.x, but the idea is the same for joins.

Heroku spring-jpa UniqueConstraint

I met a problem: in my course project I use spring-jpa and create UserEntity with two unique fields. On my local machine all works perfectly well (creates unique constraints in db), but on heroku unique constraints doesn't creates.
I use java9 + spring-jpa.
import javax.persistence.*;
#Entity(name = "UserEntity")
#Table(name = "user_entity", uniqueConstraints = {
#UniqueConstraint(columnNames = {"nickname"}, name = "nickname_constraint"),
#UniqueConstraint(columnNames = {"email"}, name = "email_constraint")
})
public class UserEntity {
private Integer id;
private String nickname;
private String email;
private String passwordHash;
private String avatarPath;
private GameResults gameResults;
public UserEntity() {
}
public UserEntity(String nickname, String email, String password) {
this.nickname = nickname;
this.email = email;
this.passwordHash = password;
}
public UserEntity(String nickname, String password) {
this.nickname = nickname;
this.passwordHash = password;
}
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
public Integer getId() {
return id;
}
#Column(name = "nickname")
public String getNickname() {
return this.nickname;
}
#Column(name = "avatar_path")
public String getAvatarPath() {
return avatarPath;
}
#Column(name = "email")
public String getEmail() {
return email;
}
#Column(name = "password_hash")
public String getPasswordHash() {
return passwordHash;
}
#OneToOne(fetch = FetchType.LAZY)
public GameResults getGameResults() {
return gameResults;
}
// setters ommited
}
Thank you jusermar10!
Really, the problem was that i have deployed application with incorrect jpa entity first time. After redeploying fixed version of application there weren't necessary constraints in the postgres. So dropping db and restarting all dynos helped me.

Resources