Spring-Batch using MySql and Spring-Data Key-Value together - spring-boot

I have PoC to test Spring-Batch with MySql and Key-Value together.
POM
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-batch</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-keyvalue</artifactId>
</dependency>
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
<scope>runtime</scope>
</dependency>
Application
#SpringBootApplication
#EnableMapRepositories
public class SpringBatchPocApplication {
public static void main(String[] args) {
SpringApplication.run(SpringBatchPocApplication.class, args);
}
}
application.properties
spring.jpa.properties.hibernate.dialect=org.hibernate.dialect.MySQL5InnoDBDialect
spring.jpa.hibernate.ddl-auto=update
spring.datasource.url=jdbc:mysql://localhost:3307/pocdb
spring.datasource.username=root
spring.datasource.password=123456
spring.datasource.driver-class-name=com.mysql.cj.jdbc.Driver
spring.sql.init.platform=mysql
file.input=animal-list.csv
spring.batch.job.enabled=false
spring.batch.jdbc.initialize-schema=always
key-value entity
#Data
#Builder(setterPrefix = "with")
#NoArgsConstructor
#AllArgsConstructor
#KeySpace("imAnimal")
public class IMAnimal {
#Id
private UUID id;
private String name;
private String type;
private boolean processed;
}
key-value repository
#Repository
public interface IMAnimalRepository extends CrudRepository<IMAnimal, UUID> {
}
Than I have a service to controll key-value repository:
#Service
public class DatabaseService {
#Autowired
private IMAnimalRepository repository;
...
Yet, when I run my application, the IMAnimalRepository is not found by spring context.
Field repository in name.jikra.springbatchpoc.service.DatabaseService required a bean of type 'name.jikra.springbatchpoc.repository.IMAnimalRepository' that could not be found.
My guess is, there is a collision with JPA dataSource or something like this.
Is there a way to put it together?

Related

The entity field value is updated after exception throwing and transaction rollback

I have developed a test project to reproduce this issue.
This is a project structure:
pom.xml file:
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.7.1</version>
<relativePath/>
</parent>
<modelVersion>4.0.0</modelVersion>
<groupId>org.example</groupId>
<artifactId>value-updated-after-fail-spring</artifactId>
<version>1.0-SNAPSHOT</version>
<properties>
<maven.compiler.source>17</maven.compiler.source>
<maven.compiler.target>17</maven.compiler.target>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
</properties>
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-lang3</artifactId>
</dependency>
</dependencies>
</project>
Persone.java file:
#Entity
#NoArgsConstructor(access = AccessLevel.PROTECTED)
#RequiredArgsConstructor
#AllArgsConstructor
#Builder
#Getter
#Setter
#ToString
#FieldDefaults(level = AccessLevel.PRIVATE)
public class Person {
#Id
#GeneratedValue(strategy = GenerationType.AUTO)
Long id;
#Column(nullable = false)
#NonNull
String name;
}
PersonRepository.java file:
#Repository
public interface PersonRepository extends JpaRepository<Person, Long> {}
PersonService.java file:
#Component
public class PersonService {
private final PersonRepository repository;
public PersonService(PersonRepository repository) {
this.repository = repository;
}
#Transactional
public Person create(String name) {
return repository.save(new Person(name));
}
#Transactional
public Person save(Person person) {
if(StringUtils.isBlank(person.getName())) {
throw new RuntimeException();
}
Person personFromDB = getById(person.getId());
personFromDB.setName(person.getName());
return repository.save(personFromDB);
}
#Transactional
public Person getById(Long id) {
return repository.findById(id)
.orElseThrow(NullPointerException::new);
}
#Transactional
public void deleteAll() {
repository.deleteAll();
}
}
application.properties file:
spring.datasource.url=jdbc:h2:mem:test;DB_CLOSE_DELAY=-1
spring.datasource.username=sa
spring.datasource.password=
spring.jpa.hibernate.ddl-auto=create-drop
spring.jpa.show-sql=true
spring.h2.console.enabled=true
spring.datasource.driver-class-name=org.h2.Driver
spring.jpa.database-platform=org.hibernate.dialect.H2Dialect
TestApplicationConfiguration.java file
#SpringBootConfiguration
#EnableAutoConfiguration
#EnableJpaRepositories
#EntityScan("net.example.model")
#ComponentScan(basePackages = "net.example")
public class TestApplicationConfiguration {}
PersonServiceTest.java file:
#DataJpaTest
#AutoConfigureTestDatabase(replace = Replace.NONE)
class PersonServiceTest {
#Autowired
private PersonService service;
#AfterEach
void tearDownEach() {
service.deleteAll();
}
#Test
void rename() {
String expected = "name";
Person person = service.create(expected);
Person personFromDB = service.getById(person.getId());
personFromDB.setName("");
assertThrows(RuntimeException.class, () -> service.save(personFromDB));
assertEquals(expected, service.getById(personFromDB.getId()).getName());
}
}
The issue: Last assertion fails
org.opentest4j.AssertionFailedError:
Expected :name
Actual :
What I already tried to fix this?
I tried to remove the #Transactional annotation for the PersonService#getById method to avoid getting the entity from the cache. - This didn't fix the issue
I tried to add spring.cache.type=none to the application.properties file to disable the cache. - This didn't fix the issue
Why do I think it's the cache?
When I debugged this, I found that the PersonService#getById() method doesn't return actual data, but the method returns a cached object with a changed title.
The database isn't changed after calling the PersonService#save method because it throws an exception
Perhaps I'm not developing the tests correctly.
Maybe I should change the method of saving changed data.
Please share best practices and articles to better understand how to update data and how to properly configure and write tests for Spring Boot applications.
Thanks a lot for the comment from Andrey B. Panfilov.
I investigated the #Transactional and the first level cache of Hibernate.
Indeed, each test method call in a class annotated with the #DataJpaTest annotation creates, runs, and rollbacks a transaction.
Each transaction creates and closes the Hibernate session. As we know, the first level cache exists until the session closes. That's why it's also called session cache.
You can see the evidence in the following screenshots:
In the first screenshot, you can see that SpringExtension, which is defined in the #DataJpaTest annotation, opens a new session before each test is called.
In the second screenshot, you can see that SpringExtension closes the session after each test is called.
I decided to override the default transaction propagation:
#Transactional(propagation = Propagation.NEVER) it doesn't create a transaction when the method is called and throw an exception if the method is called in an existing transaction
Links that helped me:
Data Access
Transaction Propagation and Isolation in Spring #Transactional
Transaction Propagation with illustrations
Hibernate Caching - First Level Cach

spring boot fails to start-- define a bean of type 'TopicRepository' in configuration

I was following this JavaBrains tutorials of Spring Boot.
My project structure is as follows:
CourseApiApp.java:
#SpringBootApplication
#ComponentScan(basePackages = {
"com.bloodynacho.rishab.topics"
})
#EntityScan("com.bloodynacho.rishab.topics")
public class CourseApiApp {
public static void main(String[] args) {
SpringApplication.run(CourseApiApp.class, args);
}
}
TopicController.java:
#RestController
public class TopicController {
#Autowired
private TopicService topicService;
#RequestMapping(
value = "/topics"
)
public List<Topic> getAllTopcs() {
return topicService.getAllTopics();
}
}
TopicService.java:
#Service
public class TopicService {
#Autowired
private TopicRepository topicRepository;
public List<Topic> getAllTopics() {
List<Topic> topics = new ArrayList<>();
this.topicRepository
.findAll()
.forEach(topics::add);
return topics;
}
}
Topic.java:
#Entity
public class Topic {
#Id
private String id;
private String name;
private String description;
}
TopicRepository.java:
#Repository
public interface TopicRepository extends CrudRepository<Topic, String>{
}
pom.xml:
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.apache.derby</groupId>
<artifactId>derby</artifactId>
<scope>runtime</scope>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<optional>true</optional>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
I was using the lombok #Getter, #Getter and #AllArgsConstructor in Topic.java but I removed it after reading one of the answers here.
I read this1, this2, this3
Still, I get
***************************
APPLICATION FAILED TO START
***************************
Description:
Field topicRepository in com.bloodynacho.rishab.topics.TopicService required a bean of type 'com.bloodynacho.rishab.topics.TopicRepository' that could not be found.
The injection point has the following annotations:
- #org.springframework.beans.factory.annotation.Autowired(required=true)
Action:
Consider defining a bean of type 'com.bloodynacho.rishab.topics.TopicRepository' in your configuration.
Process finished with exit code 1
EDIT: I read this explaining how even without actually implementing the interface the #Autowired works. I understand the solution, but I don't understand how to solve my issue. Clearly, there is some problem with the way Spring Data is set up and configured (as mentioned in the answer)
Because if your other packages hierarchies are below your main application with the #SpringBootApplication annotation, you’re covered by implicit components scan.
Therefore, one simple solution can be done by following 2 steps:
Rename the package of main class to be com.bloodynacho.rishab.
(That is what I suggest that the complete package name of main app. is supposed to be root of other packages.)
Remove #ComponentScan and #EntityScan annotation.
(Although #ComponentScan is different from #EntityScan, it can be also removed in my experience.)

Spring Boot JPA Save not working

I am migrating Spring Boot 1.2.4 running in Websphere to Spring Boot 1.5.9. This application uses Spring Data JPA with Hibernate. After migration one of the save method is not working as expected. When I try to insert a new row in table getting an exception with SQLCODE=-204, SQLSTATE=42704, but same code works fine with Spring Boot 1.2.4.
My Pom.xml looks like below.
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>1.5.9.RELEASE</version>
</parent>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
Spring Boot 1.2.4 configuration looks like below.
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
<version>1.2.4</version>
</dependency>
<dependency>
<groupId>org.hibernate</groupId>
<artifactId>hibernate-entitymanager</artifactId>
<version>4.3.10.Final</version>
</dependency>
#Repository
public interface MyDataRepository extends CurdRepository<MyEntity,Long>{}
#Service
public class MyDataService{
#Autowired
private MyDataRepository myDataRepository;
public void insertMyDate(MyData myData){
myDataRepository.save(myData)
}
}
#Entity
#Table(name="MYDATA" ,schema="MYSCHEMA")
public class MyData implements java.io.Serializable {
private int memberId;
private String firstName;
private String lastName;
private Date dateofBirth;
#GeneratedValue(strategy=GenerationType.AUTO)
#Id
#Column(name="MEMBER_ID",uniqueue=true,nullable=false)
public int getMemberId(){
return this.memberId;
}
public void setMemberId(int memberId){
return this.memberId = memberId;
}
//Geter and Setters
}
With same code when executing save method in 1.5.9 getting below exception.
could not extract ResultSet; SQL [n/a]; nested exception is org.hibernate.exception.SQLGrammarException: could not extract ResultSet
You don't need this hibernate-entitymanager dependency becz you are using repository to save data

Springboot Junit test NoClassDefFoundError EntityManagerFactoryBuilderImpl

I created a springboot (2) webflux project as follow :
JPA Entity
#Entity
#Table(name = "users")
public class User implements Serializable
{
...
}
Spring repository
public interface UserRepository extends CrudRepository<User, Long>
{
}
Service
#Service
public class UserService
{
#Autowired
private UserRepository userRepo;
...
}
Webflux Handler
#Component
public class UserHandler
{
#Autowired
private UserService userService;
public Mono<ServerResponse> getUser(ServerRequest request)
{
...
}
}
RouteConfiguration
#Configuration
public class RouteConfiguration
{
#Bean
public static RouterFunction<ServerResponse> userRoutes(UserHandler userHandler)
{
return RouterFunctions.route(RequestPredicates.GET("/user"), userHandler:: getUser);
}
WebApp
#SpringBootApplication
public class WebApplication
{
public static void main(String[] args)
{
SpringApplication.run(WebApplication.class);
}
}
POM
<dependencies>
<!-- Compile -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-webflux</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>javax.xml.bind</groupId>
<artifactId>jaxb-api</artifactId>
<version>2.3.0</version>
</dependency>
<!-- Provided -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-tomcat</artifactId>
<scope>provided</scope>
</dependency>
<!-- Runtime -->
<dependency>
<groupId>org.postgresql</groupId>
<artifactId>postgresql</artifactId>
<scope>runtime</scope>
</dependency>
<!-- Test -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>io.projectreactor</groupId>
<artifactId>reactor-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>com.h2database</groupId>
<artifactId>h2</artifactId>
<scope>test</scope>
</dependency>
</dependencies>
Everything run fine, I can start my server and use it. I would like now to code some tests. Here is what I did :
#RunWith(SpringRunner.class)
#ContextConfiguration(classes = WebApplication.class)
public class UserHandlerTest
{
#Autowired
private ApplicationContext context;
#MockBean
private UserService userService;
private WebTestClient testClient;
#Before
public void setUp()
{
testClient = WebTestClient.bindToApplicationContext(context).build();
}
#Test
public void testUser()
{
...
}
}
What ever I tried, I got an error with hibernate dependencies during "mvn clean install" process :
[ERROR] testUser(...UserHandlerTest) Time elapsed: 0 s <<< ERROR!
java.lang.IllegalStateException: Failed to load ApplicationContext
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactory' defined in class path resource [org/springframework/boot/autoconfigure/orm/jpa/HibernateJpaConfiguration.class]: Invocation of init method failed; nested exception is java.lang.NoClassDefFoundError: Could not initialize class org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl
Caused by: java.lang.NoClassDefFoundError: Could not initialize class org.hibernate.jpa.boot.internal.EntityManagerFactoryBuilderImpl
I know JPA works in blocking way but I want to avoid to use NoSQL DB for this project. Did I miss something ? Thank you a lot for help !
Probably, you are missing details which we need to provide for datasource under src/main/resources. you can check https://github.com/hantsy/spring-reactive-sample/blob/master/boot-data-mongo/src/main/resources/application.yml. this might help you.
To test my Spring Webflux controllers, I finally use the WebFluxTest annotation. It works as expected :
#RunWith(SpringRunner.class)
#ContextConfiguration(classes = {RouteConfiguration.class, UserHandler.class})
#WebFluxTest
public class UserHandlerTest
{
#Autowired
private ApplicationContext context;
#MockBean(name="userService")
private UserService userService;
private WebTestClient testClient;
#Before
public void setUp()
{
testClient = WebTestClient.bindToApplicationContext(context).build();
}
...
As I do not use RestController annotation but functional endpoints I had to use ContextConfiguration and manually instantiate the WebTestClient.

How to enable basic caching with Spring Data JPA

I am trying to enable basic caching with Spring Data JPA. But I cannot understand why the DAO methods are still querying the database instead of using the cache.
Given the following Spring Boot 1.5.1 application
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.cache.annotation.EnableCaching;
#SpringBootApplication
#EnableCaching
public class Server{
public static void main(String[] args) {
SpringApplication.run(Server.class, args);
}
}
Controller
#Controller
public class PasswordsController {
#Autowired
private PasswordService service;
#SuppressWarnings("unchecked")
#RequestMapping("/passwords.htm")
public void passwords(Map model,
HttpServletRequest request) {
model.put("passwords", service.getPasswords(request));
}
...
Service
#Service
#Transactional
public class PasswordService extends BaseService {
#Autowired
private PasswordJpaDao passwordDao;
public Collection<Password> getPasswords(HttpServletRequest request) {
Collection<Password> passwords = passwordDao.getPasswords(params);
return passwords;
}
...
Interface
#Transactional
public interface PasswordJpaDaoCustom {
public Collection<Password> getPasswords(PasswordSearchParameters params);
}
and implementation
import javax.persistence.EntityManager;
import javax.persistence.PersistenceContext;
import javax.persistence.Query;
import javax.transaction.Transactional;
import org.springframework.cache.annotation.Cacheable;
import org.springframework.stereotype.Repository;
import com.crm.entity.Password;
import com.crm.search.PasswordSearchParameters;
#Transactional
#Repository
public class PasswordJpaDaoImpl implements PasswordJpaDaoCustom {
#PersistenceContext
private EntityManager em;
#Override
#Cacheable("passwords")
public Collection<Password> getPasswords(PasswordSearchParameters params) {
System.err.println("got here");
return em.createQuery(hql, Password.class);
}
...
Maven Dependencies
<!-- Spring Boot start -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-freemarker</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-web</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-security</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-jdbc</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-jpa</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-cache</artifactId>
</dependency>
<!-- Spring Boot end -->
<dependency>
<groupId>mysql</groupId>
<artifactId>mysql-connector-java</artifactId>
</dependency>
<dependency>
<groupId>org.javassist</groupId>
<artifactId>javassist</artifactId>
</dependency>
I understand that Spring Boot will implicitly use ConcurrentHashMap for caching without any specific configuration necessary?
But the getPasswords() dao method is always called instead of using the cache. Why is this?
Yes, spring boot by default uses ConcurrentHashMap for caching and the issue with your code is that you did not set any key for your passwords cache, so it is calling the database every time for fetching the data.
So you need to the key (any unique identifier) using the params object variables as shown below:
#Cacheable(value="passwords", key="#params.id")//any unique identifier
public Collection<Password> getPasswords(PasswordSearchParameters params) {
System.err.println("got here");
return em.createQuery(hql, Password.class);
}

Resources