I am working with legacy database in which db tables has no keys. For java Sake i have to used Id annotation. My goal is to read data from .dat file and insert it into table. I'm using spring batch for the above said purpose. To improve the performance threading is used. But i'm getting insertion/updation issue that i'm unable to figure out. I have referenced many sources but none seems to solve my purpose. Kindly help me out by giving some appropriate solution or reference. Thanks in advance...
Entity.java
#Entity
#Table(name = "int_repl_mkt_val")
public class IntReplMktVal implements Serializable {
private static final long serialVersionUID = 1L;
#EmbeddedId
private IntReplMktValId id;
#Column(name = "acct_sys_cd")
private String acctSysCd;
#Column(name = "co_num", nullable = false)
private Integer coNum;
#Column(name = "last_mod_tmstmp")
#Temporal(TemporalType.TIMESTAMP)
private Date lastModTmstmp;
#Column(name = "pim_owned", nullable = false)
private String pimOwned;
#Column(name = "position", nullable = false)
private BigDecimal position;
#Column(name = "pricing_plan")
private String pricingPlan;
#Column(name="source_system",nullable=false)
private String sourceSystem;
... getter and setter
}
EmbeddedClass.java
#Embeddable
public class IntReplMktValId implements Serializable
{
private static final long serialVersionUID = 4824041485763129937L;
#Column(name = "acct_id",nullable=false)
private Integer acctId;
#Column(name = "asset_id",nullable=false)
private Integer assetId;
... getter and setter
}
jpaRepository.class
#Repository
public interface IntReplMktValRepository extends JpaRepository<IntReplMktVal, IntReplMktValId>
{
}
BatchConfiguration.class
#Configuration
public class IMAPPositionBatchConfiguration
{
#Autowired
JobBuilderFactory jobBuilderFactory;
#Autowired
StepBuilderFactory stepBuilderFactory;
#StepScope
#Bean(name="imapPositionReader")
public FlatFileItemReader<IMAPPositionInputMapperDTO> reader(#Value("#{jobParameters['fileName']}") String fileName) throws IOException
{
FlatFileItemReader<IMAPPositionInputMapperDTO> newBean = new FlatFileItemReader<>();
newBean.setName("fileReader");
newBean.setResource(new InputStreamResource(FileUtils.openInputStream(new File(fileName))));
newBean.setLineMapper(this.lineMapper());
newBean.setLinesToSkip(1);
return newBean;
}
public DefaultLineMapper<IMAPPositionInputMapperDTO> lineMapper()
{
DefaultLineMapper<IMAPPositionInputMapperDTO> lineMapper = new DefaultLineMapper<>();
lineMapper.setLineTokenizer(this.lineTokenizer());
IMAPPositionReader imapPositionReader = new IMAPPositionReader();
lineMapper.setFieldSetMapper(imapPositionReader);
return lineMapper;
}
public DelimitedLineTokenizer lineTokenizer()
{
DelimitedLineTokenizer tokenizer = new DelimitedLineTokenizer();
tokenizer.setDelimiter("|");
tokenizer.setNames("field1","field2","field3");
tokenizer.setIncludedFields(5,4,7);
return tokenizer;
}
public ItemProcessor<IMAPPositionInputMapperDTO, IntReplMktVal> processor()
{
return new IMAPPositionProcessor();
}
#Bean(name="imapPositionBatchWriter")
public ItemWriter<IntReplMktVal> writer()
{
return new IMAPPositionWriter();
}
#Bean(name="imapPositionListener")
public JobExecutionListenerSupport jobCompletionListener()
{
return new IMAPPositionJobListener();
}
#Bean(name="imapPositionTaskExecutor")
public ThreadPoolTaskExecutor taskExecutor()
{
ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(50);
executor.setMaxPoolSize(100);
return executor;
}
#Bean(name="imapPositionStep")
public Step step(#Autowired #Qualifier("imapPositionTaskExecutor")TaskExecutor taskExecutor) throws IOException
{
return stepBuilderFactory.get("imapPositionStep")
.<IMAPPositionInputMapperDTO, IntReplMktVal>chunk(100)
.reader(this.reader(null))
.processor(this.processor())
.writer(this.writer())
.taskExecutor(taskExecutor)
.build();
}
#Bean(name="imapPositionFileImportJob")
public Job importUserJob(#Autowired #Qualifier("imapPositionStep") Step step)
{
return jobBuilderFactory
.get("imapPositionFileImportJob"+new Date())
.incrementer(new RunIdIncrementer())
.listener(this.jobCompletionListener())
.flow(step)
.end()
.build();
}
}
BatchWriter.java
public class IMAPPositionWriter implements ItemWriter<IntReplMktVal>
{
#Autowired
IntReplMktValRepository intReplMktValRepository;
#Override
public void write(List<? extends IntReplMktVal> items) throws Exception
{
intReplMktValRepository.saveAll(items);
}
}
ErrorLog
2019-06-07 17:22:01,522 ERROR [scopedTarget.imapPositionTaskExecutor-4] org.hibernate.internal.ExceptionMapperStandardImpl : HHH000346: Error during managed flush [org.hibernate.HibernateException: Duplicate identifier in table for: [com.capgroup.horizon.pricecapture.entities.IntReplMktVal#component[acctId,assetId]{assetId=274800, acctId=1}]]
NOTE: I have to insert all the data into table regardless of duplication or any other issue as keys are not defined so every data is valid.
Actually the problem was due to duplication found in persistent context which was resolved by setting the chunk size to 1.
Related
I have added an attribute to all my entities - transaction id - which is a sequence generated value that I bump up once in each transaction.
I also store the transaction id with user and start/end times so I have an audit trail for every change in the database.
What is the best way to handle storing a complete graph, where I basically only want to apply the transaction id to those entities that are actually dirty?
I can put a #PrePersist and #PreUpdate on the transaction id column, but how do I retrieve the value for the current transaction id? Is there a way to store and retrieve a value on the transaction object or other JPA controller? Do I need to use a ThreadLocal solution?
Ok, here is what I did. It seems to work in all of the use cases, though I have not done any performance testing, etc. If anyone sees anything that may be non-optimal or may fail in certain situations, please point it out.
Here is the base service class that all #Service implementations must extend:
public class BaseService
{
private final ActivityService activityService;
private final ApplicationEventPublisher applicationEventPublisher;
public static ThreadLocal<Activity> transaction = new ThreadLocal<>();
public BaseService(ActivityService activityService, ApplicationEventPublisher applicationEventPublisher)
{
this.activityService = activityService;
this.applicationEventPublisher = applicationEventPublisher;
}
Object executeWithinActivity(Updater updater)
{
boolean startedLocally = false;
try
{
if (transaction.get() == null)
{
startedLocally = true;
Activity activity = activityService.startTransaction();
transaction.set(activity);
}
return updater.execute(transaction.get());
}
finally
{
if (startedLocally)
{
applicationEventPublisher.publishEvent(new TransactionEvent());
Activity activity = transaction.get();
activityService.endTransaction(activity);
}
}
}
protected interface Updater
{
Object execute (Activity activity);
}
static class TransactionEvent
{
}
}
Activity is the entity that represents the stored transaction id:
#Entity
#Getter #Setter
#Table(name = "transactions", schema = "public", catalog = "euamdb")
public class Activity
{
#Id
#Column(name = "transaction_id", nullable = false)
#GeneratedValue(strategy = GenerationType.SEQUENCE, generator = "tx_generator")
#SequenceGenerator(name = "tx_generator", sequenceName = "transaction_seq", allocationSize = 1)
private long transactionId;
#Basic
#Column(name = "user_id", length = 24)
private String userId;
#Basic
#Column(name = "transaction_start")
#CreationTimestamp
private Date transactionStart;
#Basic
#Column(name = "transaction_end")
#UpdateTimestamp
private Date transactionEnd;
#Override
public boolean equals(Object o)
{
if (this == o) return true;
if (!(o instanceof Activity)) return false;
Activity that = (Activity) o;
return transactionId == that.transactionId;
}
#Override
public int hashCode()
{
return Long.hashCode(transactionId);
}
}
ActivityService (which does not extend BaseService):
#Service
public class ActivityService
{
private final ActivityRepository activityRepository;
private final AuthUserService authService;
#Autowired
public ActivityService(ActivityRepository activityRepository, AuthUserService authService)
{
this.activityRepository = activityRepository;
this.authService = authService;
}
#Transactional
public Activity startTransaction()
{
Activity activity = new Activity();
activity.setTransactionStart(new Date());
activity.setUserId(authService.getAuthenticatedUserId());
activityRepository.save(activity);
return activity;
}
#Transactional
public void endTransaction(Activity activity)
{
activity.setTransactionEnd(new Date());
activityRepository.save(activity);
}
}
The base entity class for all entities (excepting Activity):
#MappedSuperclass
#Getter #Setter
public class BaseEntity
{
#Basic
#Column(name = "transaction_id")
private Long transactionId;
#PrePersist
#PreUpdate
public void setupTransaction ()
{
ThreadLocal<Activity> transaction = BaseService.transaction;
Activity activity = transaction.get();
long transactionId = activity.getTransactionId();
setTransactionId(transactionId);
}
}
An example of a service:
#Service
public class OrganizationService extends BaseService
{
private final OrgUserRepository orgUserRepository;
private final UserService userService;
#Autowired
public OrganizationService(ActivityService activityService,
OrgUserRepository orgUserRepository,
UserService userService,
ApplicationEventPublisher applicationEventPublisher)
{
super(activityService, applicationEventPublisher);
this.orgUserRepository = orgUserRepository;
this.userService = userService;
}
#Transactional
public OrgUser save(User user, OrgUser orgUser)
{
return (OrgUser) executeWithinActivity(activity ->
{
orgUser.setUser(userService.save(user));
return orgUserRepository.save(orgUser);
});
}
}
UserService also will extend BaseService and the save(OrgUser) method will also executeWithinActivity.
Finally, the commit listener:
#Component
public class AfterCommitListener
{
#TransactionalEventListener(phase = TransactionPhase.AFTER_COMPLETION)
public void doAfterTxComplete(BaseService.TransactionEvent event)
{
BaseService.transaction.remove();
}
}
I'm working with spring JPA and HTTP post request, fetching the data row by row then post the data into HTTP request to API and its worked fine with me, but here im working with bulk number of data, so i have to use multi-threading but im new with java and spring how do I implement to work with 10 thread and each one of them reads 1k per each time in parallel that here ?
i have read something about multithreading for 10 threads each thread of them read 1k row per each time, I have around 10 million records in my database
AccessingDataJpaApplication class :
#SpringBootApplication
public class AccessingDataJpaApplication implements CommandLineRunner {
private static final Logger logger = LoggerFactory.getLogger(AccessingDataJpaApplication.class);
#Autowired
private Bulk_repositoryRepository bulk_repositoryRepository;
public static void main(String[] args) {
SpringApplication.run(AccessingDataJpaApplication.class);
}
Date currentDate = new Date();
#Override
public void run(String... args) throws Exception {
RestTemplate restTemplate = new RestTemplate();
HttpHeaders headers = new HttpHeaders();
headers.setAccept(Arrays.asList(MediaType.APPLICATION_JSON));
headers.setBasicAuth("user", "pass");
while(true) {
Date currentDate = new Date();
logger.info("Just Started");
for (Bulk_repository churnss : bulk_repositoryRepository.findAllByStatusAndCampTypeAndCampStartDateLessThanEqualAndCampEndDateGreaterThanEqual(0,2,currentDate,currentDate)) {
System.out.print(churnss);
logger.info(churnss.toString());
AddOfferRequest AddOffer = new AddOfferRequest("113", churnss.getMsisdn(),churnss.getParam1());
logger.info(AddOffer.toString());
HttpEntity<AddOfferRequest> entity = new HttpEntity<AddOfferRequest>(AddOffer,headers);
ResponseEntity<String> responseEntity = restTemplate.exchange(
"api link", HttpMethod.POST, entity, String.class);
if(responseEntity.getStatusCode() == HttpStatus.OK){
String response = responseEntity.getBody();
churnss.setStatus(1);
churnss.setProcessDate(new Date());
churnss.setFulfilment_status(response);
logger.info(churnss.toString() + ", Response: " + response);
bulk_repositoryRepository.save(churnss);
}else {
logger.warn("Record Id: " + churnss.getId() + ", Http Failed Response: " + responseEntity.getStatusCode());
}
}
Thread.sleep(1000);
}
}
}
Bulk_repository class:
#Entity
#Table(name = "BULK_REPOSITORY")
public class Bulk_repository {
#Id
#GeneratedValue(strategy=GenerationType.AUTO)
#Column(name = "id")
private long id;
#Column(name = "msisdn")
private String msisdn;
#Column(name = "camp_start_date")
private Date campStartDate;
#Column(name = "camp_end_date")
private Date campEndDate;
#Column(name = "camp_type")
private int campType;
#Column(name = "camp_cd")
private String camp_cd;
#Column(name = "status")
private int status;
#Column(name = "process_date")
private Date processDate;
#Column(name = "entry_date")
private Date entryDate;
#Column(name = "entry_user")
private String entry_user;
#Column(name = "param1")
private String param1;
#Column(name = "param2")
private String param2;
#Column(name = "param3")
private String param3;
#Column(name = "param4")
private String param4;
#Column(name = "param5")
private String param5;
#Column(name = "error_desc")
private String error_desc;
#Column(name = "fulfilment_status")
private int fulfilment_status;
##then getter and setters and tostring
Bulk_repositoryRepository class :
public interface Bulk_repositoryRepository extends CrudRepository<Bulk_repository, Long> {
Date today = new Date();
List<Bulk_repository>findAllByStatusAndCampTypeAndCampStartDateLessThanEqualAndCampEndDateGreaterThanEqual(int status, int campType,Date today0, Date today1);
Bulk_repository findById(long id);
}
AddOfferRequest class :
public class AddOfferRequest {
private String ChannelID="113";
private String MSISDN;
private String ServiceID;
public AddOfferRequest() {
}
public AddOfferRequest(String channelID,String mSISDN,String serviceID ) {
this.MSISDN = mSISDN;
this.ServiceID = serviceID;
}
## then getter and setter and tostring
i have created AsyncConfiguration class:
package com.example.accessingdatajpa;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.scheduling.annotation.EnableAsync;
import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;
import java.util.concurrent.Executor;
#Configuration
#EnableAsync
public class AsyncConfiguration {
private static final Logger LOGGER = LoggerFactory.getLogger(AsyncConfiguration.class);
#Bean (name = "taskExecutor")
public Executor taskExecutor() {
LOGGER.debug("Creating Async Task Executor");
final ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();
executor.setCorePoolSize(2);
executor.setMaxPoolSize(2);
executor.setQueueCapacity(1000);
executor.setThreadNamePrefix("CarThread-");
executor.initialize();
return executor;
}
}
but till now i can't undertand how can combaine the findby and http post with multithreading
Rewrite your code. Instead of a List<Bulk_repository> return a Stream<Bulk_repository>. This will lazily load the records from the database, instead of trying to do everything at once.
Then use the TaskExecutor to execute the different requests per thread, just give a task to it and it will be executed when there is a free thread.
#SpringBootApplication
public class AccessingDataJpaApplication implements CommandLineRunner {
private static final Logger logger = LoggerFactory.getLogger(AccessingDataJpaApplication.class);
#Autowired
private Bulk_repositoryRepository bulk_repositoryRepository;
#Autowired
private AsyncTaskExecutor executor;
#Autowired
private RestTemplate rest;
public static void main(String[] args) {
SpringApplication.run(AccessingDataJpaApplication.class);
}
#Override
public void run(String... args) throws Exception {
Date currentDate = new Date();
Stream< Bulk_repository> results = Bulk_repository churnss : bulk_repositoryRepository.findAllByStatusAndCampTypeAndCampStartDateLessThanEqualAndCampEndDateGreaterThanEqual(0,2,currentDate,currentDate);
results.forEach(it -> executor.submit(this.process(it)));
Thread.sleep(1000);
}
private void process(RestTemplate rest, Bulk_repository churnss) {
AddOfferRequest AddOffer = new AddOfferRequest("113", churnss.getMsisdn(),churnss.getParam1());
HttpEntity<AddOfferRequest> entity = new HttpEntity<AddOfferRequest>(AddOffer,headers);
try {
ResponseEntity<String> responseEntity = restTemplate.exchange(
"api link", HttpMethod.POST, entity, String.class);
if(responseEntity.getStatusCode() == HttpStatus.OK){
String response = responseEntity.getBody();
churnss.setStatus(1);
churnss.setProcessDate(new Date());
churnss.setFulfilment_status(response);
bulk_repositoryRepository.save(churnss);
}else {
logger.warn("Record Id: {}, Http Failed Response: {}",churnss.getId(), responseEntity.getStatusCode());
}
} catch (RestClientException rce) {
logger.warn("Record Id: {} Http Failed. ", churnss.getId(), rce);
}
}
}
NOTE: This was typed from the top of my head and isn't tested. However should provide some guidance.
Using #Async annotations to implement mutithread in spring. It can help you.
https://spring.io/guides/gs/async-method/
https://docs.spring.io/spring-data/rest/docs/2.0.0.M1/reference/html/paging-chapter.html
Try with Batch Insert/Update with Hibernate/JPA. Here is a nice tutorial
spring.jpa.properties.hibernate.jdbc.batch_size=500
I'm trying to save an object with Spring Data JPA. Unfortunately I always get a NullPointerException, and I don't understand why.
I followed this tutorial: https://www.callicoder.com/spring-boot-jpa-hibernate-postgresql-restful-crud-api-example/
I don't need a CRUD API so I left out the things that belong to it.
I get the following error:
java.lang.NullPointerException
at com.niclas.elitedangerousapi.handler.SystemPopulatedHandler.insertIntoDB(SystemPopulatedHandler.java:39)
at com.niclas.elitedangerousapi.Main.main(Main.java:19)
[main] ERROR c.n.e.h.SystemPopulatedHandler - null
UPDATE
I want to fill my database, but then when I want. At the end it should be so that every night a file is downloaded and then stored in the Database. Later I want to make the data accessible via an API.I want to execute this method (systemPopulatedHandler.insertIntoDB()) at the start and every x hours.
SystemPopulated.class
#Data
#Entity
#Table(name = "systems_populated")
#JsonIgnoreProperties(ignoreUnknown = true)
public class SystemPopulated {
#Id
#Column(name = "id")
private int id;
#Column(name = "edsm_id")
private long edsm_id;
#Column(name = "name")
private String name;
#Column(name = "x")
private double x;
#Column(name = "y")
private double y;
#Column(name = "z")
private double z;
#Column(name = "population")
private long population;
#Column(name = "is_populated")
private boolean is_populated;
#Column(name = "government_id")
private long government_id;
#Column(name = "government")
private String government;
#Column(name = "allegiance_id")
private int allegiance_id;
#Column(name = "allegiance")
private String allegiance;
#Column(name = "security_id")
private int security_id;
#Column(name = "security")
private String security;
#Column(name = "primary_economy_id")
private int primary_economy_id;
#Column(name = "primary_economy")
private String primary_economy;
#Column(name = "power")
private String power;
#Column(name = "power_state")
private String power_state;
#Column(name = "power_state_id")
private int power_state_id;
#Column(name = "needs_permit")
private boolean needs_permit;
#Column(name = "updated_at")
private long updated_at;
#Column(name = "controlling_minor_faction_id")
private int controlling_minor_faction_id;
#Column(name = "controlling_minor_faction")
private String controlling_minor_faction;
#Column(name = "reserve_type_id")
private int reserve_type_id;
#Column(name = "reserve_type")
private String reserve_type;
}
My SystemPopulatedRepository.class
#Repository
public interface SystemPopulatedRepository extends JpaRepository<SystemPopulated, Integer> {
}
My Class where i want to InsertIntoDB SystemPopulatedHandler.class
#Slf4j
public class SystemPopulatedHandler {
#Autowired
private SystemPopulatedRepository systemPopulatedRepository;
public void insertIntoDB() {
BufferedReader reader;
try {
reader = new BufferedReader( new FileReader(DOWNLOAD_SAVE_PATH + FILE_NAME_SYSTEMS_POPULATED) );
String line = reader.readLine();
while( line != null ){
ObjectMapper mapper = new ObjectMapper();
systemPopulatedRepository.save( mapper.readValue( line, SystemPopulated.class ) );
line = reader.readLine();
}
reader.close();
}
catch( Exception e ) {
e.printStackTrace();
log.error( e.getLocalizedMessage() );
}
}
}
My Main.class
#SpringBootApplication
#EnableJpaRepositories
public class Main {
public static void main( String[] args ) {
SpringApplication.run( Main.class, args );
SystemPopulatedHandler systemPopulatedHandler = new SystemPopulatedHandler();
systemPopulatedHandler.insertIntoDB();
}
}
The problem is that you create SystemPopulatedHandler yourself with
SystemPopulatedHandler systemPopulatedHandler = new SystemPopulatedHandler();
That way spring isn't injecting the repository into your class because that works only if spring creates the class.
But if you want to populate a database at startup (at least it seems that you try to do that) you should check out flyway (or 85.5 in this documentation: https://docs.spring.io/spring-boot/docs/current/reference/html/howto-database-initialization.html)
If you can't use flyway for some reason you can add the following code to the main class:
#Bean
public SystemPopulatedHandler systemPopulatedHandler(SystemPopulatedRepository repository) {
SystemPopulatedHandler systemPopulatedHandler = new SystemPopulatedHandler(repository);
systemPopulatedHandler.insertIntoDB()
return systemPopulatedHandler;
}
Afterwards add the constructor to the SystemPopulatedHandler class:
public SystemPopulatedHandler(SystemPopulatedRepository systemPopulatedRepository) {
this.systemPopulatedRepository = systemPopulatedRepository;
}
and remove the #Autowired annotation.
update
You also need to add the missing annotations as mentioned by this answer: https://stackoverflow.com/a/55767393/2248239
update 2
If you want to do that action periodically you can use scheduling (like in this guide https://spring.io/guides/gs/scheduling-tasks/)
Actually that's pretty easy:
Don't do the changes I mentioned above except adding the missing annotations and just do the following:
Add #Component to SystemPopulatedHandler
Add #Scheduled to insertIntoDB() in SystemPopulatedHandler
And add #EnableScheduling to the main class
For #Scheduled just read the guide it describes what you can do with the annotation.
This is due to the repository you have auto wired is not wiring the bean.
Please annonate your repository with #Repository
And in main class specify #EnableJpaRepository.
See the spring data JPA docs for more details
The problem is that you are instantiating the SystemPopulatedHandler bean without using ApplicationContext or BeanFactory, So it's not maintained by IOC container.
To use DI in a SpringBoot application all you need is to just auto wire SystemPopulatedHandler in your controller or service and then you can call insertIntoDB() method.
Since you are using spring boot and example is so simple you needn't make separate Configuration for beans.
#Controller
public class SystemPopulatedController {
#Autowired
private SystemPopulatedHandler systemPopulatedHandler;
#RequestMapping("/")
public void insertIntoDB() {
systemPopulatedHandler.insertIntoDB();
}
}
Many thanks to all of you. I have solved the problem as follows:
SystemPopulatedRepository.class
#Repository
public interface SystemPopulatedRepository extends JpaRepository<SystemPopulated, Integer> {
}
´´´
SystemPopulatedHandler.class
#Slf4j
#Component
public class SystemPopulatedHandler {
#Autowired
private SystemPopulatedRepository systemPopulatedRepository;
#PostConstruct
#Scheduled(cron = "0 0 0 * * *")
public void insertIntoDB() {
BufferedReader reader;
try {
reader = new BufferedReader( new FileReader(DOWNLOAD_SAVE_PATH + FILE_NAME_SYSTEMS_POPULATED) );
String line = reader.readLine();
while( line != null ){
ObjectMapper mapper = new ObjectMapper();
systemPopulatedRepository.save( mapper.readValue( line, SystemPopulated.class ) );
line = reader.readLine();
}
reader.close();
}
catch( Exception e ) {
e.printStackTrace();
log.error( e.getLocalizedMessage() );
}
}
}
´´´
Main.class
#SpringBootApplication
#EnableJpaRepositories
#EnableScheduling
public class Main {
public static void main( String[] args ) {
SpringApplication.run( Main.class, args );
FileHandler fileHandler = new FileHandler();
}
}
I have configured Spring Data JPA with Redis and using RedisRepositories with provides methods like find(), findAll() etc. All these methods seem to be working just fine, but I am not able to write my custom method like.
RedisEntity findByGenderAndGrade(String gender, String grade);
RedisEntity is a simple POJO Entity class. If you want any more info, please let me know in messages.
Following is my entity:
#Data
#RedisHash("test1")
public class RedisEntity implements Serializable {
#Id
#GeneratedValue
private String id;
private String name;
private String gender;
private Integer grade;
}
Repository:
#Repository
public interface TestRepository extends JpaRepository<RedisEntity, String> {
List<RedisEntity> findAllByGender(String gender);
List<RedisEntity> findAllByGrade(Integer grade);
}
Service/Controller:
#Override
public List<RedisEntity> getById(String id) {
return testRepository.findById(id); //returns data perfectly.
}
#Override
public List<RedisEntity> getAllByGender(String gender) {
return testRepository.findAllByGender(gender); //returns []
}
#Override
public void saveEntity(RedisEntity redisEntity) {
testRepository.save(redisEntity); // saves it in redis perfectly.
}
Also,
findByGender and findAllByGender both give [], although I can see data in my redis database and save it as well.
As requested by FrançoisDupire,
#Configuration
public class RedisConfig {
#Autowired
private DeploymentProperties deploymentProperties;
private static Logger logger = LoggerFactory.getLogger(RedisConfig.class);
#Bean
JedisConnectionFactory jedisConnectionFactory() {
RedisStandaloneConfiguration redisStandaloneConfiguration = new RedisStandaloneConfiguration("localhost", 6379);
redisStandaloneConfiguration.setPassword(RedisPassword.of("root"));
return new JedisConnectionFactory(redisStandaloneConfiguration);
}
#Bean
public RedisTemplate<String, Object> redisTemplate() {
RedisTemplate<String, Object> template = new RedisTemplate<>();
template.setConnectionFactory(jedisConnectionFactory());
return template;
}
}
Also, I had referred this article: Baeldung article on Spring data redis
As mentioned by #JoshJ and verified by myself and others,
The solution to the problem is:
Adding #Indexed annotation
to all those columns/fields which need to be used with all finds.
#Data
#RedisHash("EmployeeDetails")
public class RedisEntity {
#Id
private String employeeId;
private String firstName;
private String lastName;
#Indexed
private String gender;
#Indexed
private String grade;
}
We have the Spring Data Redis Library which provides the scope to write the custom method.Attaching Sample code.
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-redis</artifactId>
<version>2.0.8.RELEASE</version>
</dependency>
Entity Definition
#Data
#RedisHash("EmployeeDetails")
public class RedisEntity {
#Id
private String employeeId;
private String firstName;
private String lastName;
private String gender;
private String grade;
}
Repository Definition
#Repository
public interface RedisEntityRepository extends CrudRepository<RedisEntity, String>{
List<RedisEntity> findAllByGenderAndGrade(String gender, String grade);
}
Implementation
#Component
public class RedisEntityImpl implements RedisEntityService {
#Autowired
private RedisEntityRepository redisEntityRepository;
#Override
public List<RedisEntity> getAllByGenderAndGrade(String gender, String grade) {
return redisEntityRepository.findAllByGenderAndGrade(gender,grade);
}
}
Properties
spring.cache.type = redis
spring.redis.host = localhost
spring.redis.port = 6379
I create a spring boot application with MySQL,JPA,Web dependencies,and manually config my database settings in .properties file of Spring boot. I passed compiling, and started application successfully, and adding one record is normal fine.
BUT, i use method 'findAll(Pageable pageable)' i got a problem, that was
Could not write JSON: failed to lazily initialize a collection of role，could not initialize proxy - no Session
I got confused, i started to debug my code, finally i found that the child collection of the result is null, and it contained an error, which is
"Exception occurred: com.sun.jdi.InvocationException occurred invoking method.."
I tried a lot to fix my code, but no use.
who can help me?
The entity relationship is a simple one to many:
TeacherInfo entity and ClassInfo entity, teacher manage multiple classes, just simple as this.
here is the enter point of my app:
#SpringBootApplication(exclude= {
DataSourceAutoConfiguration.class,
HibernateJpaAutoConfiguration.class,
DataSourceAutoConfiguration.class
})
#EnableTransactionManagement
public class OrmTestApplication {
public static void main(String[] args) {
SpringApplication.run(OrmTestApplication.class, args);
}
}
Database properties setting is here:
spring.datasource.primary.url=jdbc:mysql://localhost:3306/ormtest?useSSL=false
spring.datasource.primary.username=root
spring.datasource.primary.password=BlaNok2700
spring.datasource.primary.driver-class-name = com.mysql.jdbc.Driver
hibernate.hbm2ddl.auto = update
hibernate.show-sql = true
My Data base configure java code is here:
Configuration
#EnableJpaRepositories(basePackages = "com.lanjian.ormtest.repositories", entityManagerFactoryRef = "primaryEntityManagerFactory", transactionManagerRef = "primaryTransactionManager")
public class PrimaryDbConfig {
#Autowired
private Environment env;
#Bean
#ConfigurationProperties(prefix="spring.datasource.primary")
public DataSourceProperties primaryDataSourceProperties() {
return new DataSourceProperties();
}
#Bean
public DataSource primaryDataSource() {
DataSourceProperties dbProperty = primaryDataSourceProperties();
return DataSourceBuilder.create()
.driverClassName(dbProperty.getDriverClassName())
.url(dbProperty.getUrl())
.username(dbProperty.getUsername())
.password(dbProperty.getPassword())
.build();
}
#Bean
public LocalContainerEntityManagerFactoryBean primaryEntityManagerFactory() {
LocalContainerEntityManagerFactoryBean factory = new LocalContainerEntityManagerFactoryBean();
factory.setDataSource(primaryDataSource());
factory.setPackagesToScan("com.lanjian.ormtest.entities");
factory.setJpaVendorAdapter(new HibernateJpaVendorAdapter());
Properties jpaProperties = new Properties();
jpaProperties.put("hibernate.hbm2ddl.auto", env.getProperty("hibernate.hbm2ddl.auto"));
jpaProperties.put("hibernate.show-sql", env.getProperty("hibernate.show-sql"));
factory.setJpaProperties(jpaProperties);
return factory;
}
#Bean
public PlatformTransactionManager primaryTransactionManager() {
EntityManagerFactory factory = primaryEntityManagerFactory().getObject();
return new JpaTransactionManager(factory);
}
}
My REST controller method is here:
#Autowired
private TeacherRepository teacherRepository;
#GetMapping("/page")
public Page<TeacherInfo> page(Pageable pageable){
Page<TeacherInfo> list = teacherRepository.findAll(pageable);
return list;
}
What happened
After i started my application, and use postman send request, i got this:
got a 500 error
And i debugger my code, found this:
child collection is null
In the picture, 'classes' is a list collection, but it is null, i don't understand.
Here are the TeacherInfo entity I defined
#Entity
#Table(name = "teacher")
public class TeacherInfo {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
private String name;
private byte age;
private boolean male;
#OneToMany(cascade = CascadeType.ALL,fetch = FetchType.LAZY, mappedBy="chargedTeacher")
private List<ClassInfo> classes = new ArrayList<>();
public void initialize() {
for (ClassInfo classInfo : classes) {
classInfo.setChargedTeacher(this);
for (StudentInfo studentInfo : classInfo.getStudents()) {
studentInfo.setClassInfo(classInfo);
}
}
}
//Setters and Getters}
Here is the ClassInfo Entity i defined
#Entity
#Table(name = "class_info")
public class ClassInfo {
#Id
#GeneratedValue(strategy = GenerationType.IDENTITY)
private Long id;
private String name;
private int capacity;
#ManyToOne(fetch = FetchType.LAZY)
#JoinColumn(name = "teacher_id",nullable=false)
#JsonIgnore
private TeacherInfo chargedTeacher;
#OneToMany(cascade = CascadeType.ALL,fetch=FetchType.LAZY,mappedBy="classInfo")
private List<StudentInfo> students = new ArrayList<>();
public Long getId() {
return id;
}
public void setId(Long id) {
this.id = id;
}
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
public int getCapacity() {
return capacity;
}
public void setCapacity(int capacity) {
this.capacity = capacity;
}
public TeacherInfo getChargedTeacher() {
return chargedTeacher;
}
public void setChargedTeacher(TeacherInfo chargedTeacher) {
this.chargedTeacher = chargedTeacher;
}
public List<StudentInfo> getStudents() {
return students;
}
public void setStudents(List<StudentInfo> students) {
this.students = students;
}
}
I think that the problem may come from Transactionality and JPA Fetching types.
Your repository method is being invoked not using a transaction, which implies that the transaction is on the boundaries of the method invocation (which might not be wrong). Spring returns a Page with objects but when it tries to serialize them, transaction is gone so no way to access childs.
I would suggest to put the JPA relationship as EAGER fetching, allowing all the objects to be present on the repository result when the transaction ends.
EDIT:
Answer to comments
#Bean
public PlatformTransactionManager primaryTransactionManager(EntityManagerFactory factory) {
return new JpaTransactionManager(factory);
}