Why setting params for JpaRepo too slow for simple table with only 2 cols? - spring-boot

I am using simple Entity with 2 cols only. In my table PK col is varchar in db but in actual values stored is numerical in that col and other col is int. DB is MS SQL Server. And this table has 165 million records.
Table structure:
SECURITY_TAB
varchar(30) SECID PK;
int VERSION;
Entity
#Entity
#Table(name = "SECURITY_TAB")
public class Security {
#Id
#Column
private String secid;
#Column
private int version;
//... getter n setter n toString
}
Repository
public interface SecurityRepository extends JpaRepository<Security, String> {
#Query("SELECT s from Security s where secid= :secid")
Security findSecurityBySecid(#Param("secid") String secid)); //this is slow
//below is fine.
//#Query("SELECT s from Security s where secid='1111'")
//Security findSecurityBySecid();
}
TestClass
#RunWith(SpringRunner.class)
#SpringBootTest
public class SecurityTests {
#Autowired
private SecurityRepository securityRepository;
#Test
public void testSecurityBySecid() {
Instant start = Instant.now();
Security security = securityRepository.findSecurityBySecid("1111");
System.out.println(Duration.between(start, Instant.now()));
System.out.println(security);
System.out.println(Duration.between(start, Instant.now()));
}
}
This simple query is taking more than 20 secs,
While when I run similar query MS SQL Server Mgmt Studio or hard code the value in the query result is returned in mill seconds. So what is going wrong?

was able to solve:
The solution was to setup this property in the jdbc url : sendStringParametersAsUnicode=false
Full example if you are using MS SQL official driver : jdbc:sqlserver://yourserver;instanceName=yourInstance;databaseName=yourDBName;sendStringParametersAsUnicode=false;
Solution: sql server query running slow from java

Related

Spring Data JDBC - immutable owner of OneToMany relation

I'm trying out Spring Data JDBC, and I think that immutable entities are the key feature here.
I want to model a one-to-many relationship between Team and Player and have these entities immutable. What I have working right now is:
#Getter
#AllArgsConstructor
#NoArgsConstructor
public class Team {
#Id
#Wither
private Integer id;
private String name;
private List<Player> players;
}
#Value
public class Player {
#Id
#Wither
Integer id;
String name;
public static Player of(String name) {
return new Player(null, name);
}
}
I managed to get a OneToMany relationship working properly for mutable classes. I managed to make it work for immutable Player. However, I can't get Team mapping to work without #NoArgsConstructor, or with #Value instead of what I have right now. I get the weird message as if Spring Data JDBC started seeing different mapping when this constructor is missing:
Caused by: org.h2.jdbc.JdbcSQLSyntaxErrorException: Column "players_id" not found [42122-199]
If anybody tries to recreate that scenario, here's my h2 schema:
create table team (id serial primary key, name varchar);
create table player (id serial primary key, name varchar, team int references team(id), team_key int);
And my Spock test:
#SpringBootTest
class TeamRepositoryTest extends Specification {
#Autowired
TeamRepository teamRepository
def "Context loads"() {
given:
def rick = Player.of("Rick")
def morty = Player.of("Morty")
def team = new Team(null, "Rick and Morty", Arrays.asList(rick, morty))
when:
def savedTeam = teamRepository.save(team)
then:
def savedTeamOpt = teamRepository.findById(savedTeam.id)
Team retrievedTeam = savedTeamOpt.get()
retrievedTeam.id == savedTeam.id
retrievedTeam.name == "Rick and Morty"
retrievedTeam.players.size() == 2
}
}
You hit a bug in Spring Data JDBC: https://jira.spring.io/browse/DATAJDBC-127
Currently entities or collections of entities don't work as constructor arguments.
Create a constructor without these arguments and add setters or if you want to stay immutable withers for them.

Why is JPA query so slow?

I am implementing queries in my web application with JPA repositories. The two main tables I am querying from are FmReportTb and SpecimenTb.
Here are the two entity classes (only important attributes are listed).
//FmReportTb.java
#Entity
#Table(name="FM_REPORT_TB")
public class FmReportTb implements Serializable {
#Column(name="ROW_ID")
private long rowId;
#Column(name="FR_BLOCK_ID")
private String frBlockId;
#Column(name="FR_FULL_NAME")
private String frFullName;
#OneToOne
#JoinColumn(name="SPECIMEN_ID")
private SpecimenTb specimenTb;
FmReportTb has OneToOne relationship with SpecimenTb.
#Entity
#Table(name="SPECIMEN_TB")
public class SpecimenTb implements Serializable {
private String mrn;
#OneToOne(mappedBy="specimenTb", cascade=CascadeType.ALL)
private FmReportTb fmReportTb;
The query I am working on is to find all records in FmReportTb and show a few attributes from FmReportTb plus mrn from SpecimenTb.
Here is my JPA repository for FmReportTb:
#Repository
public interface FmReportRepository extends JpaRepository<FmReportTb, Long> {
#Query("select f from FmReportTb f where f.deleteTs is not null")
public List<FmReportTb> findAllFmReports();
Since, I am only showing part of the attributes from FmReportTb and one attribute from SpecimenTb, I decided to create a Value Object for FmReportTb. The constructor of the VO class assigns attributes from FmReportTb and grabs mrn attribute from SpecimenTb based on the OneToOne relationship. Another reason for using VO is because table FmReportTb has a lot of OneToMany children entities. For this particular query, I don't need any of them.
public class FmReportVO {
private String frBlockId;
private Date frCollectionDate;
private String frCopiedPhysician;
private String frDiagnosis;
private String frFacilityName;
private String frFullName;
private String frReportId;
private String filepath;
private String mrn;
public FmReportVO(FmReportTb fmReport) {
this.frBlockId = fmReport.getFrBlockId();
this.frCollectionDate = fmReport.getFrCollectionDate();
this.frCopiedPhysician = fmReport.getFrCopiedPhysician();
this.frDiagnosis = fmReport.getFrDiagnosis();
this.frFacilityName = fmReport.getFrFacilityName();
this.frFullName = fmReport.getFrFullName();
this.frReportId = fmReport.getFrReportId();
this.mrn = fmReport.getSpecimenTb().getMrn();
}
I implemented findall method in servicebean class to return a list of FmReportTb VOs.
//FmReportServiceBean.java
#Override
public List<FmReportVO> findAllFmReports() {
List<FmReportTb> reports = fmReportRepository.findAllFmReports();
if (reports == null) {
return null;
}
List<FmReportVO> fmReports = new ArrayList<FmReportVO>();
for (FmReportTb report : reports) {
FmReportVO reportVo = new FmReportVO(report);
String filepath = fileLoadRepository.findUriByFileLoadId(report.getFileLoadId().longValue());
reportVo.setFilepath(filepath);
fmReports.add(reportVo);
}
return fmReports;
}
Lastly, my controller looks like this:
#RequestMapping(
value = "/ristore/foundation/",
method = RequestMethod.GET,
produces = "application/json")
public ResponseEntity<List<FmReportVO>> getAllFmReports() {
List<FmReportVO> reports = ristoreService.findAllFmReports();
if (reports == null) {
return new ResponseEntity<List<FmReportVO>>(HttpStatus.NOT_FOUND);
}
return new ResponseEntity<List<FmReportVO>>(reports, HttpStatus.OK);
}
There are about 200 records in the database. Surprisingly, it took almost 2 full seconds to retrieve all the records in JSON. Even though I did not index all the tables, this is way too slow. Similar query takes about probably a few ms on the database directly. Is it because I am using Value Objects or JPA query tends to be this slow?
EDIT 1
This may have to do with the fact that FmReportTb has almost 20 OneToMany entities. Although the fetchmode of these child entities are set to LAZY, JPA Data repository tends to ignore the fetchmode. So I ended up using NamedEntityGraph to specify the attributes EAGER. This next section is added to the head of my FmReportTb entity class.
#Entity
#NamedEntityGraph(
name = "FmReportGraph",
attributeNodes = {
#NamedAttributeNode("fileLoadId"),
#NamedAttributeNode("frBlockId"),
#NamedAttributeNode("frCollectionDate"),
#NamedAttributeNode("frDiagnosis"),
#NamedAttributeNode("frFullName"),
#NamedAttributeNode("frReportId"),
#NamedAttributeNode("specimenTb")})
#Table(name="FM_REPORT_TB")
And then #EntityGraph("FmReportGraph") was added before the JPA repository query to find all records. After doing that, the performance is improved a little bit. Now fetching 1500 records only takes about 10 seconds. However, it still seems too slow given each json object is fairly small.
Answering for the benefit of others with slow JPA queries...
As #Ken Bekov hints in the comments, foreign keys can help a lot with JPA.
I had a couple of tables with a many to one relationship - a query of 100,000 records was taking hours to perform. Without any code changes I reduced this to seconds just by adding a foreign key.
In phpMyAdmin you do this by creating a Relationship from the "many" table to the "one" table. For a detailed explanation see this question: Setting up foreign keys in phpMyAdmin?
and the answer by #Devsi Odedra

I need help for persisting into oracle database

There is a problem about generating id while persisting into database.
I added the following code to my jpa entity file, however I'm getting 0 for personid.
#Id
#Column(unique=true, nullable=false, precision=10, name="PERSONID")
#SequenceGenerator(name="appUsersSeq", sequenceName="SEQ_PERSON", allocationSize=1)
#GeneratedValue(strategy=GenerationType.SEQUENCE, generator = "appUsersSeq")
private long personid;
EjbService:
#Stateless
public class EjbService implements EjbServiceRemote {
#PersistenceContext(name = "Project1245")
private EntityManager em;
#Override
public void addTperson(Tperson tp) {
em.persist(tp);
}
}
0 is default value for long type. The id will be set after invoking select query for the related sequence, which commonly is executed when you persist the entity. Are you persisting the entity? In case yes, post the database sequence definition to check it.

Spring Data MongoDB #Indexed annotation - create index error (system.indexes) during find query operation

I am facing a weird issue as this has been working earlier with no issues.
I am using latest Spring Data MongoDB 1.5.2 release with Mongo Java Driver 2.12.3
I have used mongoDB ensureIndex command to create index on a collection field manaully through Mongo Shell (MongoDB serevr running with MongoDB 2.4 version).
I have checked both on my collection with collection.getIndexes() command and system.indexes collection too that the above index is created correctly.
With Spring Data MongoDB, I have also placed #Indexed annotation on same field in domain object.
During a query find operation, I was getting below Spring Data MongoDB create index exception for creating index in system.indexes collection and complained the document obj size is greater then 16MB.
If, I remove #Indexed annotation on domain object field and re-run the find query, then there are no errors from application side.
I would like to understand as why:
1) Spring Data MongoDB is trying to create index in system.indexes during a find query operation with #Indexed annotation available.
2) Why Spring Data MongoDB complains obj size is greater then 16 MB during find query operation with #Indexed annotation, whereas I can run the same find query on MongoDB shell with no issues.
3) Is the above document in collection is corrupt? Do I have to re-import fresh data again and test the same as this has been working in the past with no issues from application side?
4) What is the life cycle of Spring Data MongoDB #Indexed annotation or How does it works? Do we have any documentation available in details?
Domain Object
#Document(collection = "Users")
public class Users implements Serializable {
#Id
ObjectId id;
#Indexed
String appUserId
String firstName;
String lastName;
}
#Repository
public interface UsersRepository extends MongoRepository<Users, String>, UsersRepositoryCustom {
// default query methods
}
#Repository
public interface UsersRepositoryCustom {
int findUserCount(String appUserId);
}
#Component
public class UsersRepositoryImpl implements UsersRepositoryCustom {
#Override
public int findUserCount(String appUserId) {
DBCursor dbCursor = null;
int count = 0;
Query query = new Query();
query.addCriteria(
Criteria.where("appUserId").is(appUserId));
try {
DBCollection dbCollection = mongoOperations.getCollection("Users");
System.out.println("Start time : " + new Date().toString());
dbCursor = dbCollection.find(query.getQueryObject());
//while (dbCursor.hasNext()) {
//do dome processing
}
count = dbCursor.count();
System.out.println("End time : " + new Date().toString());
} finally {
dbCursor.close();
}
return count;
}
}
Caused by: com.mongodb.WriteConcernException: { "serverUsed" : "XXXXXXXX:11111" , "err" : "BSONObj size: 0 (0x00000000) is invalid. Size must be between 0 and 16793600(16MB) First element: EOO" , "code" : 10334 , "n" : 0 , "connectionId" : 341 , "ok" : 1.0}
at com.mongodb.CommandResult.getWriteException(CommandResult.java:90)
at com.mongodb.CommandResult.getException(CommandResult.java:79)
at com.mongodb.CommandResult.throwOnError(CommandResult.java:131)
at com.mongodb.DBTCPConnector._checkWriteError(DBTCPConnector.java:135)
at com.mongodb.DBTCPConnector.access$000(DBTCPConnector.java:39)
at com.mongodb.DBTCPConnector$1.execute(DBTCPConnector.java:186)
at com.mongodb.DBTCPConnector$1.execute(DBTCPConnector.java:181)
at com.mongodb.DBTCPConnector.doOperation(DBTCPConnector.java:210)
at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:181)
at com.mongodb.DBCollectionImpl.insertWithWriteProtocol(DBCollectionImpl.java:530)
at com.mongodb.DBCollectionImpl.createIndex(DBCollectionImpl.java:369)
at com.mongodb.DBCollection.createIndex(DBCollection.java:564)
at org.springframework.data.mongodb.core.index.MongoPersistentEntityIndexCreator.createIndex(MongoPersistentEntityIndexCreator.java:135)
at org.springframework.data.mongodb.core.index.MongoPersistentEntityIndexCreator.checkForAndCreateIndexes(MongoPersistentEntityIndexCreator.java:129)
at org.springframework.data.mongodb.core.index.MongoPersistentEntityIndexCreator.checkForIndexes(MongoPersistentEntityIndexCreator.java:121)
at org.springframework.data.mongodb.core.index.MongoPersistentEntityIndexCreator.onApplicationEvent(MongoPersistentEntityIndexCreator.java:105)
at org.springframework.data.mongodb.core.index.MongoPersistentEntityIndexCreator.onApplicationEvent(MongoPersistentEntityIndexCreator.java:46)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:98)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:333)
at org.springframework.data.mapping.context.AbstractMappingContext.addPersistentEntity(AbstractMappingContext.java:307)
at org.springframework.data.mapping.context.AbstractMappingContext.getPersistentEntity(AbstractMappingContext.java:181)
at org.springframework.data.mapping.context.AbstractMappingContext.getPersistentEntity(AbstractMappingContext.java:141)
at org.springframework.data.mapping.context.AbstractMappingContext.getPersistentEntity(AbstractMappingContext.java:67)
at org.springframework.data.mongodb.repository.support.MongoRepositoryFactory.getEntityInformation(MongoRepositoryFactory.java:141)
at org.springframework.data.mongodb.repository.support.MongoRepositoryFactory.getTargetRepository(MongoRepositoryFactory.java:83)
at org.springframework.data.repository.core.support.RepositoryFactorySupport.getRepository(RepositoryFactorySupport.java:158)
at org.springframework.data.repository.core.support.RepositoryFactoryBeanSupport.initAndReturn(RepositoryFactoryBeanSupport.java:224)
at org.springframework.data.repository.core.support.`enter code here`RepositoryFactoryBeanSupport.afterPropertiesSet(RepositoryFactoryBeanSupport.java:210)
at org.springframework.data.mongodb.repository.support.MongoRepositoryFactoryBean.afterPropertiesSet(MongoRepositoryFactoryBean.java:108)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1612)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1549)

Selecting from a view in H2 doesn't work

I recently replaced PostgreSql unit test database with an in memory H2 Database. I couldn't figure out why couple tests are failing, it was working fine with Postgre. There are approx. 280+ unit tests in this application. The failing dao tests are selecting from a view and the entity has a EmbeddedId, those the columns of that view.
See below code (Note: I changed names, code and the sql to hide the real names when writing this email, but this was a working unit test with Postgre db)
<pre>
#Table(name = "item_view") // <- item_view is a database view
public class ItemV implements Serializable
{
.....
#EmbeddedId // <- entity has an embedded id
private ItemVId id;
.....
#Embeddable
public static class ItemVId implements Serializable //<- This is the embeddedId
{
#Column(name = "item_id", updatable=false, insertable=false)
private Long ItemId; //<- col no.1 of view
#Column(name = "item_type_id", updatable=false, insertable=false)
private Integer ItemTypeId; //<- col no.2 of view
.....
ItemType is an enum
And the view is
CREATE OR REPLACE VIEW item_view AS
( ( ( SELECT pt.id as item_id, cit.id as item_type_id
FROM xyz pt, item_type cit
WHERE pt.name::text = 'xyz'::text
UNION
SELECT z.id as item_id, cit.id as item_type_id
FROM zzz z, item_type cit
WHERE z.name::text = 'zzz'::text)
..............
and the dao method is
public ItemView find(Long itemId, ItemType itemType)
{
String hql = " from ItemV iv where iv.id.itemId = :itemId and iv.id.itemTypeId = :itemTypeId ");
List<ItemView> result = (List<ItemView>)getEntityManager()
.createQuery(hql)
.setParameter("itemId", itemId)
.setParameter("itemTypeId", itemType.getId())
.setMaxResults(1)
.getResultList();
return result.isEmpty()
? null : result.get(0);
}
This dao method always returns empty results, never finding existing rows in the view??? I know those rows exist because when I do getAll() on the same dao I see results and I see a matching row for the criteria.
Is there anything special about selecting rows from a view in H2 database?
Thanks
Ok fixed, I had to use a smaller number for LOCK_TIMEOUT value, so now I can connect to database and see values. Selecting from view problem also fixed.
I have to say H2 is really neat and elegant. I'm glad, I switched the unit test db to H2.

Resources