I am facing a weird issue as this has been working earlier with no issues.
I am using latest Spring Data MongoDB 1.5.2 release with Mongo Java Driver 2.12.3
I have used mongoDB ensureIndex command to create index on a collection field manaully through Mongo Shell (MongoDB serevr running with MongoDB 2.4 version).
I have checked both on my collection with collection.getIndexes() command and system.indexes collection too that the above index is created correctly.
With Spring Data MongoDB, I have also placed #Indexed annotation on same field in domain object.
During a query find operation, I was getting below Spring Data MongoDB create index exception for creating index in system.indexes collection and complained the document obj size is greater then 16MB.
If, I remove #Indexed annotation on domain object field and re-run the find query, then there are no errors from application side.
I would like to understand as why:
1) Spring Data MongoDB is trying to create index in system.indexes during a find query operation with #Indexed annotation available.
2) Why Spring Data MongoDB complains obj size is greater then 16 MB during find query operation with #Indexed annotation, whereas I can run the same find query on MongoDB shell with no issues.
3) Is the above document in collection is corrupt? Do I have to re-import fresh data again and test the same as this has been working in the past with no issues from application side?
4) What is the life cycle of Spring Data MongoDB #Indexed annotation or How does it works? Do we have any documentation available in details?
Domain Object
#Document(collection = "Users")
public class Users implements Serializable {
#Id
ObjectId id;
#Indexed
String appUserId
String firstName;
String lastName;
}
#Repository
public interface UsersRepository extends MongoRepository<Users, String>, UsersRepositoryCustom {
// default query methods
}
#Repository
public interface UsersRepositoryCustom {
int findUserCount(String appUserId);
}
#Component
public class UsersRepositoryImpl implements UsersRepositoryCustom {
#Override
public int findUserCount(String appUserId) {
DBCursor dbCursor = null;
int count = 0;
Query query = new Query();
query.addCriteria(
Criteria.where("appUserId").is(appUserId));
try {
DBCollection dbCollection = mongoOperations.getCollection("Users");
System.out.println("Start time : " + new Date().toString());
dbCursor = dbCollection.find(query.getQueryObject());
//while (dbCursor.hasNext()) {
//do dome processing
}
count = dbCursor.count();
System.out.println("End time : " + new Date().toString());
} finally {
dbCursor.close();
}
return count;
}
}
Caused by: com.mongodb.WriteConcernException: { "serverUsed" : "XXXXXXXX:11111" , "err" : "BSONObj size: 0 (0x00000000) is invalid. Size must be between 0 and 16793600(16MB) First element: EOO" , "code" : 10334 , "n" : 0 , "connectionId" : 341 , "ok" : 1.0}
at com.mongodb.CommandResult.getWriteException(CommandResult.java:90)
at com.mongodb.CommandResult.getException(CommandResult.java:79)
at com.mongodb.CommandResult.throwOnError(CommandResult.java:131)
at com.mongodb.DBTCPConnector._checkWriteError(DBTCPConnector.java:135)
at com.mongodb.DBTCPConnector.access$000(DBTCPConnector.java:39)
at com.mongodb.DBTCPConnector$1.execute(DBTCPConnector.java:186)
at com.mongodb.DBTCPConnector$1.execute(DBTCPConnector.java:181)
at com.mongodb.DBTCPConnector.doOperation(DBTCPConnector.java:210)
at com.mongodb.DBTCPConnector.say(DBTCPConnector.java:181)
at com.mongodb.DBCollectionImpl.insertWithWriteProtocol(DBCollectionImpl.java:530)
at com.mongodb.DBCollectionImpl.createIndex(DBCollectionImpl.java:369)
at com.mongodb.DBCollection.createIndex(DBCollection.java:564)
at org.springframework.data.mongodb.core.index.MongoPersistentEntityIndexCreator.createIndex(MongoPersistentEntityIndexCreator.java:135)
at org.springframework.data.mongodb.core.index.MongoPersistentEntityIndexCreator.checkForAndCreateIndexes(MongoPersistentEntityIndexCreator.java:129)
at org.springframework.data.mongodb.core.index.MongoPersistentEntityIndexCreator.checkForIndexes(MongoPersistentEntityIndexCreator.java:121)
at org.springframework.data.mongodb.core.index.MongoPersistentEntityIndexCreator.onApplicationEvent(MongoPersistentEntityIndexCreator.java:105)
at org.springframework.data.mongodb.core.index.MongoPersistentEntityIndexCreator.onApplicationEvent(MongoPersistentEntityIndexCreator.java:46)
at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:98)
at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:333)
at org.springframework.data.mapping.context.AbstractMappingContext.addPersistentEntity(AbstractMappingContext.java:307)
at org.springframework.data.mapping.context.AbstractMappingContext.getPersistentEntity(AbstractMappingContext.java:181)
at org.springframework.data.mapping.context.AbstractMappingContext.getPersistentEntity(AbstractMappingContext.java:141)
at org.springframework.data.mapping.context.AbstractMappingContext.getPersistentEntity(AbstractMappingContext.java:67)
at org.springframework.data.mongodb.repository.support.MongoRepositoryFactory.getEntityInformation(MongoRepositoryFactory.java:141)
at org.springframework.data.mongodb.repository.support.MongoRepositoryFactory.getTargetRepository(MongoRepositoryFactory.java:83)
at org.springframework.data.repository.core.support.RepositoryFactorySupport.getRepository(RepositoryFactorySupport.java:158)
at org.springframework.data.repository.core.support.RepositoryFactoryBeanSupport.initAndReturn(RepositoryFactoryBeanSupport.java:224)
at org.springframework.data.repository.core.support.`enter code here`RepositoryFactoryBeanSupport.afterPropertiesSet(RepositoryFactoryBeanSupport.java:210)
at org.springframework.data.mongodb.repository.support.MongoRepositoryFactoryBean.afterPropertiesSet(MongoRepositoryFactoryBean.java:108)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1612)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1549)
Related
I am using spring boot 2.2 version and elastic search 6.4 version
And using spring data elastic search which extends CurdRepository
Problem is saveAll() on elastic search spring data jpa is taking 30 sec on the first iteration to save 500 records and after some iterations it is 14 or 15 sec to save 500 records which is more time and this is happening when the object is processed from cache but the same 500 records took 1 sec when it is fetched from elastic search and saved.
Elastic search document has more columns with nested objects
Elastic search document
#Document(indexname = "example", type = "example1", shards = 1, replicas =0, refreshInterval = -1)
public class Example1 {
...
private Example2 example2;
private Example3 example3;
private List<Example6> example6List;
...
}
#Document(indexname = "example", type = "example2", shards = 1, replicas =0, refreshInterval = -1)
public class Example1 {
...
private Example4 example4;
...
}
#Document(indexname = "example", type = "example3", shards = 1, replicas =0, refreshInterval = -1)
public class Example1 {
...
private Example5 example5;
private List<Example3> example3List;
}
Elastic search repository class
public interface Example1Repository extends ElasticSearchRepository {
}
Service class
#RequiredArgsConstructor
public class ExampleService {
private final Example1Repository example1Respository;
public void exampleMethod1() {
//read data from cache
// process the logic and save the list
// save to elastic search
example1Repository.saveAll(list); <-- this is taking 30 sec for 500 records to save
}
public void exampleMethod2() {
//read data from cache
//process the logic and save the list
//read data from **elastic search** and iterate and set the values from above list
//save to elastic search
example1Repository.saveAll(list); <-- this is taking 1 sec for 500 records to save
}
}
How to improve the performance in exampleMethod1() approach ? or what is causing performance issue ?
Thank you
Spring Boot 2.2 means Spring Data Elasticsearch 3.2, both versions are out of maintenance quite a while.
From a Spring Data Elasticsearch both calls do the same: Create a bulkindex request, send that to Elasticsearch and then do a refresh.
The difference in these requests might be - and I cannot see this from the info you provide - that the first time there is no id set in the objects.
I don't know if Elasticsearch is faster on a bulk index when there already are ids in the document. But you should consider the information about bulk requests from the Elasticsearch documentation for the current version.
Have you checked if there are any relevant messages in the Elasticsearch log?
I am using simple Entity with 2 cols only. In my table PK col is varchar in db but in actual values stored is numerical in that col and other col is int. DB is MS SQL Server. And this table has 165 million records.
Table structure:
SECURITY_TAB
varchar(30) SECID PK;
int VERSION;
Entity
#Entity
#Table(name = "SECURITY_TAB")
public class Security {
#Id
#Column
private String secid;
#Column
private int version;
//... getter n setter n toString
}
Repository
public interface SecurityRepository extends JpaRepository<Security, String> {
#Query("SELECT s from Security s where secid= :secid")
Security findSecurityBySecid(#Param("secid") String secid)); //this is slow
//below is fine.
//#Query("SELECT s from Security s where secid='1111'")
//Security findSecurityBySecid();
}
TestClass
#RunWith(SpringRunner.class)
#SpringBootTest
public class SecurityTests {
#Autowired
private SecurityRepository securityRepository;
#Test
public void testSecurityBySecid() {
Instant start = Instant.now();
Security security = securityRepository.findSecurityBySecid("1111");
System.out.println(Duration.between(start, Instant.now()));
System.out.println(security);
System.out.println(Duration.between(start, Instant.now()));
}
}
This simple query is taking more than 20 secs,
While when I run similar query MS SQL Server Mgmt Studio or hard code the value in the query result is returned in mill seconds. So what is going wrong?
was able to solve:
The solution was to setup this property in the jdbc url : sendStringParametersAsUnicode=false
Full example if you are using MS SQL official driver : jdbc:sqlserver://yourserver;instanceName=yourInstance;databaseName=yourDBName;sendStringParametersAsUnicode=false;
Solution: sql server query running slow from java
I have the following document:
#Document(collection = "decision_analysis")
public class DecisionAnalysis implements Serializable {
#Id
private String id;
...
}
I need to delete multiple MongoDB documents via List<String> decisionAnalysisIds
How to properly write Spring Data MongoDB repository method in order to do it?
The following doesn't work :
void deleteByIds(List<String> decisionAnalysisIds); - error: No property ids found for type DecisionAnalysis! Did you mean 'id'?
void deleteById(List<String> decisionAnalysisIds); - works, but delete only one document
Use the in clause like this:
void deleteByIdIn(List<String> ids);
I have an entity ProjectCycle mapped to mongo DB collection ProjectCycle. I am trying to retrieve 2 fields, _id and Status. I am able to retrieve both like the following
#Document(collection="ProjectCycle")
public class ProjectCycle {
#Id
private String id;
#Field("Status")
private String status;
//getters and setters
}
Application.java
Query query = new Query();
query.fields().include("Status");
Criteria criteria = new Criteria();
criteria.and("_id").is("1000");
query.addCriteria(criteria);
Iterable<ProjectCycle> objectList = mongoOperations.find(query, ProjectCycle.class);
for(ProjectCycle obj : objectList) {
System.out.println("_id "+obj.getId());
System.out.println("status "+obj.getStatus());
}
Output
_id 1000
status Approved
But, the problem is when i use an Entity with field private DBObject basicDbObject; instead of private String status; i am getting value as null instead of Approved
I have tried like the following
public class ProjectCycle {
#Id
private String id;
private DBObject basicDbObject;
//getter & setter
}
What I am trying to achieve is that, the collection 'ProjectCycle' is very large and creating a POJO corresponding to it is quiet difficult. Also I am only reading data from mongoDB. So creating the entire POJO is time wasting and tedious.
How I can achieve mapping between any field/fields from mongo Collection to entity?.
Will it be possible to create a Map<String, BasicDBObject> objectMap; to fields returned from query? I am using Spring-data-mongodb for the same.
Version details
Spring 4.0.7.RELEASE
spring-data-mongodb 1.7.2.RELEASE
Try mapping your query like below.
Iterable<BasicDBObject> objectList = mongoOperations.find(query, BasicDBObject.class, collectionname);
for(BasicDBObject obj : objectList) {
System.out.println("_id "+obj.get("id"));
System.out.println("status "+obj.get("status"));
}
I have been working in Spring Boot with the Spring Data MongoDB project and I am seeing behavior I am not clear on. I understand that the id field will go to _id in the Mongo repository per http://docs.spring.io/spring-data/mongodb/docs/current/reference/html/#mapping.conventions.id-field. My problem is that it also seems to be happening for child entities which does not seem correct.
For example I have these classes (leaving out setters and getters for brevity) :
public class MessageBuild {
#Id
private String id;
private String name;
private TopLevelMessage.MessageType messageType;
private TopLevelMessage message;
}
public interface TopLevelMessage {
public enum MessageType {
MapData
}
}
public class MapData implements TopLevelMessage {
private String layerType;
private Vector<Intersection> intersections;
private Vector<RoadSegment> roadSegments;
}
public class RoadSegment {
private int id;
private String name;
private Double laneWidth;
}
and I create an object graph using this I use the appropriate MongoRepository class to save I end up with an example document like this (with _class left out):
{
"_id" : ObjectId("57c0c05568a6c4941830a626"),
"_class" : "com.etranssystems.coreobjects.persistable.MessageBuild",
"name" : "TestMessage",
"messageType" : "MapData",
"message" : {
"layerType" : "IntersectionData",
"roadSegments" : [
{
"_id" : 2001,
"name" : "Road Segment 1",
"laneWidth" : 3.3
}
]
}
}
In this case a child object with a field named id has its mapping converted to _id in the MongoDB repository. Not the end of the world although not expected. The biggest problem is now that this is exposed by REST MVC the _id fields are not returned from a query. I have tried to set the exposeIdsFor in my RepositoryRestConfigurerAdapter for this class and it exposes the id for the top level document but not the child ones.
So circling around the 2 questions/issues I have are:
Why are child object fields mapped to _id? My understanding is that this should only happen on the top level since things underneath are not really documents in their own right.
Shouldn't the configuration to expose id fields work for child objects in a document if it is mapping the field names?
Am I wrong to think that RoadSegment does not contain a getId() ? From Spring's documentation:
A property or field without an annotation but named id will be mapped
to the _id field.
I believe Spring Data does this even to nested classes, when it finds an id field. You may either add a getId(), so that the field is named id or annotate it with #Field:
public class RoadSegment {
#Field("id")
private int id;
private String name;
private Double laneWidth;
}
I agree this automatic conversion of id/_id should only be done at the top level in my opinion.
However, the way Spring Data Mongo conversion is coded, all java ojects go through the exact same code to be converted into json (both top and nested objects):
public class MappingMongoConverter {
...
protected void writeInternal(Object obj, final DBObject dbo, MongoPersistentEntity<?> entity) {
...
if (!dbo.containsField("_id") && null != idProperty) {
try {
Object id = accessor.getProperty(idProperty);
dbo.put("_id", idMapper.convertId(id));
} catch (ConversionException ignored) {}
}
...
if (!conversions.isSimpleType(propertyObj.getClass())) {
// The following line recursively calls writeInternal with the nested object
writePropertyInternal(propertyObj, dbo, prop);
} else {
writeSimpleInternal(propertyObj, dbo, prop);
}
}
writeInternal is called on the top level object, and then recalled recursively for each subobjects (aka SimpleTypes). So they both go through the same logic of adding _id.
Perhaps this is how we should read Spring's documentation:
Mongo's restrictions on Mongo Documents:
MongoDB requires that you have an _id field for all documents. If you
don’t provide one the driver will assign a ObjectId with a generated
value.
Spring Data's restrictions on java classes:
If no field or property specified above is present in the Java class
then an implicit _id file will be generated by the driver but not
mapped to a property or field of the Java class.