I wonder how can i build a Cypher query that will combine Fulltext and Simple indexes using spring data neo4j. Consider the following node entity:
#NodeEntity
public class SomeObject {
public SomeObject() {
}
public SomeObject(String name, int height) {
this.name = name;
this.height = height;
}
#Indexed(indexType = IndexType.FULLTEXT, indexName = "search_name")
String name;
#Indexed(numeric = false)
int height;
public String getName() {
return name;
}
public void setName(String name) {
this.name = name;
}
}
OK, so my question is how can i run a query (By using a SomeObject Graph Repository) that will start from SomeObject nodes, by referencing their simple indexes and full-text indexes in the same query. For example i would like to write something like that:
START n=node:SomeObject('name: Roy AND height: [170 TO 190]') RETURN n
I know that i cannot write it exactly like that, because spring data neo4j forces me to give a seperate index name for fields that are needed to be FULLTEXT indexed. But what if i need make an index lookup for my SomeObject entity which combines both fileds? (name & height)
What are the best practices in such case? Is there a way to combine them both in the same query? or maybe should i query each of them separably, and then to perform some some kind of intersection between the two results, so i will get exactly the nodes the meet my original query lookup condition? (name: Roy AND height: [170 TO 190]).
Thanks!
Roy.
I'd never launch two separate queries. Maybe just use one index as a starting point in your query?
START n=node:search_name('name: Roy')
WHERE n.height >= 170 AND n.height <= 190
RETURN n
How's the performance of this query? This bypasses the SomeObject index, but I don't see any other option as you indeed cannot combine both indexes.
I as also thinking about the following query, but you'd still end up with duplicates:
START n=node:search_name('name: Roy'), m=node:SomeObject('height: [170 TO 190]')
RETURN DISTINCT n,m
Related
I created one class
class Employee { Integer id; String name; String departments; }
and in sql server database i have records
I stored departments as ";" separated. For Example Department = Computer;Civil
1,Chaitanya,Computer;Civil
2,Tom,Physics;Chemistry
3,Harry,Economics;Commerce
4,Henry,Computer;Civil;Mechanical
5,Ravi,null
Now i want to filter data with departments let's say there is one multiselect in frontend where i have list of departments and i select two departments for example-> Computer,Civil and in backend i got List<String> deparmentFilter as parameter say Computer;Civil
Now as per my requirement i have to return two data from Spring Boot Controller
1,Chaitanya,Computer;Civil
4,Henry,Computer;Civil;Mechanical
Right Now what i did is i executed the query to fetch all the records and then i right below logic
List<Employee> employeesToBeRemoved = new ArrayList<>();
if (!departmentNames.isEmpty()) {
allEmployees.forEach(employee -> {
if (employee.getDepartment() != null) {
Set<String> departmentNamesResult = new HashSet<>(Arrays.asList(employee.getDepartment().
split(";")));
Boolean isExist = Collections.disjoint(departmentNamesResult, departmentNames);
if (Boolean.TRUE.equals(isExist)) {
employeesToBeRemoved.add(employee);
}
} else {
employeesToBeRemoved.add(employee);
}
});
}
allEmployees.removeAll(employeesToBeRemoved);
I tried to move it to predicates but not able to do that, This solution is taking much time to execute,
Please suggest me some other better ways (optimized way) to improve performance.
Is there is any way to add this filter in predicates?
Another approach i am thinking (12/05/2022)
Let's say i have one table employee_department_mapping and in that table i have employeeId and departmentName so in this correct way to add predicate?
CriteriaQuery<Object> subQuery1 = criteriaBuilder.createQuery();
Root<EmployeeDepartmentMapping> subQueryEmpDptMp = subQuery1.from(EmployeeDepartmentMapping.class);
predicates1.add(subQueryEmpDptMp.get("departmentName").in(departmentNames));
You might achieve better performance by splitting your table and using join:
class Employee { Integer id; String name; Integer departmentsId; }
class EmployeeDepartments { Integer departmentsId; String department; }
You may use Element Collection to achieve this.
Now, instead of having a the following row:
1,Chaitanya,Computer;Civil
You will have the following:
table1:
1,Chaitanya,123
table2:
123,Compter
123,Civil
Execute a join to get all row from table2 with table1 to get your result
I have a document like that:
'subject' : {
'name' :"...."
'facebookPosts':[
{
date:"14/02/2017 20:20:03" , // it is a string
text:"facebook post text here",
other stuff here
}
]
}
and I want to count the facebookPosts within a specific objects that their date field contains e.g "23/07/2016".
Now, I do that by extracting all the documents and count in the client side (spring ) , But I think that's not efficient.
You need to aggregate your results.
final Aggregation aggregation = Aggregation.newAggregation(
Aggregation.match(Criteria.where("facebookPosts.date").regex(REGEX)),
Aggregation.unwind("facebookPosts"),
Aggregation.group().count().as("count"));
Regex might not be the best solution, just an example.
unwind will split array into separate elements you can then count.
Create a class that will hold the count, something like:
public class PostCount {
private Long count;
// getters, setters
}
And then execute it like this:
AggregationResults<PostCount> postCount = mongoTemplate.aggregate(aggregation, Subject.class, PostCount.class);
long count = postCount.getMappedResults().get(0).getCount();
In spring data mongodb using mongotemplate or mongorepository, how to achieve pagination for aggregateion
This is an answer to an old post, but I'll provide an answer in case anyone else comes along while searching for something like this.
Building on the previous solution by Fırat KÜÇÜK, giving the results.size() as the value for the "total" field in the PageImpl constructor will not making paging work the way, well, you expect paging to work. It sets the total size to the page size every time, so instead, you need to find out the actual total number of results that your query would return:
public Page<UserListItemView> list(final Pageable pageable) {
long total = getCount(<your property name>, <your property value>);
final Aggregation agg = newAggregation(
skip(pageable.getPageNumber() * pageable.getPageSize()),
limit(pageable.getPageSize())
);
final List<UserListItemView> results = mongoTemplate
.aggregate(agg, User.class, UserListItemView.class)
.getMappedResults();
return new PageImpl<>(results, pageable, total);
}
Now, then, the best way to get the total number of results is another question, and it is one that I am currently trying to figure out. The method that I tried (and it worked) was to almost run the same aggregation twice, (once to get the total count, and again to get the actual results for paging) but using only the MatchOperation followed by a GroupOperation to get the count:
private long getCount(String propertyName, String propertyValue) {
MatchOperation matchOperation = match(Criteria.where(propertyName).is(propertyValue));
GroupOperation groupOperation = group(propertyName).count().as("count");
Aggregation aggregation = newAggregation(matchOperation, groupOperation);
return mongoTemplate.aggregate(aggregation, Foo.class, NumberOfResults.class).getMappedResults().get(0).getCount();
}
private class NumberOfResults {
private int count;
public int getCount() {
return count;
}
public void setCount(int count) {
this.count = count;
}
}
It seems kind of inefficient to run nearly the same query twice, but if you are going to page results, the pageable object must know the total number of results if you really want it to behave like paging. If anyone can improve on my method to get the total count of results, that would be awesome!
Edit: This will also provide the count, and it is simpler because you do not need a wrapper object to hold the result, so you can replace the entire previous code block with this one:
private long getCount(String propertyName, String propertyValue) {
Query countQuery = new Query(Criteria.where(propertyName).is(propertyValue));
return mongoTemplate.count(countQuery, Foo.class);
}
In addition to ssouris solution you can use Pageable classes for the results.
public Page<UserListItemView> list(final Pageable pageable) {
final Aggregation agg = newAggregation(
skip(pageable.getPageNumber() * pageable.getPageSize()),
limit(pageable.getPageSize())
);
final List<UserListItemView> results = mongoTemplate
.aggregate(agg, User.class, UserListItemView.class)
.getMappedResults();
return new PageImpl<>(results, pageable, results.size())
}
You can use MongoTemplate
org.spring.framework.data.mongodb.core.aggregation.Aggregation#skip
and
org.springframework.data.mongodb.core.aggregation.Aggregation#limit
Aggregation agg = newAggregation(
project("tags"),
skip(10),
limit(10)
);
AggregationResults<TagCount> results = mongoTemplate.aggregate(agg, "tags", TagCount.class);
List<TagCount> tagCount = results.getMappedResults();
As per the answer https://stackoverflow.com/a/39784851/4546949 I wrote code for Java.
Use aggregation group to get count and array of data with other paging information.
AggregationOperation group = Aggregation.group().count().as("total")
.addToSet(pageable.getPageNumber()).as("pageNumber")
.addToSet(pageable.getPageSize()).as("pageSize")
.addToSet(pageable.getOffset()).as("offset")
.push("$$ROOT").as("data");
Use Aggregation project to slice as per the paging information.
AggregationOperation project = Aggregation.project()
.andInclude("pageSize", "pageNumber", "total", "offset")
.and(ArrayOperators.Slice.sliceArrayOf("data").offset((int) pageable.getOffset()).itemCount(pageable.getPageSize()))
.as("data");
Use mongo template to aggregate.
Aggregation aggr = newAggregation(group, project);
CustomPage page = mongoTemplate.aggregate(aggregation, Foo.class, CustomPage.class).getUniqueMappedResult();
Create a CustomPage.
public class CustomPage {
private long pageSize;
private long pageNumber;
private long offset;
private long total;
private List<Foo> data;
}
Here is my generic solution:
public Page<ResultObject> list(Pageable pageable) {
// build your main stages
List<AggregationOperation> mainStages = Arrays.asList(match(....), group(....));
return pageAggregation(pageable, mainStages, "target-collection", ResultObject.class);
}
public <T> Page<T> pageAggregation(
final Pageable pageable,
final List<AggregationOperation> mainStages,
final String collection,
final Class<T> clazz) {
final List<AggregationOperation> stagesWithCount = new ArrayList<>(mainStages);
stagesWithCount.add(count().as("count"));
final Aggregation countAgg = newAggregation(stagesWithCount);
final Long count = Optional
.ofNullable(mongoTemplate.aggregate(countAgg, collection, Document.class).getUniqueMappedResult())
.map(doc -> ((Integer) doc.get("count")).longValue())
.orElse(0L);
final List<AggregationOperation> stagesWithPaging = new ArrayList<>(mainStages);
stagesWithPaging.add(sort(pageable.getSort()));
stagesWithPaging.add(skip(pageable.getOffset()));
stagesWithPaging.add(limit(pageable.getPageSize()));
final Aggregation resultAgg = newAggregation(stagesWithPaging);
final List<T> result = mongoTemplate.aggregate(resultAgg, collection, clazz).getMappedResults();
return new PageImpl<>(result, pageable, count);
}
To return a Paged Object with correct value of pageable object , I find this is the best and simple way.
Aggregation aggregation = Aggregation.newAggregation(Aggregation.match(Criteria.where("type").is("project")),
Aggregation.group("id").last("id").as("id"), Aggregation.project("id"),
Aggregation.skip(pageable.getPageNumber() * pageable.getPageSize()),
Aggregation.limit(pageable.getPageSize()));
PageableExecutionUtils.getPage(mongoTemplate.aggregate(aggregation, Draft.class, Draft.class).getMappedResults(), pageable,() -> mongoTemplate.count(Query.of(query).limit(-1).skip(-1), Draft.class));
Another approach would be to extend the PagingAndSortingRepository<T, ID> interface. Then, you can create an #Aggregation query method like this:
#Aggregation(pipeline = {
"{ $match: { someField: ?0 } }",
"{ $project: { _id: 0, someField: 1} }"
})
List<StuffAggregateModel> aggregateStuff(final String somePropertyName, final Pageable pageable);
Just call this from your business logic service class and construct the Pageable (which also contains sort options, if desired) and call the repo method. I like this approach because of the simplicity and the sheer minimization of the amount of code that you have to write. If your query (aggregation pipeline) is simple enough, this is probably the best solution. Maintenance coding for this approach is nearly effortless.
My answer with MongoDB $facet
// User(_id, first name, etc), Car (user_id, brand, etc..)
LookupOperation lookupStageCar = Aggregation.lookup(‘cars ’, ‘user_id’, ‘_id’, ‘car’);
MatchOperation matchStage = Aggregation.match(Criteria.where(‘car.user_id ‘).exists(true));
CountOperation countOperation = Aggregation.count().as("total");
AddFieldsOperation addFieldsOperation = Aggregation.addFields().addFieldWithValue("page", pageable.getPageNumber()).build();
SkipOperation skipOperation = Aggregation.skip(Long.valueOf(pageable.getPageNumber() * pageable.getPageSize()));
LimitOperation limitOperation = Aggregation.limit(pageable.getPageSize());
// here the magic
FacetOperation facetOperation = Aggregation.facet( countOperation, addFieldsOperation).as("metadata")
.and(skipOperation, limitOperation).as("data");
// users with car
List<AggrigationResults> map = mongoTemplate.aggregate(Aggregation.newAggregation( lookupStageCar, matchStage, facetOperation), "User", AggrigationResults.class).getMappedResults();
———————————————————————————
public class AggrigationResults {
private List<Metadata> metadata;
private List<User> data;
}
public class Metadata {
private long total;
private long page;
}
———————————————————————————
output:
{
"metadata" : [
{
"total" : 300,
"page" : 3
}
],
"data" : [
{
... original document ...
},
{
... another document ...
},
{
... etc up to 10 docs ...
}
]
}
see : How to use MongoDB aggregation for pagination?
I'm trying to find Near by places.
Below code is working fine.
But i'm not able to get actual distance of place from my given lat,lng.
Criteria criteria = new Criteria("coordinates")
.near(new Point(searchRequest.getLat(),searchRequest.getLng()));
Query query = new Query();
query.addCriteria(criteria);
query.addCriteria(criteriaName);
query.limit(5);
List<Place> ls = (List<Place>) mongoTemplate.find(query, Place.class);
You can do it with geoNear aggregation. In spring-data-mongodb GeoNearOperation is representing this aggregation.
Extend or create inherit Place class with field where you would like to have distance information (example with inheritance):
public class PlaceWithDistance extends Place {
private double distance;
public double getDistance() {
return distance;
}
public void setDistance(final double distance) {
this.distance = distance;
}
}
Instead of Criteria with Query use aggregation. Second argument of geoNear is name of field where distance should be set:
final NearQuery nearQuery = NearQuery
.near(new Point(searchRequest.getLat(), searchRequest.getLng()));
nearQuery.num(5);
nearQuery.spherical(true); // if using 2dsphere index, otherwise delete or set false
// "distance" argument is name of field for distance
final Aggregation a = newAggregation(geoNear(nearQuery, "distance"));
final AggregationResults<PlaceWithDistance> results =
mongoTemplate.aggregate(a, Place.class, PlaceWithDistance.class);
// results.forEach(System.out::println);
List<PlaceWithDistance> ls = results.getMappedResults();
Just to make it easier - associated imports:
import static org.springframework.data.mongodb.core.aggregation.Aggregation.geoNear;
import static org.springframework.data.mongodb.core.aggregation.Aggregation.newAggregation;
import org.springframework.data.mongodb.core.aggregation.Aggregation;
import org.springframework.data.mongodb.core.aggregation.AggregationResults;
import org.springframework.data.mongodb.core.aggregation.GeoNearOperation;
import org.springframework.data.mongodb.core.query.NearQuery;
Walery Strauch's example was useful for me...
However I wanted to :
run aggregate query to get all the points in 2dsphere index with-in given distance in Kilometers or Meters. You can use Metrics.KILOMETERS & Metrics.MILES
collection name is not specified as part of pojo
I have 2dsphere index with old way of representation in MongoDB. I am using Mongo as sharded databased for Geo-Spatial queries. My nearSphere query (without aggregation) was failing only when there is a shard key added into the same collection where I have 2dsphere index.
After using below implementation with shard key in the same collection. I am successfully able to fetch the required data.
Here is the sample :
import org.springframework.data.geo.Metrics;
final NearQuery query = NearQuery.near(new Point(longitude, latitude), Metrics.KILOMETERS)
.num(limit)
.minDistance(distanceInKiloMeters)
.maxDistance(maxNearByUEDistanceInKiloMeters)
.spherical(true);
final Aggregation a = newAggregation(geoNear(query, "distance"));
final AggregationResults<PlaceWithDistance> results = offlineMongoTemplate.aggregate(a, "myCollectionName", PlaceWithDistance.class);
final List<PlaceWithDistance> measurements = new ArrayList<PlaceWithDistance>(results.getMappedResults());
Is it possible to fetch data in user defined ranges [int starting record -int last record]?
In my case user will define in query String in which range he wants to fetch data.
I have tried something like this
Pageable pageable = new PageRequest(0, 10);
Page<Project> list = projectRepository.findAll(spec, pageable);
Where spec is my defined specification but unfortunately this do not help.
May be I am doing something wrong here.
I have seen other spring jpa provided methods but nothing are of much help.
user can enter something like this localhost:8080/Section/employee? range{"columnName":name,"from":6,"to":20}
So this says to fetch employee data and it will fetch the first 15 records (sorted by columnName ) does not matter as of now.
If you can suggest me something better that would be great.if you think I have not provided enough information please let me know, I will provide required information.
Update :I do not want to use native or Create query statements (until I don't have any other option).
May be something like this:
Pageable pageable = new PageRequest(0, 10);
Page<Project> list = projectRepository.findAll(spec, new pageable(int startIndex,int endIndex){
// here my logic.
});
If you have better options, you can suggest me that as well.
Thanks.
Your approach didn't work, because new PageRequest(0, 10); doens't do what you think. As stated in docs, the input arguments are page and size, not limit and offset.
As far as I know (and somebody correct me if I'm wrong), there is no "out of the box" support for what you need in default SrpingData repositories. But you can create custom implementation of Pagable, that will take limit/offset parameters. Here is basic example - Spring data Pageable and LIMIT/OFFSET
We can do this with Pagination and by setting the database table column name, value & row counts as below:
#Transactional(readOnly=true)
public List<String> queryEmployeeDetails(String columnName,String columnData, int startRecord, int endRecord) {
Query query = sessionFactory.getCurrentSession().createQuery(" from Employee emp where emp.col= :"+columnName);
query.setParameter(columnName, columnData);
query.setFirstResult(startRecord);
query.setMaxResults(endRecord);
List<String> list = (List<String>)query.list();
return list;
}
If I am understanding your problem correctly, you want your repository to allow user to
Provide criteria for query (through Specification)
Provide column to sort
Provide the range of result to retrieve.
If my understanding is correctly, then:
In order to achieve 1., you can make use of JpaSpecificationExecutor from Spring Data JPA, which allow you to pass in Specificiation for query.
Both 2 and 3 is achievable in JpaSpecificationExecutor by use of Pagable. Pageable allow you to provide the starting index, number of record, and sorting columns for your query. You will need to implement your range-based Pageable. PageRequest is a good reference on what you can implement (or you can extend it I believe).
So i got this working as one of the answer suggested ,i implemented my own Pageable and overrided getPagesize(),getOffset(),getSort() thats it.(In my case i did not need more)
public Range(int startIndex, int endIndex, String sortBy) {
this.startIndex = startIndex;
this.endIndex = endIndex;
this.sortBy = sortBy;
}
#Override
public int getPageSize() {
if (endIndex == 0)
return 0;
return endIndex - startIndex;
}
#Override
public int getOffset() {
// TODO Auto-generated method stub
return startIndex;
}
#Override
public Sort getSort() {
// TODO Auto-generated method stub
if (sortBy != null && !sortBy.equalsIgnoreCase(""))
return new Sort(Direction.ASC, sortBy);
else
return new Sort(Direction.ASC, "id");
}
where startIndex ,endIndex are starting and last index of record.
to access it :
repository.findAll(spec,new Range(0,20,"id");
There is no offset parameter you can simply pass. However there is a very simple solution for this:
int pageNumber = Math.floor(offset / limit) + ( offset % limit );
PageRequest pReq = PageRequest.of(pageNumber, limit);
The client just have to keep track on the offset instead of page number. By this I mean your controller would receive the offset instead of the page number.
Hope this helps!