Spring data mongodb Aggregation group and count - spring

I want to make group and count a collection data by its nested field type.name. However, spring data mongo db's Aggregation count static method return 1 as count for each type, eventhough there are more then 1 data for each type. Is there any mistake my pipeline functions ? If so could you correct me please ? Thanks in advance
My collection data structure is:
java code to fetch data:
private List<OrganizationSummary> getOrganizationSummary(){
Aggregation agg = newAggregation(
match(Criteria.where("status").is(Boolean.TRUE).and("deleted").is(Boolean.FALSE)),
group("type.name").count().as("count"),
project("count").and("name").previousOperation()
);
AggregationResults<OrganizationSummary> groupResults
= mongoTemplate.aggregate(agg, Organization.class, OrganizationSummary.class);
List<OrganizationSummary> result = groupResults.getMappedResults();
return result;
}
result from debugger. As you see all count fields are default 1, eventhough there are more then 1 data for each type.
Result from mongoshell:
Result dto to be projected:
#Getter
#Setter
#Builder
#AllArgsConstructor
#NoArgsConstructor
public class OrganizationSummary {
private String name;
private Long count;
}
Logs after mongoTemplate.aggregate() method is executed.
06-10-2022 15:04:18 [DEBUG] org.springframework.data.mongodb.core.MongoTemplate : Executing aggregation: [{ "$match" : { "status" : true, "deleted" : false}}, { "$group" : { "_id" : "$type.name", "count" : { "$sum" : { "$date" : "1970-01-01T00:00:00.001Z"}}}}, { "$project" : { "count" : { "$date" : "1970-01-01T00:00:00.001Z"}, "_id" : { "$date" : "1970-01-01T00:00:00Z"}, "name" : "$_id"}}] in collection organization
06-10-2022 15:04:18 [DEBUG] org.mongodb.driver.protocol.command : Sending command '{"aggregate": "organization", "pipeline": [{"$match": {"status": true, "deleted": false}}, {"$group": {"_id": "$type.name", "count": {"$sum": {"$date": "1970-01-01T00:00:00.001Z"}}}}, {"$project": {"count": {"$date": "1970-01-01T00:00:00.001Z"}, "_id": {"$date": "1970-01-01T00:00:00Z"}, "name": "$_id"}}], "cursor": {}, "allowDiskUse": false, "$db": "organizationidentity", "lsid": {"id": {"$binary": {"base64": "GA/dUO4HRlWxF1m9NVs1iQ==", "subType": "04"}}}}' with request id 22 to database organizationidentity on connection [connectionId{localValue:3, serverValue:78}] to server localhost:27017
06-10-2022 15:04:18 [DEBUG] org.mongodb.driver.protocol.command : Execution of command with request id 22 completed successfully in 13.70 ms on connection [connectionId{localValue:3, serverValue:78}] to server localhost:27017
06-10-2022 15:04:18 [DEBUG] org.mongodb.driver.operation

Related

How to fire DRL rules deployed on KIE Server using spring boot?

I created a simple project in Drools work bench. The project has 1 data object and 2 DRL files. I build and deployed the project to the KIE server. And I have created a simple spring boot application which loads data into the data object using a rest service and fires the rules. Below is the code:
public class Main {
private static String containerId = "ImportProducts_1.0.1-LATEST";
private static String user = "kieserver";
private static String password = "kieserver1!";
private static String url = "http://localhost:8180/kieserver/services/rest/server";
private static final Logger LOGGER = LoggerFactory.getLogger(Main.class);
private static final MarshallingFormat FORMAT = MarshallingFormat.JSON;
private static String CLASS_NAME = "ImportProduct";
public static void main(String[] args) {
List<ImportProduct> prods = new ArrayList<>();
prods.add(new ImportProduct("1", "Grocery - Milk", "OK", 25.0));
prods.add(new ImportProduct("2", "Fashion - Trouser", "NOT_OK", 1300.0));
prods.add(new ImportProduct("3", "Grocery - Wheat", "OK", 425.0));
prods.add(new ImportProduct("4", "Grocery - Dairy Milk Chocolate", "OK", 100.0));
KieServicesConfiguration config = KieServicesFactory.newRestConfiguration(url, user, password, 60000);
config.setMarshallingFormat(MarshallingFormat.JSON);
RuleServicesClient client = KieServicesFactory.newKieServicesClient(config).getServicesClient(RuleServicesClient.class);
List<Command<?>> cmds = new ArrayList<>();
KieCommands commands = KieServices.Factory.get().getCommands();
cmds.add(commands.newInsert(prods, CLASS_NAME));
cmds.add(commands.newFireAllRules());
BatchExecutionCommand myCommands = CommandFactory.newBatchExecution(cmds);
ServiceResponse<ExecutionResults> response = client.executeCommandsWithResults(containerId, myCommands);
if (response.getType() == ServiceResponse.ResponseType.SUCCESS) {
LOGGER.info("Commands executed with success! Response: ");
LOGGER.info("{}", response.getResult());
List<ImportProduct> prodUpdated = (List<ImportProduct>) response.getResult().getValue(CLASS_NAME);
//sale.setDiscount(saleUpdated.getDiscount());
LOGGER.info("Response is: {}", response.getMsg());
LOGGER.info("Output is: {}", prodUpdated.toString());
} else {
LOGGER.error("Error executing rules. Message: {}", response.getMsg());
}
//KieServices kieServices = KieServices.Factory.get();
//ReleaseId releaseId = (ReleaseId) kieServices.newReleaseId( "com.test", "ImportProducts", "1.0.1-LATEST" );
}
}
The application runs without errors but the there is no impact of the rules on the data fed in the KIE container. The rule deletes the data object if the status != 'OK'. I am getting all the data back I sent in the request body of the POST request service. I think DRL capability is enabled as shown in the stack trace below:
22:11:40.139 [main] DEBUG org.kie.server.client.impl.AbstractKieServicesClientImpl - About to deserialize content:
'{
"type" : "SUCCESS",
"msg" : "Kie Server info",
"result" : {
"kie-server-info" : {
"id" : "kie-server-a017ffcb29dc",
"version" : "7.55.0.Final",
"name" : "kie-server-a017ffcb29dc",
"location" : "http://172.17.0.3:8080/kie-server/services/rest/server",
**"capabilities" : [ "KieServer", "BRM", "BPM", "CaseMgmt", "BPM-UI", "BRP", "DMN", "Swagger" ],**
"messages" : [ {
"severity" : "INFO",
"timestamp" : {
"java.util.Date" : 1623926563857
},
"content" : [ "Server KieServerInfo{serverId='kie-server-a017ffcb29dc', version='7.55.0.Final', name='kie-server-a017ffcb29dc', location='http://172.17.0.3:8080/kie-server/services/rest/server', capabilities=[KieServer, BRM, BPM, CaseMgmt, BPM-UI, BRP, DMN, Swagger]', messages=null', mode=DEVELOPMENT}started successfully at Thu Jun 17 10:42:43 UTC 2021" ]
} ],
"mode" : "DEVELOPMENT"
}
}
}'
Below is the response I got back:
22:11:41.515 [main] DEBUG org.kie.server.client.impl.AbstractKieServicesClientImpl - About to deserialize content:
'{
"type" : "SUCCESS",
"msg" : "Container ImportProducts_1.0.1-LATEST successfully called.",
"result" : {
"execution-results" : {
"results" : [ {
"value" : [{"com.test.importproducts.ImportProduct":{
"id" : "1",
"category" : "Grocery - Milk",
"status" : "OK",
"price" : 25.0
}},{"com.test.importproducts.ImportProduct":{
"id" : "2",
"category" : "Fashion - Trouser",
"status" : "NOT_OK",
"price" : 1300.0
}},{"com.test.importproducts.ImportProduct":{
"id" : "3",
"category" : "Grocery - Wheat",
"status" : "OK",
"price" : 425.0
}},{"com.test.importproducts.ImportProduct":{
"id" : "4",
"category" : "Grocery - Dairy Milk Chocolate",
"status" : "OK",
"price" : 100.0
}}],
"key" : "ImportProduct"
} ],
"facts" : [ {
"value" : {"org.drools.core.common.DefaultFactHandle":{
"external-form" : "0:3:459769708:-992461270:3:DEFAULT:NON_TRAIT:java.util.ArrayList"
}},
"key" : "ImportProduct"
} ]
}
}
}'
Please help.
I fixed the issue. Below is the working code:
private Command<?> prepareCommands(List<ImportProduct> facts, String sessionName, String outIdentifier) {
KieCommands commandsFactory = KieServices.Factory.get().getCommands();
List<Command> commands = facts.stream().map(commandsFactory::newInsert).collect(Collectors.toList());
commands.add(commandsFactory.newFireAllRules());
ObjectFilter factsFilter = new ClassObjectFilter(ImportProduct.class);
//System.out.println("Check1: "+commandsFactory.newGetObjects(factsFilter, CLASS_NAME));
//System.out.println("Check2: "+commands.toString());
commands.add(commandsFactory.newGetObjects(factsFilter, CLASS_NAME));
return commandsFactory.newBatchExecution(commands, sessionName);
}

Cosmos DB Collection not using _id index when querying by _id?

I have a CosmosDb - MongoDb collection that I'm using purely as a key/value store for arbitrary data where the _id is the key for my collection.
When I run the query below:
globaldb:PRIMARY> db.FieldData.find({_id : new BinData(3, "xIAPpVWVkEaspHxRbLjaRA==")}).explain(true)
I get this result:
{
"_t" : "ExplainResponse",
"ok" : 1,
"queryPlanner" : {
"plannerVersion" : 1,
"namespace" : "data.FieldData",
"indexFilterSet" : false,
"parsedQuery" : {
"$and" : [ ]
},
"winningPlan" : {
},
"rejectedPlans" : [ ]
},
"executionStats" : {
"executionSuccess" : true,
"nReturned" : 1,
"executionTimeMillis" : 106,
"totalKeysExamined" : 0,
"totalDocsExamined" : 3571,
"executionStages" : {
},
"allPlansExecution" : [ ]
},
"serverInfo" : #REMOVED#
}
Notice that the totalKeysExamined is 0 and the totalDocsExamined is 3571 and the query took over 106ms. If i run without .explain() it does find the document.
I would have expected this query to be lightning quick given that the _id field is automatically indexed as a unique primary key on the collection. As this collection grows in size, I only expect this problem to get worse.
I'm definitely not understanding something about the index and how it works here. Any help would be most appreciated.
Thanks!

$sort pipeline after $group not working using Spring aggregate class

I have a User collection which looks like below sample :
User :{
"_id" : ObjectId("59f6dc660a975a3e3290ea01"),
"basicInfo" : {
"name" : "xxxx",
"age" : 27,
"gender" : "Male"
}
"otherInfo" {
"projects" : [
{
"_id" : ObjectId("59f6f9230a975a67cc7d7638"),
"name" : "Test Project",
"projectImage" : "images/project/59f6f9230a975a67cc7d7638.jpg",
"desc" : "This is a testing project",
"status" : "Active",
"verifyDet" : {
"method" : "Admin",
"status" : "PENDING",
"isVerified" : false
}
},
{
"_id" : ObjectId("59f6f9230a975a67cc7d5556"),
"name" : "Test Project Two",
"projectImage" : "images/project/59f6f9230a975a67cc7d5556.jpg",
"desc" : "This is a testing project",
"status" : "Closed",
"verifyDet" : {
"method" : "Admin",
"status" : "APPROVED",
"isVerified" : true
}
}
]
}
}
Note: One user can be part of multiple projects. But he needs approval from Admin to participate in the project activities. Verification is managed by verifyDet and projects are managed by projects array.
Actual requirement is to show the list of members in such a way that members having verification pending comes on top in alphabetic order and then approved/verified members in alphabetic order to Admin.
When I run below query on mongo shell I get list of Users with only one project detail(_id=59f6f9230a975a67cc7d7638) for which I want to search and result sorted by Verification pending users and User name. The result comes appropriately.
db.User.aggregate(
{$unwind:"$otherInfo.projects"},
{
$match:{
"otherInfo.projects._id":ObjectId("59f6f9230a975a67cc7d7638"),
"otherInfo.projects.status":"Active"
}
},
{$group: {_id: {"_id":"$_id", "basicInfo":"$basicInfo"}, "projects": {$push: "$otherInfo.projects"}}},
{$project:{"_id":"$_id._id", "basicInfo":"$_id.basicInfo", "otherInfo.projects":"$projects"}},
{$sort:{"otherInfo.projects.verifyDet.isVerified":1, "basicInfo.name":1}}
)
But when I create same aggregate in Spring like mentioned below I get exception:
public List<Map> fetchUsersList(String projectId, Pageable pageable) {
//unwind operation
AggregationOperation unwindOp = Aggregation.unwind("$otherInfo.projects");
Criteria criteria = Criteria.where("otherInfo.projects._id").is(new ObjectId(projectId));
criteria.and("otherInfo.projects.status").is("Active");
AggregationOperation matchOp = Aggregation.match(criteria);
AggregationOperation groupOp = Aggregation.group(
Fields.from(Fields.field("_id", "$_id")).and(Fields.field("basicInfo","$basicInfo"))).push("$otherInfo.projects").as("projects");
AggregationOperation projectOp = Aggregation.project(
Fields.from(Fields.field("_id","$_id._id"),
Fields.field("basicInfo","$_id.basicInfo"),
Fields.field("otherInfo.projects","$projects")));
AggregationOperation sortOp = Aggregation.sort(Direction.DESC, "otherInfo.projects.verifyDet.isVerified").and(Direction.DESC, "basicInfo.name");
Aggregation agg = Aggregation.newAggregation(unwindOp, matchOp, groupOp, projectOp, sortOp);
AggregationResults<User> results = mongoTemplate.aggregate(agg,
"User", User.class);
return results.getMappedResults();
}
Exception :
2017-12-15 19:24:31,852 ERROR GlobalExceptionHandler:75 - Exception Stack Trace :
java.lang.IllegalArgumentException: Invalid reference 'otherInfo.projects.verifyDet.isVerified'!
at org.springframework.data.mongodb.core.aggregation.ExposedFieldsAggregationOperationContext.getReference(ExposedFieldsAggregationOperationContext.java:99)
at org.springframework.data.mongodb.core.aggregation.ExposedFieldsAggregationOperationContext.getReference(ExposedFieldsAggregationOperationContext.java:80)
at org.springframework.data.mongodb.core.aggregation.SortOperation.toDBObject(SortOperation.java:73)
at org.springframework.data.mongodb.core.aggregation.AggregationOperationRenderer.toDBObject(AggregationOperationRenderer.java:56)
at org.springframework.data.mongodb.core.aggregation.Aggregation.toDbObject(Aggregation.java:580)
at org.springframework.data.mongodb.core.aggregation.Aggregation.toString(Aggregation.java:596)
at com.grpbk.gp.repository.impl.UserRepositoryCustomImpl.fetchUsersList(UserRepositoryCustomImpl.java:1128)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
Please let me know what I am doing wrong.
"otherInfo.projects.verifyDet.isVerified" field needs to be present in Project Oepration or group operation so that sort would get a reference for that.

Spring mongodb Find document if a single field matches in a list within a document

I have some data stored as
{
"_id" : ObjectId("abc"),
"_class" : "com.xxx.Team",
"name" : "Team 1",
"members" : [
{"userId" : 1, "email" : "a#x.com" },
{"userId" : 2, "email" : "b#x.com" },
]
}
{
"_id" : ObjectId("xyz"),
"_class" : "com.xxx.Team",
"name" : "Team 2",
"members" : [
{"userId" : 2, "email" : "b#x.com" },
{"userId" : 3, "email" : "c#x.com" }
]
}
I have 2 POJO classes Team (mapped to entire document),TeamMember (mapped to members inside a document).
Now I want to find to which team a specific user belongs to. For example if I search for a#x.com it should return me the document for Team 1. Similarly searching for b#x.com should return both of them as its in both the documents.
As I am very new to spring, not able to find out how to solve this.
Note: I am using MongoTemplate
somthing like this will do
final QueryBuilder queryBuilder = QueryBuilder.start();
//queryBuilder.and("members.email").is("a#x.com") This will work as well. try it out.
queryBuilder.and("members.email").in(Arrays.asList("a#x.com"))
final BasicDBObject projection = new BasicDBObject();
projection.put("fieldRequired", 1);
try (final DBCursor cursor = mongoTemplate.getCollection(collectionName).find(queryBuilder.get(), projection)
.batchSize(this.readBatchSize)) {
while (cursor.hasNext()) {
DBObject next = cursor.next();
........
// read the fields using next.get("field")
.........
}
}
batchsize and projection is not mandatory. Use projection if you don't want to fetch the whole document. You can specify which field in the document you want to fetch in the result.
You can use below code with the MongoTemplate
Query findQuery = new Query();
Criteria findCriteria = Criteria.where("members.email").is("b#x.com");
findQuery.addCriteria(findCriteria);
List<Team> teams = mongoTemplate.find(findQuery, Team.class);

Springdata mongodb aggregation match

After asking question to understand a bit more of the aggregation framework in MongoDB I finally found the way to do aggregation for my need (thanks to a StackExchange user)
So basically here is a document from my collection:
{
"_id" : ObjectId("s4dcsd5s4d6c54s6d"),
"items" : [
{
type : "TYPE_1",
text : "blablabla"
},
{
type : "TYPE_2",
text : "blablabla"
},
{
type : "TYPE_3",
text : "blablabla"
},
{
type : "TYPE_1",
text : "blablabla"
},
{
type : "TYPE_2",
text : "blablabla"
},
{
type : "TYPE_1",
text : "blablabla"
}
]
}
The idea was to be able to filter only some elements of my collections (avoiding Type 2 and 3). In fact I have more than 30 types and 6 are not allowed but for simplicity I made this example.
So the aggregation command in command line is this one:
db.history.aggregate([{
$match: {
_id: ObjectId("s4dcsd5s4d6c54s6d")
}
}, {
$unwind: '$items'
}, {
$match: {
'items.type': { '$nin': [ "TYPE_2" , "TYPE_3"] }
}
},
{ $limit: 10 }
]);
With this I am able to retrieve the 10 elements items of this document which do not match TYPE_2 and TYPE_3
However when I am using spring data there is no output. I looked a bit at the example to build mine but its still not working.
So I did:
Aggregation aggregation = newAggregation(
match(Criteria.where("id").is(myID)),
unwind("items"),
match(Criteria.where("items.type").nin(ignoreditemstype)),
limit(3),
skip(offsetLong)
);
AggregationResults<PersonnalHistory> results = mongAccess.getOperation().aggregate(query,
"items", PersonnalHistory.class);
PersonnalHistory is marked with annotation #Document(collection = "history") and id with the #id annotation
ignoreditemstype is a list containing TYPE_2 and TYPE_3
Here is what I have in the toString method of aggregation:
{
"aggregate" : "__collection__" ,
"pipeline" : [
{ "$match": { "id" : "s4dcsd5s4d6c54s6d"} },
{ "$unwind": "$items"},
{ "$match": { "items.type": { "$nin" : [ "TYPE_2" , "TYPE_3" ] } } },
{ "$limit" : 3},
{ "$skip" : 0 }
]
}
I tried a lot of stuff (to have at least an answer :) ) like removing id or the nin:
aggregation = newAggregation(
unwind("items"),
match(Criteria.where("items.type").nin(ignoreditemstype)),
limit(3),
skip(offsetLong)
);
aggregation = newAggregation(
match(Criteria.where("id").is(myid)),
unwind("items")
);
For information when I do a simple query like:
query.addCriteria(Criteria.where("id").is(myID));
My document is returned. However I have thousands of items. So I just want to have the 15 first (in fact the 15 first are the 15 last added)
Do you maybe see what I am doing wrong?
Yeah looks like you are passing simple String while it is expecting ObjectId
Aggregation aggregation = newAggregation(
match(Criteria.where("_id").is(new ObjectId(myID))),
unwind("items"),
match(Criteria.where("items.type").nin(ignoreditemstype)),
limit(3),
skip(offsetLong)
);
Now the question is why it works with simple query, my answer would be because spring-data driver is not that mature at least not with aggregation pipeline.

Resources