Here is how my document looks like:
{
"_id" : ObjectId("583cb6bcce047d1e68339b64"),
"variantDetails" : [
{
"variants" : {
"_" : "_"
},
"sku" : "069563-59690"
},
{
"variants" : {
"size" : "35"
},
"sku" : "069563-59690-35",
"barcode" : "809702246941"
},
{
"variants" : {
"size" : "36"
},
"sku" : "069563-59690-36",
"barcode" : "809702246958"
}
......
] }
And I would like to use a complex aggregation query like this:
db.getCollection('product').aggregate([
{ '$match': { 'variantDetails.sku': { '$in': ['069563-59690', '069563-59690-36', '069563-59690-37', '511534-01001'] } } },
{ '$project': {'_id': 1, 'variantDetails': 1, 'variantLength': { '$size': '$variantDetails' } } },
{ '$unwind': '$variantDetails' },
{ '$match': { 'variantDetails.sku': { '$in': ['069563-59690', '069563-59690-36', '069563-59690-37', '511534-01001'] } } },
{ '$match': { '$or': [
{'variantLength': { '$ne': 1 }, 'variantDetails.variants._': { '$ne': '_' } },
{'variantLength': 1 }
] } },
{ '$group': { '_id': '$_id', 'variantDetails': { '$push': '$variantDetails' } } },
{ '$project': {'_id': 1, 'variantDetails.sku': 1, 'variantDetails.barcode': 1} }
])
And here is my java code:
final Aggregation agg = Aggregation.newAggregation(
Aggregation.match(Criteria.where("variantDetails.sku").in(skus)),
Aggregation.project("_id", "variantDetails").and("variantDetails").project("size").as("variantLength"),
Aggregation.unwind("variantDetails"),
Aggregation.match(Criteria.where("variantDetails.sku").in(skus)),
Aggregation.match(new Criteria().orOperator(Criteria.where("variantLength").is(1), Criteria.where("variantLength").ne(1).and("variantDetails.variants._").is("_"))),
Aggregation.group("_id").push("variantDetails").as("variantDetails"),
Aggregation.project("_id", "variantDetails.sku", "variantDetails.barcode")
);
final AggregationResults<Product> result = this.mongo.aggregate(agg, this.mongo.getCollectionName(Product.class), Product.class);
return result.getMappedResults();
The problem is that spring translate
Aggregation.project("_id", "variantDetails.sku", "variantDetails.barcode")
To
{ "$project" : { "_id" : 1 , "sku" : "$variantDetails.sku" , "barcode" : "$variantDetails.barcode"}
But I'm expecting
{ '$project': {'_id': 1, 'variantDetails.sku': 1, 'variantDetails.barcode': 1} }
Could someone let me know how to make it right?
I had the same issue and this way works:
Aggregation.project("_id")
.andExpression("variantDetails.sku").as("variantDetails.sku")
.andExpression("variantDetails.barcode").as("variantDetails.barcode"));
The projection will be:
{'$project': {'_id': 1, 'variantDetails.sku': '$variantDetails.sku',
'variantDetails.barcode': '$variantDetails.barcode'} }
You just need to specify the label as alias in the projection operation as the default that spring provides doesnt match. Use Spring 1.8.5 version
Aggregation.project("_id")
.and(context -> new BasicDBObject("$arrayElemAt", Arrays.asList("variantDetails.sku", 0))).as("variantDetails.sku")
.and(context -> new BasicDBObject("$arrayElemAt", Arrays.asList("variantDetails.barcode", 0))).as("variantDetails.barcode"));
May be an old question, but I faced the same issue pointed by Sean.
If found that if you want the expected result
{ '$project': {'_id': 1, 'variantDetails.sku': 1, 'variantDetails.barcode': 1} }
a solution can be:
Aggregation.project("_id")
.andExpression("1").as("variantDetails.sku")
.andExpression("1").as("variantDetails.barcode")
Virginia León's answer was the starting point for finding this solution
Related
Excuse my newbie question but I can't figure it out.
This is my collection:
[
{
_id: "A",
uuid: "12345",
version: 1,
test: "data1"
},
{
_id: "B",
uuid: "56566",
version: 1,
test: "data2"
},
{
_id: "C",
uuid: "12345",
version: 2,
test: "data3"
}
]
I'm looking for a query with a UuidContains condition and with a exact condition.
findByUuidContains(5)
-> Result: [B,C] as Object Array
findByUuidContains(12345)
-> Result: [C] as Object Array
findByUuidContains(66)
-> Result: [B]
Is this kind of query possible?
In words:
Select all Object that uuid contains ${value} and from the resultset select only one per uuid with the highest Version.
EDIT1:
I changed the group projection from answer:
db.collection.aggregate([
{
"$redact": {
"$cond": [
{
"$gt": [
{
"$indexOfCP": [
{
"$toLower": "$uuid"
},
"5"
]
},
-1
]
},
"$$KEEP",
"$$PRUNE"
]
}
},
{
$sort: {
version: -1
}
},
{
$group: {
_id: {
uuid: "$uuid"
},
version: {
$first: "$version"
},
id: {
$first: "$_id"
},
test: {
$first: "$test"
}
}
},
{
"$project": {
num: "1",
id: 1,
_id: 0,
version: 1,
test: 1
}
},
{
"$group": {
"_id": "$num",
"result": {
"$addToSet": {
id: "$id",
version: "$version",
test: "$test"
}
}
}
},
{
"$project": {
_id: 0,
result: 1
}
}
])
and I added some test data attributes to my documents. Now I have to 'translate' it into the spring boot 'language'
EDIT2:
I'm currently trying to translate the second answer but I can't figure out how the GroupOpertaion in Spring works. Somebody familiar with it? The first and second operation works like the mongo query operations but it failed by the group operation
String uuidRegexExp = String.format(".*%s.*", uuidSegment);
Pattern uuidPattern = Pattern.compile(uuidRegexExp);
MatchOperation match = new MatchOperation(Criteria.where("uuid").regex(uuidPattern));
SortOperation sort = Aggregation.sort(Sort.Direction.DESC,"version");
GroupOperation grup = Aggregation.group("version").first("version").as("version");
Aggregation aggregate = Aggregation.newAggregation(
match, sort, grup
);
AggregationResults<Example> aggregate1 = mongoTemplate.aggregate(aggregate, Example.COLLECTION_NAME, Example.class);
aggregate1.getMappedResults().forEach(er -> log.info(er.toString()));
This is the example class:
#Data
#Document(Example.COLLECTION_NAME)
public class Example {
public static final String COLLECTION_NAME = "Example";
public static final String FIELD_UUID_NAME = "uuid";
public static final String FIELD_HOST_NAME = "host";
public static final String FIELD_URL_NAME = "url";
public static final String FIELD_VERSION_NAME = "version";
public static final String FIELD_ID_NAME = "_id";
#Field(FIELD_ID_NAME)
private ObjectId _id;
#Field(FIELD_UUID_NAME)
private String uuid;
#Field(FIELD_HOST_NAME)
private String host;
#Field(FIELD_URL_NAME)
private String url;
#Field(FIELD_VERSION_NAME)
private Long version;
}
EDIT3:
I think I have done it. Here is the Code in a not pretty version:
String uuidRegexExp = String.format(".*%s.*", uuidSegment);
Pattern uuidPattern = Pattern.compile(uuidRegexExp);
MatchOperation match = new MatchOperation(Criteria.where("uuid").regex(uuidPattern));
SortOperation sort = Aggregation.sort(Sort.Direction.DESC,"version");
GroupOperation grup = Aggregation.group("uuid").first("version").as("version").first("_id").as("id");
ProjectionOperation project = Aggregation.project().and("_id").as("uuid").and("version").as("version").and("id").as("_id");
Aggregation aggregate = Aggregation.newAggregation(
match, sort, grup,project
);
AggregationResults<Example> aggregate1 = mongoTemplate.aggregate(aggregate, SingleRawArticle.COLLECTION_NAME, Example.class);
Is this something you are looking for? I have created mongo playground for it. You can check the query by passing diffrent parameters. I have used 5 in example like below. But i have also tried with 12345 and 66 and it looks fine to me.
{
"$indexOfCP": [
{
"$toLower": "$uuid"
},
"5"
]
},
Mongo Playground
Here is the query :
db.collection.aggregate([
{
"$redact": {
"$cond": [
{
"$gt": [
{
"$indexOfCP": [
{
"$toLower": "$uuid"
},
"5"
]
},
-1
]
},
"$$KEEP",
"$$PRUNE"
]
}
},
{
$sort: {
version: -1
}
},
{
$group: {
_id: {
uuid: "$uuid"
},
version: {
$first: "$version"
},
id: {
$first: "$_id"
}
}
},
{
"$project": {
num: "1",
id: 1,
_id: 0
}
},
{
"$group": {
"_id": "$num",
"result": {
"$addToSet": "$id"
}
}
},
{
"$project": {
_id: 0,
result: 1
}
}
])
check the below query to get the documents matching the given string. I have used regex to match the input string.
db.collection.aggregate(
[
{
"$match" : {
"uuid" : {
"$regex" : ".*5.*"
}
}
},
{
"$sort" : {
"version" : -1.0
}
},
{
"$group" : {
"_id" : {
"uuid" : "$uuid"
},
"uuid" : {
"$first" : "$uuid"
},
"id" : {
"$first" : "$_id"
},
"version" : {
"$first" : "$version"
}
}
},
{
"$project" : {
"_id" : "$id",
"uuid" : 1.0,
"version" : 1.0
}
}
],
{
"allowDiskUse" : false
}
);
Output:
{
"uuid" : "12345",
"version" : 2.0,
"_id" : "C"
}
{
"uuid" : "56566",
"version" : 1.0,
"_id" : "B"
}
Java code equivalent to query. Modified your edit according to the latest changes. Changed variable names to be more specific.
String uuidRegexExp = String.format(".*%s.*", uuidSegment);
MatchOperation match = new MatchOperation(Criteria.where("uuid").regex(Pattern.compile(uuidRegexExp)));
SortOperation sort = Aggregation.sort(Sort.Direction.DESC,"version");
GroupOperation group = Aggregation.group("uuid").first("version").as("version").first("_id").as("id").first("uuid").as("uuid");
ProjectionOperation project = Aggregation.project().and("uuid").as("uuid").and("version").as("version").and("id").as("_id");
Aggregation aggregation = Aggregation.newAggregation(
match, sort, group,project
);
AggregationResults<Example> aggregate = mongoTemplate.aggregate(aggregation, SingleRawArticle.COLLECTION_NAME, Example.class);
I have mongo collection with survey answers submitted by each user. I would like to get the count of users selected as an option. Only one user has selected the option O12. The output should be 1.
{
"_id" : ObjectId("5ea179eb39ff117948f19266"),
"_class" : "model.survey.Answer",
"survey_id" : "5ea178c239ff117948f19265",
"survey_user" : [
{
"user_id" : 1072,
"user_option" : [
{
"question_id" : "Q1",
"option_id" : "O11"
},
{
"question_id" : "Q2",
"option_id" : "O21"
},
{
"question_id" : "Q3",
"option_id" : "O31"
},
{
"question_id" : "Q4",
"option_id" : "O41"
}
]
},
{
"user_id" : 1073,
"user_option" : [
{
"question_id" : "Q1",
"option_id" : "O12"
},
{
"question_id" : "Q2",
"option_id" : "O21"
},
{
"question_id" : "Q3",
"option_id" : "O31"
},
{
"question_id" : "Q4",
"option_id" : "O41"
}
]
}
]
}
You can do that using MongoDB's aggregation-pipeline :
Different ways to do it, One way is to use $unwind:
Type 1 - Query 1 :
db.collection.aggregate([
/** Optional but will be good on huge collections to lessen data for further stages */
{
$match: { "survey_user.user_option.option_id": "O12" }
},
{
$unwind: "$survey_user"
},
/** When you unwind a each object/element in array gets it's own document after `unwind` stage */
{
$match: { "survey_user.user_option.option_id": "O12" }
},
/** After match you'll only have objects which met the criteria in `survey_user` array */
/** group on `_id` & push entire original doc to data field */
{
$group: { _id: "$_id", survey_user: { $push: "$survey_user" }, data: { $first: "$$ROOT" } }
},
/** Add `survey_user` array to `data.survey_user` & it's size to `data.optedCount` field */
{
$addFields: { "data.survey_user": "$survey_user", "data.optedCount": { $size: "$survey_user" } }
},
/** Make `data` as new root to doc */
{
$replaceRoot: { newRoot: "$data" }
}
])
Test : mongoplayground
Just in case if you just need count but not needed the entire doc to be returned there will be a minor change in above query :
Type 1 - Query 2 :
db.collection.aggregate([
{
$match: { "survey_user.user_option.option_id": "O12" }
},
{
$unwind: "$survey_user"
},
{
$match: { "survey_user.user_option.option_id": "O12" }
},
/** Just group on `_id` & count no.of docs, maintain `survey_id` */
{
$group: { _id: "$_id", optedCount: { $sum: 1 }, survey_id: { $first: "$survey_id" } }
}
])
Test : mongoplayground
Using array iterator $reduce, which might be helpful if your collections data is so huge, as unwind will explode your docs.
Type 2 - Query :
db.collection.aggregate([
{
$match: {
"survey_user.user_option.option_id": "O12",
},
},
/** Instead of `$addFields`, you can use `$project` to project fewer needed fields (which can be help improve query with performance benefits ) */
{
$addFields: {
optedCount: {
$reduce: {
input: "$survey_user",
initialValue: 0,
in: {
$cond: [
{ $in: ["O12", "$$this.user_option.option_id"] },
{ $add: ["$$value", 1] },
"$$value",
]
}
}
}
}
}
]);
Test : mongoplayground
prolog
MySQL table name: ar_tmp has two columns id int and name int
to do
I execute sql
insert into ar_tmp (id, name) values (1, 4);
and CaptureChangeMySQL captures this CDC and flow-content like this
{
"type":"insert",
"timestamp":1550221517000,
"binlog_filename":"mysql-bin.013920",
"binlog_position":241518646,
"database":"platform_data",
"table_name":"ar_tmp",
"table_id":2899035,
"columns":[
{
"id":1,
"name":"id",
"column_type":4,
"value":1
},
{
"id":2,
"name":"name",
"column_type":4,
"value":4
},
{
"id":3,
"value":4
}
]
}
But I want the result in this format
{
"type":"insert",
"timestamp":1550221517000,
"binlog_filename":"mysql-bin.013920",
"binlog_position":241518646,
"database":"platform_data",
"table_name":"ar_tmp",
"table_id":2899035,
"columns":[
{
"id":1,
"name":4
}
]
}
or
{
"id":1,
"name":4
}
solution
It can be done by hard coding using jsonPath function
But maybe to 'hard' to do this because each column using the same code which making the processor redundance(say 50 columns). Worsely, it's danger when the column name changed.
Any ideas?
JoltTransformJSON can help you here.
Try it out here
Jolt Spec for your demo input json:
[
{
"operation": "shift",
"spec": {
"columns": {
"*": {
"value": "columns.#(1,name)"
}
},
"*": "&"
}
}
]
The result is:
{
"type" : "insert",
"timestamp" : 1550221517000,
"binlog_filename" : "mysql-bin.013920",
"binlog_position" : 241518646,
"database" : "platform_data",
"table_name" : "ar_tmp",
"table_id" : 2899035,
"columns" : {
"id" : 1,
"name" : 4
}
}
Need help to build dynamic MongoDB query.
everything inside the "$or" Array is dynamic.
db.group.find({
"version" : NumberLong(0),
"$or" : [{
"$and" : [
{
"object_type" : "D"
},
{
"type" : "R"
},
{
"name" : "1"
}
]
},{
"$and" : [
{
"object_type" : "D"
},
{
"type" : "E"
},
{
"name" : "2"
}
]
]
});
Did the below spring data query but doesn't work
Criteria criteria = Criteria.where("version").is("123");
List<Criteria> docCriterias = new ArrayList<Criteria>();
groups.stream().forEach(grp -> {
docCriterias.add(Criteria.where("type").is(grp.get("type").toString())
.andOperator(Criteria.where("object_type").is(grp.get("objectType").toString()))
.andOperator(Criteria.where("name").is(grp.get("name").toString())));
});
criteria.orOperator((Criteria[]) docCriterias.toArray());
Query q = new Query(criteria);
Thanks for the help
You should pay attention to how you combine the operators.
The ff code should work for you (note this is groovy remember to change the closure into to java lambda expression):
List<Criteria> docCriterias = new ArrayList<Criteria>();
List groups = [
[
type: "type1",
object_type: "object_type1",
name: "name1"
],
[
type: "type2",
object_type: "object_type2",
name: "name2"
],
[
type: "type3",
object_type: "object_type3",
name: "name3"
],
]
groups.stream().each {grp ->
docCriterias.add(new Criteria().andOperator(
Criteria.where("type").is(grp.get("type")),
Criteria.where("object_type").is(grp.get("object_type")),
Criteria.where("name").is(grp.get("name"))
))
};
Criteria criteria = new Criteria().andOperator(
Criteria.where("version").is("123"),
new Criteria().orOperator(docCriterias.toArray(new Criteria[docCriterias.size()]))
);
Query q = new Query(criteria);
Which will give you this query:
{
"$and":[
{
"version":"123"
},
{
"$or":[
{
"$and":[
{
"type":"type1"
},
{
"object_type":"object_type1"
},
{
"name":"name1"
}
]
},
{
"$and":[
{
"type":"type2"
},
{
"object_type":"object_type2"
},
{
"name":"name2"
}
]
},
{
"$and":[
{
"type":"type3"
},
{
"object_type":"object_type3"
},
{
"name":"name3"
}
]
}
]
}
]
},
Fields:{
},
Sort:{
}
You could reach this using MongoDB Aggregation Pipeline in Json and Apache Velocity to customize more the Query, then execute this using db.runCommand using Spring MongoTemplate.
Example:
monodb_client_dynamic_query.vm
{
"aggregate": "client",
"pipeline": [
{
"$match" : {
"$and" : [
{
"is_removed" : {
"$ne" : [
true
]
}
},
{
"errors" : {
"$size" : 0.0
}
},
{
"client_id": "$velocityMap.client_id"
}
]
}
},
{
"$project" : {
"_id" : -1.0,
"account" : "$_id.account",
"person_id" : "$_id.person_id",
"begin_date": { $dateToString: { format: "%Y-%m-%d", date: "$value.begin_date" } },
"end_date": { $dateToString: { format: "%Y-%m-%d", date: "$value.end_date" } }
}
}
]
}
Then execute using MondoTemplate:
String script = ...load from file the script monodb_client_dynamic_query.vm
Map parameters = ... put all variables to replace in the mongodb script
String scriptNoSql = VelocityUtil.loadTemplateVM(script, parameters);
DBObject dbObject = (BasicDBObject) JSON.parse(scriptNoSql);
if (null == dbObject) {
return;
}
DB db = mongoTemplate.getDb();
CommandResult result = db.command(dbObject);
if(!result.ok()) {
throw result.getException();
}
db.flm_conversation.aggregate(
[
{$match: {"conversationRecords.isPrimary":true,"conversationRecords.commentTime":{'$gte': new Date('2016-12-18 00:00:00'), '$lte': new Date('2016-12-18 23:59:59')} } },
{$unwind:"$conversationRecords"},
{ $group: {
_id: {
"commentLevel": "$commentLevel",
"time":{"$add": [
{
"$subtract": [
{ "$subtract": [
"$conversationRecords.commentTime",
new Date(0)
]}
,
{ "$mod": [
{ "$subtract": [
"$conversationRecords.commentTime",
new Date(0)
]},
1000 * 60 * 30
]}
]
},
new Date(0)
]}
},
count: { "$sum": 1 }
}},
{ $group: {_id: "$_id.commentLevel",count: { "$sum": 1 },pointrecord:{$push: {time:"$_id.time",count:"$count"} } }},
{ $project: { _id: 1,count:1,pointrecord:1 } }
])
How to convert this query using Spring Mongodb Aggregation apis?
AggregationOperation match = Aggregation.match(Criteria.where("conversationRecords.isPrimary").is(true)
.and("conversationRecords.commentTime").gte(DateUtils.stringToDate("2016-12-18 00:00:00","yyyy-MM-dd HH:mm:ss")).lte(DateUtils.stringToDate("2016-12-18 23:59:59","yyyy-MM-dd HH:mm:ss")));
AggregationOperation unwind = Aggregation.unwind("conversationRecords");
AggregationOperation group = Aggregation.group("commentLevel");
Aggregation agg = newAggregation(match,unwind,group);
AggregationResults<SummaryRecordAggre> groupResults
= mongoTemplate.aggregate(agg, COLLECTION_NAME, SummaryRecordAggre.class);
I don't know group "$add" how to convert?I find by
http://docs.spring.io/spring-data/mongodb/docs/current/reference/html/#mongo.aggregation
You can try something like this. I removed the extra operators which I felt was not needed but we can easily put them back should you need.
The first group which is something you need to build using MongoDB objects as currently all aggregation operators are not supported out of box in spring mongo.
Similar way for push operator in second group as you cant push multiple values.
AggregationOperation match = Aggregation.match(Criteria.where("conversationRecords.isPrimary").is(true)
.and("conversationRecords.commentTime").gte(DateUtils.stringToDate("2016-12-18 00:00:00", "yyyy-MM-dd HH:mm:ss")).lte(DateUtils.stringToDate("2016-12-18 23:59:59", "yyyy-MM-dd HH:mm:ss")));
AggregationOperation unwind = Aggregation.unwind("conversationRecords");
AggregationOperation firstGroup = context -> context.getMappedObject(new BasicDBObject(
"$group", new BasicDBObject(
"_id", new BasicDBObject("commentLevel", "$commentLevel")
.append(
"time", new BasicDBObject(
"$subtract", new Object[]{"$conversationRecords.commentTime", new BasicDBObject(
"$mod", new Object[]{"$conversationRecords.commentTime", 1000 * 60 * 30})}))).append("count", new BasicDBObject("$sum", 1))));
AggregationOperation secondGroup = Aggregation.group("_id.commentLevel").count().as("count").push(new BasicDBObject("time", "$_id.time").append("count", "$count")).as("pointrecord");
AggregationOperation project = Aggregation.project("_id", "count", "pointrecord");
Aggregation agg = newAggregation(match, unwind, firstGroup, secondGroup, project);
This is equivalent of below query
[{
"$match": {
"conversationRecords.isPrimary": true,
"conversationRecords.commentTime": {
"$gte": {
"$date": "2016-12-27T14:46:50.896Z"
},
"$lte": {
"$date": "2016-12-27T14:46:50.896Z"
}
}
}
}, {
"$unwind": "$conversationRecords"
}, {
"$group": {
"_id": {
"commentLevel": "$commentLevel",
"time": {
"$subtract": ["$conversationRecords.commentTime", {
"$mod": ["$conversationRecords.commentTime", 1800000]
}]
}
},
"count": {
"$sum": 1
}
}
}, {
"$group": {
"_id": "$_id.commentLevel",
"count": {
"$sum": 1
},
"pointrecord": {
"$push": {
"time": "$_id.time",
"count": "$count"
}
}
}
}, {
"$project": {
"_id": 1,
"count": 1,
"pointrecord": 1
}
}]
}