I have a finder defined as a Spring Data repository derived from MongoRepository which searches for 3 different attributes in MongoDB. All three have a single index.
public Page<Content> findByIdInOrAuthorUserNameInOrTagsIdIn(
#Param("ids") Collection ids,
#Param("userNames") Collection userName,
#Param("tagIds") Collection tagIds,
#Param("pageable") Pageable pageable);
The problem is that one attributes has a result set of 2,5 mio entries:
"page": {
"size": 20,
"totalElements": 2531397,
"totalPages": 126570,
"number": 5
}
So the query for a page is quite fast (13ms) as seen in the mongo log file:
2017-04-10T12:50:27.562+0200 I COMMAND [conn68] command content.content command: find { find: "content", filter: { $or: [ { $or: [ { _id: { $in: [ "..." ] } }, { author.userName: { $in: [ "...", "..." ] } } ] }, { tags._id: { $in: [ "..." ] } } ] }, skip: 100, limit: 20 } planSummary: IXSCAN { _id: 1 }, IXSCAN { tags._id: 1 }, IXSCAN { author.userName: 1 } keysExamined:120 docsExamined:120 cursorExhausted:1 numYields:0 nreturned:20 reslen:21185 locks:{ Global: { acquireCount: { r: 2 } }, Database: { acquireCount: { r: 1 } }, Collection: { acquireCount: { r: 1 } } } protocol:op_query 13ms
But it seems that the page summary which counts the result takes ~117s:
2017-04-10T12:52:24.172+0200 I COMMAND [conn68] command content.content command: count { count: "content", query: { $or: [ { $or: [ { _id: { $in: [ "..." ] } }, { author.userName: { $in: [ "...", "..." ] } } ] }, { tags._id: { $in: [ "..." ] } } ] } } planSummary: IXSCAN { _id: 1 }, IXSCAN { tags._id: 1 }, IXSCAN { author.userName: 1 } keysExamined:2531466 docsExamined:2531397 numYields:21190 reslen:44 locks:{ Global: { acquireCount: { r: 42382 } }, Database: { acquireCount: { r: 21191 } }, Collection: { acquireCount: { r: 21191 } } } protocol:op_query 116592ms
Is there a way to switch off the page summary or speed up the counting some how?
Use Slice instead of Page. It is very similar to Page, but doesn't need the total count of elements.
Related
Running the following query, I get two results:
query x($keyword: String = "Pulp", $limit: Int, $nexttoken: String) {
listProducts(filter: {or: [{name: {contains: $keyword}}, {description: {contains: $keyword}}]}, limit: $limit, nextToken: $nexttoken) {
items {
id
}
nextToken
}
}
Response:
{
"data": {
"listProducts": {
"items": [
{
"id": "ba8048f3-1099-4184-8e78-f0e57b521465"
},
{
"id": "335770ad-f566-4385-a861-7d0dab6c9dd8"
}
],
"nextToken": null
}
}
}
By setting $limit: Int=1, I get an empty array of items:
{
"data": {
"listProducts": {
"items": [],
"nextToken": "xxxxxx"
}
}
}
By setting $limit: Int=2, I get one item:
{
"data": {
"listProducts": {
"items": [
{
"id": "ba8048f3-1099-4184-8e78-f0e57b521465"
}
],
"nextToken": "xxxxxx"
}
}
}
In order to get all 2 two items, I need to set $limit=4. Why?
I have this query (in Hasura in case that matters):
query MyQuery {
records(distinct_on:[recordId], where: { modelId: {_eq: "2f1f70b8-cb7b-487c-9e4c-ca03624ce926"}}) {
recordId
inboundEdges(where: {fromModelId: {_eq: "f0e19461-6d38-4148-8041-54eba6451293"}}) {
fromRecord {
property_path_values(where:{stringValue:{_eq:"2021-08-26"}}) {
stringValue
}
}
}
}
}
I get this result back:
{
"data": {
"records": [
{
"recordId": "2fbe37b1-78db-4b22-b713-2388cfb52597",
"inboundEdges": [
{
"fromRecord": {
"property_path_values": [
{
"stringValue": "2021-08-26"
}
]
}
},
{
"fromRecord": {
"property_path_values": [
{
"stringValue": "2021-08-26"
},
{
"stringValue": "2021-08-26"
}
]
}
}
]
},
{
"recordId": "7b34e85d-f4e1-4099-89d9-02483128a6cd",
"inboundEdges": [
{
"fromRecord": {
"property_path_values": [
{
"stringValue": "2021-08-26"
}
]
}
}
]
},
{
"recordId": "840f52e2-0f2e-4591-810d-19f9e8840a49",
"inboundEdges": []
}
]
}
}
I do not want the third result in the response, because it's inboundEdges array is empty.
What I am trying to say is: find me all records that have at least one inboundEdge with a fromRecord that has at least one property_path_value with a stringValue equal to 2021-08-26. I do not want to have to parse the response needing to exclude results with inboundEdges === []
Seems I was confusing the selection set with the place to state the query. The right way to do what I wanted is:
query MyQuery {
records(where: {inboundEdges: {fromModelId: {_eq: "f0e19461-6d38-4148-8041-54eba6451293"}, fromRecord: {propertyPathValues: {stringValue: {_eq: "2021-08-26"}}}}, modelId: {_eq: "2f1f70b8-cb7b-487c-9e4c-ca03624ce926"}}) {
recordId
}
}
i.e. put the query in the where clause, like a normal person, not the selection set
This is the Json variable this filter is working fine in kibana dashbaord here is the screenshot for that
My Requirement is that want to get the same output as am getting from Kibana using filter query from Elastic Search Dot net Search Request here is the link which i followed from stack over flow to the point where get till now
{
""version"": true,
""size"": 500,
""sort"": [
{
""AddedOn"": {
""order"": ""desc"",
""unmapped_type"": ""boolean""
}
}
],
""stored_fields"": [
""*""
],
""script_fields"": { },
""docvalue_fields"": [
{
""field"": ""#timestamp"",
""format"": ""date_time""
},
{
""field"": ""AddedOn"",
""format"": ""date_time""
}
],
""_source"": {
""excludes"": []
},
""query"": {
""bool"": {
""must"": [],
""filter"": [
{
""match_all"": { }
},
{
""match_all"": { }
},
{
""bool"": {
""filter"": [
{
""match_all"": { }
},
{
""match_all"": { }
},
{
""range"": {
""AddedOn"": {
""format"": ""strict_date_optional_time"",
""gte"": ""2019-10-26T09:20:14.087Z"",
""lte"": ""2020-10-26T09:20:14.087Z""
}
}
}
],
""must"": [],
""must_not"": [],
""should"": []
}
},
{
""range"": {
""AddedOn"": {
""gte"": ""2019-10-26T10:38:34.169Z"",
""lte"": ""2020-10-26T10:38:34.170Z"",
""format"": ""strict_date_optional_time""
}
}
}
],
""should"": [],
""must_not"": []
}
},
""highlight"": {
""pre_tags"": [
""#kibana-highlighted-field#""
],
""post_tags"": [
""#/kibana-highlighted-field#""
],
""fields"": {
""*"": { }
},
""fragment_size"": 2147483647
}
}
"
This is the code which is doing searching in elasticsearch index
_elasticClient.LowLevel.Search<SearchResponse<object>>(FraudIndex, "type", json4); >>> The errror i am getting is that cannot convert from string to ElasticClient.searchRequestparameters
This was the solution found
var SearchResponse =await_elasticClient.LowLevel.SearchAsync<SearchResponse<object>>(FraudIndex, json4);
I have the following data:
{
"_id": ObjectID("5e2fa881c3a1a70006c5743c"),
"name": "Some name",
"policies": [
{
"cId": "dasefa-2738-4cf0-90e0d568",
"weight": 12
},
{
"cId": "c640ad67dasd0-92f981583568",
"weight": 50
}
]
}
I'm able to query this with Spring Mongo fine, however I want to be able to order the policies by weight
At the moment I get my results fine with:
return mongoTemplate.find(query, CArea::class.java)
However say I make the following aggregations:
val unwind = Aggregation.unwind("policies")
val sort = Aggregation.sort(Sort.Direction.DESC,"policies.weight")
How can I go and actually apply those to the returned results above? I was hoping that the dot annotation would do the job in my query however didnt do anything e.g. Query().with(Sort.by(options.sortDirection, "policies.weight"))
Any help appreciated.
Thanks.
I am not familier with Spring Mongo, but I guess you can convert the following aggregation to spring code.
db.collection.aggregate([
{
$unwind: "$policies"
},
{
$sort: {
"policies.weight": -1
}
},
{
$group: {
_id: "$_id",
"policies": {
"$push": "$policies"
},
parentFields: {
$first: "$$ROOT"
}
}
},
{
$replaceRoot: {
newRoot: {
$mergeObjects: [
"$parentFields",
{
policies: "$policies"
}
]
}
}
}
])
This will result:
[
{
"_id": "5e2fa881c3a1a70006c5743c",
"name": "Some name",
"policies": [
{
"cId": "c640ad67dasd0-92f981583568",
"weight": 50
},
{
"cId": "dasefa-2738-4cf0-90e0d568",
"weight": 12
}
]
}
]
Playground
Need help to build dynamic MongoDB query.
everything inside the "$or" Array is dynamic.
db.group.find({
"version" : NumberLong(0),
"$or" : [{
"$and" : [
{
"object_type" : "D"
},
{
"type" : "R"
},
{
"name" : "1"
}
]
},{
"$and" : [
{
"object_type" : "D"
},
{
"type" : "E"
},
{
"name" : "2"
}
]
]
});
Did the below spring data query but doesn't work
Criteria criteria = Criteria.where("version").is("123");
List<Criteria> docCriterias = new ArrayList<Criteria>();
groups.stream().forEach(grp -> {
docCriterias.add(Criteria.where("type").is(grp.get("type").toString())
.andOperator(Criteria.where("object_type").is(grp.get("objectType").toString()))
.andOperator(Criteria.where("name").is(grp.get("name").toString())));
});
criteria.orOperator((Criteria[]) docCriterias.toArray());
Query q = new Query(criteria);
Thanks for the help
You should pay attention to how you combine the operators.
The ff code should work for you (note this is groovy remember to change the closure into to java lambda expression):
List<Criteria> docCriterias = new ArrayList<Criteria>();
List groups = [
[
type: "type1",
object_type: "object_type1",
name: "name1"
],
[
type: "type2",
object_type: "object_type2",
name: "name2"
],
[
type: "type3",
object_type: "object_type3",
name: "name3"
],
]
groups.stream().each {grp ->
docCriterias.add(new Criteria().andOperator(
Criteria.where("type").is(grp.get("type")),
Criteria.where("object_type").is(grp.get("object_type")),
Criteria.where("name").is(grp.get("name"))
))
};
Criteria criteria = new Criteria().andOperator(
Criteria.where("version").is("123"),
new Criteria().orOperator(docCriterias.toArray(new Criteria[docCriterias.size()]))
);
Query q = new Query(criteria);
Which will give you this query:
{
"$and":[
{
"version":"123"
},
{
"$or":[
{
"$and":[
{
"type":"type1"
},
{
"object_type":"object_type1"
},
{
"name":"name1"
}
]
},
{
"$and":[
{
"type":"type2"
},
{
"object_type":"object_type2"
},
{
"name":"name2"
}
]
},
{
"$and":[
{
"type":"type3"
},
{
"object_type":"object_type3"
},
{
"name":"name3"
}
]
}
]
}
]
},
Fields:{
},
Sort:{
}
You could reach this using MongoDB Aggregation Pipeline in Json and Apache Velocity to customize more the Query, then execute this using db.runCommand using Spring MongoTemplate.
Example:
monodb_client_dynamic_query.vm
{
"aggregate": "client",
"pipeline": [
{
"$match" : {
"$and" : [
{
"is_removed" : {
"$ne" : [
true
]
}
},
{
"errors" : {
"$size" : 0.0
}
},
{
"client_id": "$velocityMap.client_id"
}
]
}
},
{
"$project" : {
"_id" : -1.0,
"account" : "$_id.account",
"person_id" : "$_id.person_id",
"begin_date": { $dateToString: { format: "%Y-%m-%d", date: "$value.begin_date" } },
"end_date": { $dateToString: { format: "%Y-%m-%d", date: "$value.end_date" } }
}
}
]
}
Then execute using MondoTemplate:
String script = ...load from file the script monodb_client_dynamic_query.vm
Map parameters = ... put all variables to replace in the mongodb script
String scriptNoSql = VelocityUtil.loadTemplateVM(script, parameters);
DBObject dbObject = (BasicDBObject) JSON.parse(scriptNoSql);
if (null == dbObject) {
return;
}
DB db = mongoTemplate.getDb();
CommandResult result = db.command(dbObject);
if(!result.ok()) {
throw result.getException();
}