So I am trying to understand how the match_phrase query works under certain circumstances
with elastic search [We have version 6.8 set up as of now ] . When I give it a string with multiple tokens it shows while profiling its running a phrase query but when I run it with a single token while profiling it shows its running a termsquery internally . I am trying to understand shouldn't it be independent of the input and if the positioning of terms is not correct fail to return a match ? Attaching queries and o/p -
Query with multiple tokens -
GET potato_testv3/_search
{"profile": "true",
"query": {
"bool": {
"must": [
{ "match_phrase": { "skill_set": {"query":"potato farmer"} }}
]
}
}
}
Output of the above -
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 0.5753642,
"hits" : [
{
"_index" : "potato_testv3",
"_type" : "recruiterinsightsv11",
"_id" : "4RShdnkBc8OOeUFVkncD",
"_score" : 0.5753642,
"_source" : {
"skill_set" : [
"silly webdriver",
"uft",
"uft/qtp",
"potato farmer"
]
}
}
]
},
"profile" : {
"shards" : [
{
"id" : "[5QVxJbTCSU-ruYT9EHsujA][potato_testv3][0]",
"searches" : [
{
"query" : [
{
"type" : "PhraseQuery",
"description" : """skill_set:"potato farmer"""",
"time_in_nanos" : 338986,
"breakdown" : {
"score" : 15362,
"build_scorer_count" : 2,
"match_count" : 1,
"create_weight" : 55661,
"next_doc" : 74248,
"match" : 39624,
"create_weight_count" : 1,
"next_doc_count" : 2,
"score_count" : 1,
"build_scorer" : 154084,
"advance" : 0,
"advance_count" : 0
}
}
],
"rewrite_time" : 3932,
"collector" : [
{
"name" : "CancellableCollector",
"reason" : "search_cancelled",
"time_in_nanos" : 48431,
"children" : [
{
"name" : "SimpleTopScoreDocCollector",
"reason" : "search_top_hits",
"time_in_nanos" : 19840
}
]
}
]
}
],
"aggregations" : [ ]
}
]
}
}
Query with single token -
GET potato_testv3/_search
{"profile": "true",
"query": {
"bool": {
"must": [
{ "match_phrase": { "skill_set": {"query":"potato"} }}
]
}
}
}
Output of above -
{
"took" : 2,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 0.2876821,
"hits" : [
{
"_index" : "potato_testv3",
"_type" : "recruiterinsightsv11",
"_id" : "4RShdnkBc8OOeUFVkncD",
"_score" : 0.2876821,
"_source" : {
"skill_set" : [
"silly webdriver",
"uft",
"uft/qtp",
"potato farmer"
]
}
}
]
},
"profile" : {
"shards" : [
{
"id" : "[TeKxvYLJQfG_GVtD3bmpiw][potato_testv3][0]",
"searches" : [
{
"query" : [
{
"type" : "TermQuery",
"description" : "skill_set:potato",
"time_in_nanos" : 52214,
"breakdown" : {
"score" : 11310,
"build_scorer_count" : 2,
"match_count" : 0,
"create_weight" : 30974,
"next_doc" : 1314,
"match" : 0,
"create_weight_count" : 1,
"next_doc_count" : 2,
"score_count" : 1,
"build_scorer" : 8610,
"advance" : 0,
"advance_count" : 0
}
}
],
"rewrite_time" : 3761,
"collector" : [
{
"name" : "CancellableCollector",
"reason" : "search_cancelled",
"time_in_nanos" : 20912,
"children" : [
{
"name" : "SimpleTopScoreDocCollector",
"reason" : "search_top_hits",
"time_in_nanos" : 15758
}
]
}
]
}
],
"aggregations" : [ ]
}
]
}
}
In case if it helps , schema of the index used -
{
"potato_testv3" : {
"mappings" : {
"recruiterinsightsv11" : {
"dynamic" : "false",
"properties" : {
"skill_set" : {
"type" : "text",
"norms" : false,
"fielddata" : true
}
}
}
}
}
}
You are executing the same match_phrase query, once with a search string made up of multiple terms, once with a search string of a single token.
When executing an Elasticsearch query, Elasticsearch will optimise the query and translate it to the relevant queries on Lucene level. A phrase-query is more expensive to execute as
all terms of the search string need to match, and on top of that
the positions of the terms in a matching document need to be in the very same order as in the search string
If your search string only consist of a single term Elasticsearch can skip all of that extra effort and simply query for documents matching that single search term. What you observe therefore, is making perfect sense. It shows you how Elasticsearch is optimising the query while executing it.
Related
I'm trying to run a query elastic search. When run this query
GET accounts/_search/
{
"query": {
"term": {
"address_line_1": "1000"
}
}
}
I get back multiple records like
"hits" : [
{
"_index" : "accounts",
"_type" : "_doc",
"_id" : "...",
"_score" : 8.355149,
"_source" : {
"state_id" : 35,
"first_name" : "...",
"last_name" : "...",
"middle_name" : "P",
"dob" : "...",
"status" : "ACTIVE",
"address_line_1" : "1000 BROADROCK CT",
"address_line_2" : "",
"address_city" : "PARMA",
"address_zip" : "",
"address_zip_plus_4" : ""
}
},
But when I try to expand it to include the more like below I don't get any matches
GET accounts/_search/
{
"query": {
"term": {
"address_line_1": "1000 B"
}
}
}
The response is
{
"took" : 0,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 0,
"relation" : "eq"
},
"max_score" : null,
"hits" : [ ]
}
}
The term query is looking for exact matches. Your address_line_* fields were most probably indexed with the standard analyzer which lowercase-s all the letters which in turn prevents the query from matching.
So either use
GET accounts/_search/
{
"query": {
"match": { <--
"address_line_1": "1000 B"
}
}
}
which does not really 'care' about B being lower/upper case or adjust your field analyzers such that the capitalization is preserved.
We're having indices that are partitioned by year, e.g.:
items-2019
items-2020
Consider the following data:
POST items-2019/_doc
{
"#timestamp": "2019-01-01"
}
POST items-2020/_doc
{
"#timestamp": "2020-01-01"
}
POST /_aliases
{
"actions": [
{
"add": {
"index": "items-*",
"alias": "items"
}
}
]
}
Now when I query data and explicitly sort results, it would skip items-2020 shards:
GET items/_search
{
"query": {
"range": {
"#timestamp": {
"lt": "2020-01-01"
}
}
},
"sort": {
"#timestamp": "desc"
}
}
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 2,
"successful" : 2,
"skipped" : 1, <--- skipped
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : null,
"hits" : [
{
"_index" : "items-2019",
"_type" : "_doc",
"_id" : "BTdSb3UBRFH0Yqe1vm_W",
"_score" : null,
"_source" : {
"#timestamp" : "2019-01-01"
},
"sort" : [
1546300800000
]
}
]
}
}
However when I don't sort results explicitly, it wouldn't skip the shards, however ES would issue a MatchNoDocsQuery:
GET items/_search
{
"profile": "true",
"query": {
"range": {
"#timestamp": {
"lt": "2020-01-01"
}
}
}
}
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 2,
"successful" : 2,
"skipped" : 0, <--- nothing skipped
"failed" : 0
},
"hits" : {
"total" : {
"value" : 1,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "items-2019",
"_type" : "_doc",
"_id" : "BTdSb3UBRFH0Yqe1vm_W",
"_score" : 1.0,
"_source" : {
"#timestamp" : "2019-01-01"
}
}
]
},
"profile" : {
"shards" : [
{
"id" : "[Axyv60mYQEGAREa2TwbgMQ][items-2019][0]",
"searches" : [
{
"query" : [
{
"type" : "ConstantScoreQuery",
"description" : "ConstantScore(DocValuesFieldExistsQuery [field=#timestamp])",
"time_in_nanos" : 69525,
"breakdown" : {
"set_min_competitive_score_count" : 0,
"match_count" : 0,
"shallow_advance_count" : 0,
"set_min_competitive_score" : 0,
"next_doc" : 3766,
"match" : 0,
"next_doc_count" : 1,
"score_count" : 1,
"compute_max_score_count" : 0,
"compute_max_score" : 0,
"advance" : 4123,
"advance_count" : 1,
"score" : 1123,
"build_scorer_count" : 2,
"create_weight" : 29745,
"shallow_advance" : 0,
"create_weight_count" : 1,
"build_scorer" : 30768
},
"children" : [
{
"type" : "DocValuesFieldExistsQuery",
"description" : "DocValuesFieldExistsQuery [field=#timestamp]",
"time_in_nanos" : 18317,
"breakdown" : {
"set_min_competitive_score_count" : 0,
"match_count" : 0,
"shallow_advance_count" : 0,
"set_min_competitive_score" : 0,
"next_doc" : 1474,
"match" : 0,
"next_doc_count" : 1,
"score_count" : 0,
"compute_max_score_count" : 0,
"compute_max_score" : 0,
"advance" : 1541,
"advance_count" : 1,
"score" : 0,
"build_scorer_count" : 2,
"create_weight" : 1184,
"shallow_advance" : 0,
"create_weight_count" : 1,
"build_scorer" : 14118
}
}
]
}
],
"rewrite_time" : 4660,
"collector" : [
{
"name" : "SimpleTopScoreDocCollector",
"reason" : "search_top_hits",
"time_in_nanos" : 22374
}
]
}
],
"aggregations" : [ ]
},
{
"id" : "[Axyv60mYQEGAREa2TwbgMQ][items-2020][0]",
"searches" : [
{
"query" : [
{
"type" : "MatchNoDocsQuery",
"description" : """MatchNoDocsQuery("User requested "match_none" query.")""", <-- here
"time_in_nanos" : 4166,
"breakdown" : {
"set_min_competitive_score_count" : 0,
"match_count" : 0,
"shallow_advance_count" : 0,
"set_min_competitive_score" : 0,
"next_doc" : 0,
"match" : 0,
"next_doc_count" : 0,
"score_count" : 0,
"compute_max_score_count" : 0,
"compute_max_score" : 0,
"advance" : 0,
"advance_count" : 0,
"score" : 0,
"build_scorer_count" : 1,
"create_weight" : 1791,
"shallow_advance" : 0,
"create_weight_count" : 1,
"build_scorer" : 2375
}
}
],
"rewrite_time" : 4353,
"collector" : [
{
"name" : "SimpleTopScoreDocCollector",
"reason" : "search_top_hits",
"time_in_nanos" : 12887
}
]
}
],
"aggregations" : [ ]
}
]
}
}
So there are couple of questions here:
Does skipping truly skip shards?
How are skipped shards and MatchNoDocsQuery different?
What's the cost of MatchNoDocsQuery?
How does sorting allow shards to be skipped?
If we sort results, do we really completely skip shards and not even touch them during search?
That's a great deal of questions bundled into one, but here's my attempt:
Does skipping truly skip shards?
How does sorting allow shards to be skipped?
If we sort results, do we really completely skip shards and not even touch them during search?
Yes, ES tries to be smart enough to figure out which shards to hit before actually sending the query to those shards. The _search_shards API helps here but not only as can be seen from the explanation in this issue.
If you search issues for the keywords can_match, skip and shard you'll find plenty of other optimizations implemented all over the place that aim at making ES execution plan smarter and faster.
If you want to see how this is coded, you can start in the SearchService.canMatch() method. That's where the service can decide whether the query can be rewritten to MatchNoDocsQuery. If you add a suggest or global aggregation (which must visit all documents no matter what), you'll see that shards are not skipped any more, even with the sort present.
What's the cost of MatchNoDocsQuery?
I wouldn't worry about it, as it's not only negligible, but out of your hands.
How does sorting allow shards to be skipped?
As explained in the issue #51852 I linked above, This change will rewrite the shard queries to match none if the bottom sort value computed in prior shards is better than all values in the shard. In other words, ES is smart enough to know which will contain valid hits or not depending on the sort value. In your case, since the sort on the timestamp excludes all values from 2020, ES knows that the shard(s) from the 2020 index can be excluded since none will match.
Another possibility is to leverage index sorting so that terms are sorted at indexing time. Terms are sorted in each segment of the index but also every time segments are merged, the new merged set of terms needs to be resorted again, so this can have performance implications. Test before use!
I need to compare two fields in Elasticsearch. I tried with below query using Kibana. But it's a gave runtime exception. Kindly help me how to compare these fields.
GET /eps/_search
{
"query": {
"bool" : {
"filter" : {
"script" : {
"script" : {
"source": "doc['output_record_count'].value < doc['input_record_count'].value",
"lang" : "painless"
}
}
}
}
}
}
Note:
For complete Match query I am getting below success response.
Query:
GET /eps/_search
{
"query": {
"match_all": {}
}
}
The Response is :
{
"took" : 0,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : {
"value" : 5,
"relation" : "eq"
},
"max_score" : 1.0,
"hits" : [
{
"_index" : "eps",
"_type" : "_doc",
"_id" : "9bNkeXEBLNJ-eURYKdv1",
"_score" : 1.0,
"_source" : {
"experience" : "EPS",
"#version" : "1",
"sdcids" : "013bb234-0840-11ea-8e7d-515f88cf3efa",
"output_record_count" : 13,
"input_record_count" : 10,
"#timestamp" : "2020-04-14T15:52:19.582Z",
"SDC_Ids" : "013bb234-0840-11ea-8e7d-515f88cf3efa"
}
}
]
}
}
Ok, I have example result on my data in elastic search :
"hits" : [
{
"_index" : "solutionpedia_data",
"_type" : "doc",
"_id" : "nyODP24BA840z5O6WguE",
"_score" : 46.63439,
"_source" : {
"ID" : "1",
"PRODUCT_NAME" : "ATM",
"UPDATEDATE" : "13-FEB-18",
"PROPOSAL" : [
{
}
],
"MARKETING_KIT" : [ ],
"VIDEO" : [ ]
}
},
{
"_index" : "classification",
"_type" : "doc",
"_id" : "5M-r5m4BNYha4zuWalJa",
"_score" : 39.25268,
"_source" : {
"productId" : "1",
"productName" : "ATM",
"productIconUrl" : "media/8ae0f0c3-1402-4559-901e-7ec9b874ce68-prod032.webp",
"type" : "nonconnectivity",
"businessLineId" : "",
"subsidiaries" : "",
"segment" : [],
"productType" : "Efisien",
"tariff" : null,
"tags" : [ ],
"contact" : [],
"mediaId" : [
"Med391"
],
"documentId" : [
"doc260",
"doc261"
],
"createdAt" : "2019-09-22T05:22:46.956Z",
"updatedAt" : "2019-09-22T05:22:46.956Z",
"totalClick" : 46
}
}
]
this is a result of my alias. can we search for the same data based on 2 different fields, the example above is the ID and productId fields. Can we make these 2 objects in one bucket or compare?
i was try with some aggregate but nothing :
{
"query": {
"match_all": {}
},
"size": 0,
"aggregations": {
"product catalog": {
"terms": {
"field": "productId.keyword",
"min_doc_count": 2,
"size": 100
},
"aggregations": {
"product solped": {
"terms": {
"field": "ID.keyword",
"min_doc_count": 2
}
}
}
}
}
}
result :
{
"took" : 9,
"timed_out" : false,
"_shards" : {
"total" : 10,
"successful" : 10,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 1276,
"max_score" : 0.0,
"hits" : [ ]
},
"aggregations" : {
"product catalog" : {
"doc_count_error_upper_bound" : 0,
"sum_other_doc_count" : 0,
"buckets" : [ ]
}
}
}
You can achieve this with a Scripted Bucket Aggregation, using script logic to define your buckets (pseudo code: if field a exists value of field a, if field b exists value of field b).
Another (and better) way to achieve this is to change your data model and indexing logic on Elasticsearch side and store the information in a field of the same name.
You could also consider the alias data type to make fields with different names in different indices accessible under one common field name. This is also the approach Elastic takes with the Elastic Common Schema specification.
I'm using Elasticsearch 6.6 and have an index (1 shard, 1 replica) with the geonames (https://www.geonames.org/) dataset indexed (indexsize =1.3 gb, 11.8 mio geopoints).
I was playing around a bit with the geo distance sorting query, sorting the whole index for some origin points. So after some testing I saw that sorting ascending is always faster than sorting descending. here is an example query (i also tested with bigger "size"-parameter):
POST /geonames/_search?request_cache=false
{
"size":1,
"sort" : [
{
"_geo_distance" : {
"location" : [8, 49],
"order" : "asc",
"unit" : "m",
"mode" : "min",
"distance_type" : "arc",
"ignore_unmapped": true
}
}
]
}
Here is the answer for ascending sorting (with explain and profile True):
{
"took" : 1374,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 11858060,
"max_score" : null,
"hits" : [
{
"_shard" : "[geonames][0]",
"_node" : "qXTymyB9QLmxhPtGEtA_mA",
"_index" : "geonames",
"_type" : "doc",
"_id" : "L781LmkBrQo0YN4qP48D",
"_score" : null,
"_source" : {
"id" : "3034701",
"name" : "ForĂȘt de Wissembourg",
"location" : {
"lat" : "49.00924",
"lon" : "8.01542"
}
},
"sort" : [
1523.4121312414704
],
"_explanation" : {
"value" : 1.0,
"description" : "*:*",
"details" : [ ]
}
}
]
},
"profile" : {
"shards" : [
{
"id" : "[qXTymyB9QLmxhPtGEtA_mA][geonames][0]",
"searches" : [
{
"query" : [
{
"type" : "MatchAllDocsQuery",
"description" : "*:*",
"time_in_nanos" : 265223567,
"breakdown" : {
"score" : 0,
"build_scorer_count" : 54,
"match_count" : 0,
"create_weight" : 10209,
"next_doc" : 253091268,
"match" : 0,
"create_weight_count" : 1,
"next_doc_count" : 11858087,
"score_count" : 0,
"build_scorer" : 263948,
"advance" : 0,
"advance_count" : 0
}
}
],
"rewrite_time" : 1097,
"collector" : [
{
"name" : "CancellableCollector",
"reason" : "search_cancelled",
"time_in_nanos" : 1044167746,
"children" : [
{
"name" : "SimpleFieldCollector",
"reason" : "search_top_hits",
"time_in_nanos" : 508296683
}
]
}
]
}
],
"aggregations" : [ ]
}
]
}
}
and here for descending, just switched the parameter from asc to desc (also with profile and explain):
{
"took" : 2226,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 11858060,
"max_score" : null,
"hits" : [
{
"_shard" : "[geonames][0]",
"_node" : "qXTymyB9QLmxhPtGEtA_mA",
"_index" : "geonames",
"_type" : "doc",
"_id" : "Mq80LmkBrQo0YN4q11bA",
"_score" : null,
"_source" : {
"id" : "4036351",
"name" : "Bollons Seamount",
"location" : {
"lat" : "-49.66667",
"lon" : "-176.16667"
}
},
"sort" : [
1.970427111052182E7
],
"_explanation" : {
"value" : 1.0,
"description" : "*:*",
"details" : [ ]
}
}
]
},
"profile" : {
"shards" : [
{
"id" : "[qXTymyB9QLmxhPtGEtA_mA][geonames][0]",
"searches" : [
{
"query" : [
{
"type" : "MatchAllDocsQuery",
"description" : "*:*",
"time_in_nanos" : 268521404,
"breakdown" : {
"score" : 0,
"build_scorer_count" : 54,
"match_count" : 0,
"create_weight" : 9333,
"next_doc" : 256458664,
"match" : 0,
"create_weight_count" : 1,
"next_doc_count" : 11858087,
"score_count" : 0,
"build_scorer" : 195265,
"advance" : 0,
"advance_count" : 0
}
}
],
"rewrite_time" : 1142,
"collector" : [
{
"name" : "CancellableCollector",
"reason" : "search_cancelled",
"time_in_nanos" : 1898324618,
"children" : [
{
"name" : "SimpleFieldCollector",
"reason" : "search_top_hits",
"time_in_nanos" : 1368306442
}
]
}
]
}
],
"aggregations" : [ ]
}
]
}
}
So my question is, why is it like this ? As I understood Es calculates the distance from the origin point to every other point and then sorts them. So why is the descending sorting so much slower ?
Asking the same question on the Elasticsearch board and getting an answer.
So apparantly Elasticsearch uses differnt searching strategies/algorithms for ascending end descending distance sorting.
For the descending sorting it calculates the distance from the origin to every point end then sorts.
For the ascending sorting it uses boundingboxes to filter points near the origin and only calculate the distances for points inside the boundingboxes.