I have a use case to represent an enum for difficulty level (EASY, MEDIUM, DIFFICULT) in elastic search with support of sorting on this field. If this field is indexed as string the sorting will not work as expected.
One way to support this is to index integer values for each enumeration in ES and map it to string values when sorted results are returned by ES.
Are there other alternatives such that ES itself takes care of sorting in the enumeration order while this field is indexed as string? Can I specify custom sort function for a field? function_score is an option, but given that I have to sort based on enum ordering is there better way than defining custom function_score?
In my use case there are multiple such enumeration defining scale across dimensions like difficulty, height (low, medium, high), grades (good, average, poor), etc. Both the above solution requires custom work as a new dimension is introduced. Can either of the above approach be generalzied?
You can check the answer to the same question here. You will need to use script_score like below:
GET /my-index-2/_search
{
"query": {
"script_score": {
"query": {
"match_all":{}
},
"script": {
"source": "if (doc['field name'].value == 'EASY'){2} else if(doc['field name'].value == 'MEDIUM') {1} else if(doc['field name'].value == 'DIFFICULT') {0}"
}
}
}
}
Related
I have an Elasticsearch index with document structure like below.
{
"id": "foo",
"tags": ["Tag1", "Tag2", "Tag3"],
"special_tags": ["SpecialTag1", "SpecialTag2", "SpecialTag3"],
"reserved_tags": ["ReservedTag1", "ReservedTag2", "Tag1", "SpecialTag2"],
// rest of the document
}
The fields tags, special_tags, reserved_tags are stored separately for multiple use cases. In one of the queries, I want to order the documents by number of occurrences for searched tags in all the three fields.
For example, if I am searching with three tags Tag1,
Tag4 and SpecialTag3, total occurrences are 2 in the above document. Using this number, I want to add a custom score to this document and sort by the score.
I am already using function_score as there are few other attributes on which the scoring depends. To compute the matched number, I tried painless script like below.
def matchedTags = 0;
def searchedTags = ["Tag1", "Tag4", "SpecialTag3"];
for (int i = 0; i < searchedTags.length; ++i) {
if (doc['tags'].contains(searchedTags[i])) {
matchedTags++;
continue;
}
if (doc['special_tags'].contains(searchedTags[i])) {
matchedTags++;
continue;
}
if (doc['reserved_tags'].contains(searchedTags[i])) {
matchedTags++;
}
}
// logic to score on matchedTags (returning matchedTags for simplicity)
return matchedTags;
This runs as expected, but extremely slow. I assume that ES has to count the occurrences for each doc and cannot use indexes here. (If someone can shed light on how this will work internally or provide documentation/resources links, that would be helpful.)
I want to have two scoring functions.
Score as a function of number of occurrences
Score higher for higher occurrences. This is basically same as 1, but the repeated occurrences would be counted.
Is there any way where I can get benefits of both faster searching and also the custom scoring using script?
Any help is appreciated. Thanks.
I have an ElasticSearch index with a list of "shops".
I'd like to allow customers to search these shops by both geo_distance (so, search for a point and get shops near that location), and textual match, like matches on shop name / address.
I'd like to get results that match either of these two criteria, and I'd like the order of these results to be a combination of both. The stronger the textual match, and the closer to the point searched, the higher the result. (Obviously, there's going to be a formula to combine these two, that'll need tweaking, not too worried about that part yet).
My issue / what I've tried:
geo_distance is a filter, not a query, so I can't combine both on the query part of the request.
I can use a bool => should filter (rather than query) that matches on either name or location. This gives me the results I want, but not in order.
I can also have _geo_distance as part of a sort clause so that documents closer to the point rank higher.
What I haven't figured out is how I would take the "regular" _score that ElasticSearch gives to documents when doing textual matches, and combine that with the geo_distance score.
By having the textual match in the filter, it doesn't seem to affect the score of documents (which makes sense). And I don't see how I could combine the textual match in the query part and a geo_distance filter so it's an OR rather than an AND.
I guess my best bet would be the equivalent of this:
{
function_score: {
query: { ... },
functions: [
{ geo_distance function },
{ multi_match_result score },
],
score_mode: 'multiply'
}
}
but I'm not sure you can do geo_distance as a score function, and I don't know how to have multi_match_result score as a score function, or if it's even possible.
Any pointers will be greatly appreciated.
I'm working with ElasticSearch v1.4, but I can upgrade if necessary.
but I'm not sure you can do geo_distance as a score function, and I don't know how to have multi_match_result score as a score function, or if it's even possible.
You can't really do it in the way that you're asking, but you can do what you want just as easily. For the simpler case, you get scoring just by using a normal query.
The problem with filters is that they're yes/no questions, so if you use them in a function_score, then it either boosts the score or it doesn't. What you probably want is degradation of the score as the distance from the origin grows. It's the yes/no nature that stops them from impacting the score at all. There's no improvement to relevancy implied by matching a filter -- it just means that it's part of the answer, but it doesn't make sense to say that it should be closer to the top/bottom as a result.
This is where the Decay function score helps. It works with numbers, dates, and -- most helpfully here -- geo_points. In addition to the types of data it accepts, it can decay using either gaussian, exponential, or linear decay functions. The one that you want to choose is honestly arbitrary and you should give the one that chooses the best "experience". I would suggest to start with gauss.
"function_score": {
"functions": [
"gauss": {
"my_geo_point_field": {
"origin": "0, 1",
"scale": "5km",
"offset": "500m",
"decay": 0.5
}
}
]
}
Note that origin is in x, y format (due to standard GeoJSON), which is longitude, latitude.
Each one of the values impacts how the score decays based on the graph (taken wholesale from the documentation). If you would use an offset of 0, then the score begins to drop once it's not exactly at the origin. With the offset, it allows it some buffer to be considered just as good.
The scale is directly associated with the decay in that the score will be chopped down by the decay value once it is scale-distance away from the origin (+/- the offset). In my above example, anything 5km from the origin would get half of the score as anything at the origin.
Again, just note that the different types of decay functions change the shape of scoring.
I'd like the order of these results to be a combination of both.
This is the purpose of the bool / should compound query. You get OR behavior with score improvement based on each match. Combining this with the above, you'd want something like:
{
"query": {
"bool": {
"should": [
{
"multi_match": { ... }
},
{
"function_score": {
"functions": [
"gauss": {
"my_geo_point_field": {
"origin": "0, 1",
"scale": "5km",
"offset": "500m",
"decay": 0.5
}
}
]
}
}
]
}
}
}
NOTE: If you add a must, then the should behavior changes from literal OR-like behavior (at least 1 must match) to completely optional behavior (none must match).
I'm working with ElasticSearch v1.4, but I can upgrade if necessary.
Starting with Elasticsearch 2.0, every filter is a query and every query is also a filter. The only difference is the context that it's used in. This doesn't change my answer here, but it's something that may help you in the future in addition to what I say next.
Geo-related performance increased dramatically in ES 2.2+. You should upgrade (and recreate your geo-related indices) to take advantage of those changes. ES 5.0 will have similar benefits!
Given an index of books that have a title, an author, and a description, I'd like the resulting search results to be sorted this way:
all books that match the title sorted by downloads (a numeric value)
all books that match on author sorted by downloads
all books that match on description sorted by downloads
I use the search query below, but the problem is that each entry has a different score thus making sorting by downloads irrelevant.
e.g. when the search term is 'sorting' - title: 'sorting in elastic search' will score higher than title: 'postgresql sorting is awesome' (because of the word position).
query = QueryBuilders.multiMatchQuery(queryString, "title^16", "author^8", "description^4")
elasticClient.prepareSearch(Index)
.setTypes(Book)
.setQuery(query)
.addSort(SortBuilders.scoreSort())
.addSort(SortBuilders.fieldSort("downloads").order(SortOrder.DESC))
How do I construct my query so that I could get the desired book sorting?
I use standard analysers and I need to the search query to be analysed, also I will have to handle multi-word search query strings.
Thx.
What you need here is a way to compute score based on three weighted field and a numeric field. Sort will sum the score obtained from both , due to which if either one of them is too large , it will supersede the other.
Hence a better approach would be to multiple downloads with the score obtained by the match.
So i would recommend function score query -
{
"query": {
"function_score": {
"query": {
"multi_match": {
"query": "sorting",
"fields": [
"title^16",
"author^8",
"description^4"
]
}
},
"function": [
{
"field_value_factor": {
"field": "downloads"
}
}
],
"boost_mode": "multiply"
}
}
}
This will compute the score based on all three fields. And then multiply that score with the value in download field to get the final score. The multiply boost_mode decides how the value computed by functions are clubbed together with the score computed by query.
I want to know how it's possible to get distinct value of a field in elastic search. I read an article here shows how to do that with facets, but I read facets are deprecated:
http://elasticsearch-users.115913.n3.nabble.com/Getting-Distinct-Values-td3830953.html
Is there any other way to do that? if not is it possible to tell me how to do that? it's abit hard to understand solutions like this: Elastic Search - display all distinct values of an array
Use aggregations:
GET /my_index/my_type/_search?search_type=count
{
"aggs": {
"my_fields": {
"terms": {
"field": "name",
"size": 1000
}
}
}
}
You can use the Cardinality metric
Although the counts returned aren't guaranteed to be 100% accurate, they almost always are for low cardinality terms and the precision is configurable via the precision_threshold param.
http://www.elastic.co/guide/en/elasticsearch/guide/current/cardinality.html
I would like to retrieve data about a string field like the min, max and average length (by counting the number of characters inside the string). My issue is that aggregations can only be used for numeric fields. Besides, I tried it using a simple statistical facet,
"query":{
"match_all": {}
},
"facets":{
"stat1":{
"statistical":{
"field":"title"}
}
}
but I get shard failures and SearchPhaseExecutionException. When trying with a script field the error returned is an OutOfMemoryError:
"query":{
"match_all": {}
},
"script_fields":{
"test1":{"script": "doc[\"title\"].value" }
}
Is it possible to retrive such data about a simple "title" string field using CURL? Thank you!
I haven't actually tried the following, but I believe it should work.
First some useful doc-references:
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-facets-statistical-facet.html.
In order to implement the statistical facet, the relevant field values
are loaded into memory from the index. This means that per shard,
there should be enough memory to contain them. Since by default,
dynamic introduced types are long and double, one option to reduce the
memory footprint is to explicitly set the types for the relevant
fields to either short, integer, or float when possible.
I'm not sure directly how to set the type of the script-field to 'short' which is probably what you want. to reduce memory. it SHOULD be possible though.
ALSO: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-script-fields.html
It’s important to understand the difference between
doc['my_field'].value and _source.my_field. The first, using the doc
keyword, will cause the terms for that field to be loaded to memory
(cached), which will result in faster execution, but more memory
consumption. Also, the doc[...] notation only allows for simple valued
fields (can’t return a json object from it) and make sense only on
non-analyzed or single term based fields.
So ALTERNATIVE: would be to use _source instead of doc which would not cache the lengths.
Gives:
{
"query" : {
"match_all" : {}
},
"facets" : {
"stat1" : {
"statistical" : {
"script" : "doc['title'].value.length()
//"script" : "_source.title.length() //ALTERNATIVE which isn't cached
}
}
}
}