Multi_match and match queries together - elasticsearch

I have the following queries in elastic search :
{
"query": {
"multi_match": {
"query": "bluefin bat",
"type": "phrase",
"fields": [
"title^5",
"body.value"
]
}
},
"highlight": {
"fields": {
"body.value": {
"number_of_fragments": 3
}
}
},
"fields": [
"title",
"id"
]
}
I have tried using "dis_max" but then two of my fields have to be searched for the same query.
The remaining match query has a different query text.
The remaining match query is like this:
{
"query": {
"match": {
"ingredients": "key1, key2",
"analyzer": "keyword_analyzer"
}
}
}
How can I integrate these two queries without using dis_max for joining.

I figured out the answer. multi_match internally applies :
"dis_max"
Hence, you cannot apply dis_max with multi_match.
But what I could do is I could apply bool query to solve this type of problem.
I could apply should which actually translates to OR boolean value or I could apply must which is equivalent to AND.
So this is how I modified my query :
{
"query": {
"bool":{
"should": [
{"multi_match":
{"query": "SOME_QUERY",
"type": "phrase",
"fields": ["title^5","body"]
}
},
{
"match":{
"labels" :{
"query": "SOME_QUERY",
"analyzer": "keyword_analyzer"
}
}
},
{
"match":{
"displayName" :{
"query": "SOME_QUERY",
"fuzziness": "AUTO"
}
}
}
],
"minimum_number_should_match": "50%"
}
},
"fields": ["title","id","labels","displayName","username"],
"highlight": {
"fields": {
"body.storage.value": {
"number_of_fragments": 3}
}
}
}
I hope this helps someone in future.

Related

Elasticsearch "AND in query_string" vs. "default_operator AND"

elasticsearch v7.1.1
I dont understand the difference between a query_string containing "AND"
vs. "default_operator AND"
I thought it should yield the same result, but doesnt:
HTTP POST http://localhost:9200/umlautsuche
{
"settings": {
"analysis": {
"char_filter": {
"my_char_filter": {
"type": "mapping",
"mappings": ["ph => f"]
}
},
"filter": {
"my_ngram": {
"type": "edge_ngram",
"min_gram": 3,
"max_gram": 10
}
},
"analyzer": {
"my_name_analyzer": {
"tokenizer": "standard",
"char_filter": [
"my_char_filter"
],
"filter": [
"lowercase",
"german_normalization"
]
}
}
}
},
"mappings": {
"date_detection": false,
"dynamic_templates": [
{
"string_fields_german": {
"match_mapping_type": "string",
"match": "*",
"mapping": {
"type": "text",
"analyzer": "my_name_analyzer"
}
}
},
{
"dates": {
"match": "lastModified",
"match_pattern": "regex",
"mapping": {
"type": "date",
"ignore_malformed": true
}
}
}
]
}
}
HTTP POST http://localhost:9200/_bulk
{ "index" : { "_index" : "umlautsuche", "_id" : "1" } }
{"vorname": "Stephan-Jörg", "nachname": "Müller", "ort": "Hollabrunn"}
{ "index" : { "_index" : "umlautsuche", "_id" : "2" } }
{"vorname": "Stephan-Joerg", "nachname": "Mueller", "ort": "Hollabrunn"}
{ "index" : { "_index" : "umlautsuche", "_id" : "3" } }
{"vorname": "Stephan-Jörg", "nachname": "Müll", "ort": "Hollabrunn"}
No results here - unexpected by me:
HTTP POST http://localhost:9200/umlautsuche/_search
{
"query": {
"query_string": {
"query": "Stefan Müller Jör*",
"analyze_wildcard": true,
"default_operator": "AND",
"fields": ["vorname", "nachname"]
}
}
}
This query gives the results as expected by me:
HTTP POST http://localhost:9200/umlautsuche/_search
{
"query": {
"query_string": {
"query": "Stefan AND Müller AND Jör*",
"analyze_wildcard": true,
"default_operator": "AND",
"fields": ["vorname", "nachname"]
}
}
}
How do I configure query/analyzer so I dont need these "AND" between my search terms?
What you are facing is an obscurity of boolean logic of query_string boolean operators, and possibly an undocumented behavior. Because of this obscurity I believe it is better to either use bool query with explicit logic, or to use a copy_to.
Let me explain in a bit more detail what's going on and how can you fix it.
Why doesn't the first query match?
In order to see how the query gets executed, let's set profile: true:
POST /umlautsuche/_search
{
"query": {
"query_string": {
"query": "Stefan Müller Jör*",
"analyze_wildcard": true,
"default_operator": "AND",
"fields": [
"vorname",
"nachname"
]
}
},
"profile": true
}
In the ES response we will see:
"profile": {
"shards": [
{
"id": "[QCANVs5gR0GOiiGCmEwj7w][umlautsuche][0]",
"searches": [
{
"query": [
{
"type": "BooleanQuery",
"description": "+((+nachname:stefan +nachname:muller) | (+vorname:stefan +vorname:muller)) +(nachname:jor* | vorname:jor*)",
"time_in_nanos": 17787641,
"breakdown": {
"set_min_competitive_score_count": 0,
We are interested in this part:
"+((+nachname:stefan +nachname:muller) | (+vorname:stefan +vorname:muller)) +(nachname:jor* | vorname:jor*)"
Without going into deep analysis, we can tell that this query wants to find documents with surname stefan and with surname muller, which is impossible (because stefan is never a surname among the documents).
What we actually want to do, I presume, is "find people whose full name is Stefan Müller Jör*". This is not what the query generated by Elasticsearch does.
Why does the second query match?
Let's do the same trick with explain: true. The response would contain this:
"profile": {
"shards": [
{
"id": "[QCANVs5gR0GOiiGCmEwj7w][umlautsuche][0]",
"searches": [
{
"query": [
{
"type": "BooleanQuery",
"description": "+(nachname:stefan | vorname:stefan) +(nachname:muller | vorname:muller) +(nachname:jor* | vorname:jor*)",
"time_in_nanos": 17970342,
"breakdown": {
We can see that the query got interpreted like this:
"+(nachname:stefan | vorname:stefan) +(nachname:muller | vorname:muller) +(nachname:jor* | vorname:jor*)"
Which we can roughly interpret as "find people whose name or surname is one these three names", which is what we expect it to do.
In the documentation of query_string query it says that with default_operator: AND it should interpret spaces as ANDs:
The default operator used if no explicit operator is specified. For
example, with a default operator of OR, the query capital of Hungary
is translated to capital OR of OR Hungary, and with default operator
of AND, the same query is translated to capital AND of AND Hungary.
The default value is OR.
Although, from what we have just seen, this does not seem to be correct - at least in case of querying multiple fields.
So what can we do about it?
Use bool with explicit logic
This query seems to work:
POST /umlautsuche/_search
{
"query": {
"bool": {
"must": [
{
"query_string": {
"query": "Stefan Müller Jör*",
"analyze_wildcard": true,
"fields": [
"vorname"
]
}
},
{
"query_string": {
"query": "Stefan Müller Jör*",
"analyze_wildcard": true,
"fields": [
"nachname"
]
}
}
]
}
}
}
This query is not an exact equivalent, consider it as an example. For instance, if we would have another record like this, without "Jörg":
{"vorname": "Stephan", "nachname": "Müll", "ort": "Hollabrunn"}
the bool query above would match it despite missing "Jörg". To overcome this you can write a more complex bool query, but this will not do if you wanted to avoid parsing user input.
How can we still use plain, unparsed query string?
Introduce a copy_to field
We can try to use copy_to capability. It will copy the content of several fields into another field and will analyze these fields all together.
We will have to modify the mapping configuration (unfortunately the existing index will have to be recreated):
"mappings": {
"date_detection": false,
"dynamic_templates": [
{
"name_fields_german": {
"match_mapping_type": "string",
"match": "*name",
"mapping": {
"type": "text",
"analyzer": "my_name_analyzer",
"copy_to": "full_name"
}
}
},
{
"string_fields_german": {
"match_mapping_type": "string",
"match": "*",
"mapping": {
"type": "text",
"analyzer": "my_name_analyzer"
}
}
},
{
"dates": {
"match": "lastModified",
"match_pattern": "regex",
"mapping": {
"type": "date",
"ignore_malformed": true
}
}
}
]
}
Then we can populate the index in exactly the same manner as we did before.
Now we can query the new field full_name with the following query:
POST /umlautsuche/_search
{
"query": {
"bool": {
"must": [
{
"query_string": {
"query": "Stefan Müller Jör*",
"analyze_wildcard": true,
"default_operator": "AND",
"fields": [
"full_name"
]
}
}
]
}
}
}
This query will return same 2 documents as the second query. Thus, in this case default_operator: AND behaves as we would expect it, asking for all tokens from the query to be matched.
Hope that helps!

Exclude certain fields from search - Elasticsearch

I have indexed documents with each over 100 field each analysed using Edge gram tokenizer to support Auto-Suggestion. I do require free text search that searches on all fields. When i am trying to do so, search is also happening fields with auto complete analyzed(ex. Data.autocomplete_analyzed). I have to restrict this by searching only fields analysed with type "text"(ex. Data). Is there a method to do so in 1. Index time 2. Query time.
Mapping file:
"mappings": {
"_doc": {
"properties": {
"Data": {
"type": "text",
"fields": {
"autocomplete_analyzed": {
"type": "text",
"analyzer": "autocomplete"
},
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
Search query :
{
"query": {
"bool": {
"should": [
{
"multi_match": {
"query": "aim",
"type": "phrase",
"slop": "2",
"fields": []
}
},
{
"multi_match": {
"query": "aim",
"fuzziness": "1",
"fields": []
}
}
],
"minimum_should_match": 1
}
In query time you can use Source filtering to choose the fields you want.
GET /_search
{
"_source": [ "obj1.*", "obj2.*" ],
"query" : {
"term" : { "user" : "kimchy" }
}
}
If you use query_string for search you can use fields
GET /_search
{
"query": {
"query_string": {
"query": "this AND that OR thus",
"fields": [
"docfilename",
"filepath",
"singlewordfield"
]
}
}
}

Searching in specific fields of types

Consider the following query:
{
"query" : {
"match_phrase" : {
"_all" : "Smith"
}
}
}
How would I specify in which fields of which types it may search, instead of searching in everything? (field names may be non-unique across types)
I've tried the query below, but it didn't work (it doesn't return results, it does when I remove person. from all fields):
{
"query": {
"multi_match": {
"query": "Smith",
"fields": [
"person.first_name",
"person.last_name",
"person.age"
],
"lenient": true
}
}
}
I'm sending these queries to http://localhost:9200/tsf-model/_search.
If you can build your query dynamically, I think you can use a combination of your multi_match query and a type query for each type, in order to achieve what you want:
{
"query": {
"bool": {
"minimum_should_match": 1,
"should": [
{
"bool": {
"filter": [
{
"type": {
"value": "type1"
}
},
{
"multi_match": {
"query": "Smith",
"fields": [
"field1",
"field3",
"field5"
]
}
}
]
}
},
{
"bool": {
"filter": [
{
"type": {
"value": "type2"
}
},
{
"multi_match": {
"query": "Smith",
"fields": [
"field2",
"field4",
"field6"
]
}
}
]
}
}
]
}
}
}

Elastic : search two terms, one on _all, other one on a field

I would like to mix a search on a whole document (eg "developer") and a search on some field for another term (eg "php").
I can do each search separately but I can't mix them.
Here my example (simplified to show only my issue) :
{
"query": {
"function_score": {
"query": {
"match": {
"_all": "developer"
},
"multi_match": {
"query": "php",
"fields": [
"skills.description",
"skills.description",
"skills.details"
],
"operator": "or",
"type": "most_fields"
}
}
}
}
If I run this example I have an error :
Parse Failure [Failed to parse source
Is there a way to search on both _all and specific fields with two terms?
Thanks.
Yes, you're almost there, you need to combine them into a bool/must query:
{
"query": {
"function_score": {
"query": {
"bool": {
"must": [
{
"match": {
"_all": "developer"
}
},
{
"multi_match": {
"query": "php",
"fields": [
"skills.description",
"skills.description",
"skills.details"
],
"operator": "or",
"type": "most_fields"
}
}
]
}
}
}
}
}

elasticsearch : boosting documents in has_parent query

Is there a way to boost the docs that "come" from the has_parent query ?
{
"query": {
"function_score": {
"query": {
"bool": {
"should": [
{
"multi_match": {
"fields": ["name^3", "tags^2", "content"],
"query": "xx"
}
},
{
"has_parent": {
"type": "theparent",
"query": {
"multi_match": {
"type": "best_fields",
"fields": ["name^5", "content"],
"query": "xx"
}
}
}
},
{
"has_child": {
"type": "thechild",
"query": {
"multi_match": {
"fields": ["name^3","content"],
"query": "xx"
}
}
}
}
]
}
},
"score_mode": "sum",
"functions": [
{
"linear": {
"date": {
"origin": "2014-08-29",
"scale": "700d",
"decay": 0.6
}
}
}
]
}
}
More exactly, I would like to boost those docs only when the query matches the name field of the parent
(I haven't found a way to refer to a parent field in the functions i.e. theparent._source.name ~= "xx")
According to the sources from Github (see line 104), the boost parameter is allowed within a has_parent query.
Based on this attribute, you can boost specifically the should clause containing the has_parent query. In your case, the result would be :
...
{
"has_parent": {
"type": "theparent",
"query": {
"multi_match": {
"type": "best_fields",
"fields": ["name^5", "content"],
"query": "xx"
}
},
"boost": 5
}
}
...
I don't know if it can help you, but you will find more insights about boosting query clauses here.

Resources