Elasticsearch query filter combination issue - elasticsearch

Im trying to understand why the below elasticsearch query does not work.
EDIT:
The fields mentioned in the query are from different indices. For example Filter has classification field which is in a different index to the fields mentioned in the query string.
The expectation of the filter query is that when the user searches specifically on classification field i.e. secret or protected then the values are displayed. Else if the user searches for any other field from a different index for example firstname or person, then it should not consider any filter applied as firstname or person is not part of the filter
{
"query": {
"bool": {
"filter": {
"terms": {
"classification": [
"secret",
"protected"
]
}
},
"must": {
"query_string": {
"query": "*john*",
"fields": [
"classification",
"firstname",
"releasability",
"person"
]
}
}
}
}
}
The result expected is john in the field person is returned. This works when there is no filter applied in the above code as
{
"query": {
"query_string": {
"query": "*john*",
"fields": [
"classification",
"firstname",
"releasability",
"person"
]
}
}
}
The purpose of the filter is only to filter records when the said fields contain the values mentioned, otherwise it should work for all values.
Why is it not producing the results for john and only producing results for classification values only?

Adding a working example with sample index data and search query.
To know more about Bool query refer this official documentation
Index Data:
Index data in my_index index
{
"name":"John",
"title":"b"
}
{
"name":"Johns",
"title":"a"
}
Index data in my_index1 index
{
"classification":"protected"
}
{
"classification":"secret"
}
Search Query :
POST http://localhost:9200/_search
{
"query": {
"bool": {
"should": [
{
"bool": {
"filter": [
{
"terms": {
"classification": [
"secret",
"protected"
]
}
}
]
}
},
{
"bool": {
"must": [
{
"query_string": {
"query": "*john*",
"fields": [
"name",
"title"
]
}
}
]
}
}
]
}
}
}
Search Result:
"hits": [
{
"_index": "my_index",
"_type": "_doc",
"_id": "1",
"_score": 1.0,
"_source": {
"name": "John",
"title": "b"
}
},
{
"_index": "my_index",
"_type": "_doc",
"_id": "2",
"_score": 1.0,
"_source": {
"name": "Johns",
"title": "a"
}
},
{
"_index": "my_index1",
"_type": "_doc",
"_id": "1",
"_score": 0.0,
"_source": {
"classification": "secret"
}
},
{
"_index": "my_index1",
"_type": "_doc",
"_id": "2",
"_score": 0.0,
"_source": {
"classification": "protected"
}
}
]

Related

ElasticSearch: more_like_this query

I have an index = "es_demo" , where I need to find similar documents to given "_id",
I don't think it is working as the returned results have same "_id" as mentioned in the query .
And as written in the elastic documentation having "include" parameter as "false" will not be returning the "ids" mentioned in the query.
{
"query": {
"more_like_this": {
"fields": "_doc",
"like": {
"docs": [
{
"_id": "5fac83afdce931230ef44c0a"
},
{
"_id": "5f80096adce931230e8bdb2d"
}
]
}
}
},
"include": "false"
}
Can someone please help me out here I think the query I wrote is wrong.
I also tried these queries :
{
"query": {
"more_like_this": {
"fields": "_doc",
"like": [
{
"_id": "5fac83afdce931230ef44c0a"
},
{
"_id": "5f80096adce931230e8bdb2d"
}
]
}
}
}
{
"query": {
"more_like_this": {
"fields": "_doc",
"like": [
{
"_id": "5fac83afdce931230ef44c0a"
},
{
"_id": "5f80096adce931230e8bdb2d"
}
]
}
},
"include": "False"
}
The first result I got was the same document with "_id": "5fac83afdce931230ef44c0a" for every query
The query below works for my index movies.
Remember about parameter fields:
A list of fields to fetch and analyze the text from. Defaults to the
index.query.default_field index setting, which has a default value of
*. The * value matches all fields eligible for term-level queries, excluding metadata fields.
Query (edit for you case)
GET idx_movies/_search
{
"_source": [
"title"
],
"query": {
"more_like_this": {
"fields": [
"title", "description"
],
"like": [
{
"_id": "GjP1WYUBQB-6H-4Z96IG"
}
],
"min_term_freq":1
}
}
}

Elasticsearch - unify search results from different indexes

I want to perform a search query on different indexes with different search queries and unify the results.
I know there is a multi-target syntax, which allows me to perform specific query over multiple indexes.
What I want is different query for each index and then perform something like UNION (SQL).
Is there a way to achieve that?
You can use the _index metadata field. This will help you to query on multiple indexes with different queries
Adding a working example with index data, search query and search result
Index Data
POST /index1/_doc/1
{
"name":"foo"
}
POST /index2/_doc/1
{
"name":"bar"
}
Search Query:
{
"query": {
"bool": {
"should": [
{
"bool": {
"must": [
{
"match": {
"name": "foo"
}
},
{
"term": {
"_index": "index1"
}
}
]
}
},
{
"bool": {
"must": [
{
"match": {
"name": "bar"
}
},
{
"term": {
"_index": "index2"
}
}
]
}
}
]
}
}
}
Search Result:
"hits": [
{
"_index": "index1",
"_type": "_doc",
"_id": "1",
"_score": 1.287682,
"_source": {
"name": "foo"
}
},
{
"_index": "index2",
"_type": "_doc",
"_id": "1",
"_score": 1.287682,
"_source": {
"name": "bar"
}
}
]

Elastic Search 1.4 phrase query with OR operator with hyphen (-) in search string

I have a issue in Elastic search 1.4 phrase query. I am creating a below index with the data.
curl -XPUT localhost:9200/test
curl -XPOST localhost:9200/test/doc/1 -d '{"field1" : "abc-xyz"}'
curl -XPOST localhost:9200/test/doc/2 -d '{"field1" : "bcd-gyz"}'
So by default field1 is analyzed by elastic search with default analyzer.
I am searching below phrase query but its not returning any result.
{
"query": {
"filtered": {
"filter": {
"bool": {
"should": [
{
"query": {
"multi_match": {
"query": "abc\\-xyz OR bcd\\-gyz",
"type": "phrase",
"fields": [
"field1"
]
}
}
}
]
}
}
}
}
}
So elastic search phrase query is not working with OR operator. Any idea why its not working, is it a limitation of elastic search because of special character hyphen (-) in text?
Based on the comment, adding a answer using query string which works with OR in phrase with multiple search, it didn't work with multiple multi-match hence have to use query string.
Using the same indexed doc, added in previous answer, but with below search query.
{
"query": {
"bool": {
"must": [
{
"query_string": {
"query": "\"abc-xyz\" OR \"bcd-gyz\"",
"fields": [
"title"
]
}
}
]
}
}
}
Search results
"hits": [
{
"_index": "phrasemulti",
"_type": "doc",
"_id": "1",
"_score": 0.05626005,
"_source": {
"title": "bcd-gyz"
}
},
{
"_index": "phrasemulti",
"_type": "doc",
"_id": "2",
"_score": 0.05626005,
"_source": {
"title": "abc-xyz"
}
}
]
When you remove few char, pharse query won't work or when you change operator to AND, sample data doesn't return search results which is expected.
{
"query": {
"bool": {
"must": [
{
"query_string": {
"query": "\"abc-xyz\" OR \"bcd-gz\"",
"fields": [
"title"
]
}
}
]
}
}
}
Returns only one search result, as there is no phrase bcd-gz exist in sample data.
"hits": [
{
"_index": "phrasemulti",
"_type": "doc",
"_id": "2",
"_score": 0.05626005,
"_source": {
"title": "abc-xyz"
}
}
]
Below query works fine for me
{
"query": {
"filtered": {
"filter": {
"bool": {
"should": [
{
"query": {
"multi_match": {
"query": "abc-xyz", // note passing only one query without escaping hyphen
"type": "phrase",
"fields": [
"title"
]
}
}
}
]
}
}
}
}
}
Search results with explain param
"hits": [
{
"_shard": 3,
"_node": "1h3iipehS2abfclj51Vtsg",
"_index": "phrasemulti",
"_type": "doc",
"_id": "2",
"_score": 1.0,
"_source": {
"title": "abc-xyz"
},
"_explanation": {
"value": 1.0,
"description": "ConstantScore(BooleanFilter(QueryWrapperFilter(title:\"abc xyz\"))), product of:",
"details": [
{
"value": 1.0,
"description": "boost"
},
{
"value": 1.0,
"description": "queryNorm"
}
]
}
}
]
Verified its returning results according to phrase as query abc-xy doesn't return any result.

Elasticsearch associating exact match terms

I have a search index of filenames containing over 100,000 entries that share about 500 unique variations of the main filename field. I have recently made some modifications to certain filename values that are being generated from my data. I was wondering if there is a way to link certain queries to return an exact match. In the following query:
"query": {
"bool": {
"must": [
{
"match": {
"filename": "foo-bar"
}
}
],
}
}
how would it be possible to modify the index and associate the results so that above query will also match results foo-bar-baz, but not foo-bar-foo or any other variation?
Thanks in advance for your help
You can use a term query instead of a match query. Perfect to use on a keyword:
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-term-query.html
Adding a working example with index data and search query. (Using the default mapping)
Index Data:
{
"fileName": "foo-bar"
}
{
"fileName": "foo-bar-baz"
}
{
"fileName": "foo-bar-foo"
}
Search Query:
{
"query": {
"bool": {
"should": [
{
"match": {
"fileName.keyword": "foo-bar"
}
},
{
"match": {
"fileName.keyword": "foo-bar-baz"
}
}
]
}
}
}
Search Result:
"hits": [
{
"_index": "test",
"_type": "_doc",
"_id": "1",
"_score": 0.9808291,
"_source": {
"fileName": "foo-bar"
}
},
{
"_index": "test",
"_type": "_doc",
"_id": "2",
"_score": 0.9808291,
"_source": {
"fileName": "foo-bar-baz"
}
}
]

elasticSearch: bool query with multiple values on one field

This works:
GET /bitbucket$$pull-request-activity/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"prid": "12343"
}
},
{
"match": {
"repoSlug": "com.xxx.vserver"
}
}
]
}
}
}
But I would like to capture multiple prids in one call.
This does not work however:
GET /bitbucket$$pull-request-activity/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"prid": "[12343, 11234, 13421]"
}
},
{
"match": {
"repoSlug": "com.xxx.vserver"
}
}
]
}
}
}
any hints?
As you are using must in your bool query, then this represents logical AND, so be sure that all the documents that you are Matching of the prid field, should also match with "repoSlug": "com.xxx.vserver".
If none of the documents match with "repoSlug": "com.xxx.vserver", then no result will return.
And, if only 2 documents match, then only 2 of them will be returned in the search result, and not all the documents.
Adding Working example with mapping, sample docs and search query
Index Sample Data :
{
"id":"1",
"message":"hello"
}
{
"id":"2",
"message":"hello"
}
{
"id":"3",
"message":"hello-bye"
}
Search Query:
{
"query": {
"bool": {
"must": [
{
"match": {
"id": "[1, 2, 3]"
}
},
{
"match": {
"message": "hello"
}
}
]
}
}
}
Search Result :
"hits": [
{
"_index": "foo14",
"_type": "_doc",
"_id": "1",
"_score": 1.5924306,
"_source": {
"id": "1",
"message": "hello"
}
},
{
"_index": "foo14",
"_type": "_doc",
"_id": "3",
"_score": 1.4903541,
"_source": {
"id": "3",
"message": "hello-bye"
}
},
{
"_index": "foo14",
"_type": "_doc",
"_id": "2",
"_score": 1.081605,
"_source": {
"id": "2",
"message": "hello"
}
}
]

Resources