I have 7.5.0 ELK stack. And see very strange situation. I have index with "message" field and for example this line in it:
[comgId:1773182151883136235;expired:false;delivered:true;secured:false;querySecured:true]
when I do query "message : 1773182151883136235" it finds nothing (from Discovery page in Kibana)
but if I add filter under the search bar "message is 1773182151883136235" :
{
"query": {
"match": {
"message": {
"query": "1773182151883136235",
"type": "phrase"
}
}
}
}
elastic finds that string. Why???
When inspect both queries I see that search parameters passed in "query" -> "bool" -> "filter": array.
But when I do query in search bar - /index/_search request has :
"bool": {
"should": [
{
"match": {
"message": 1773182151883136300
}
}
],
"minimum_should_match": 1
}
and find nothing, but with added filter - /index/_search request has:
{
"match_phrase": {
"message": {
"query": "1773182151883136235"
}
}
},
and search succeeded
For users more convenient to write search queries in familiar search bar, instead create filter. Why this happened - is this bug or normal behavior ?
You don't have a message field in your sample document, I'll just assume we are talking about comgId.
The new Kibana Query Language (KQL) is a bit pickier about the datatype. Searching for comgId: 1773182151883136235 doesn't find anything, but changing it to comgId: "1773182151883136235" (note the double quotes) works as expected.
PS: Switching to the Lucene query syntax both with and without double quotes works.
Related
Could you please help me on this? My Kibana Database within "Discover" contains a list of trades. I know want to find all trades within this DB that have been done in specific instruments (ISIN-Number). When I add a filter manually and switch to Elasticserach Query DSL, I find the following:
{
"query": {
"bool": {
"should": [
{
"match_phrase": {
"obdetails.isin": "CH0253592783"
}
},
{
"match_phrase": {
"obdetails.isin": "CH0315622966"
}
},
{
"match_phrase": {
"obdetails.isin": "CH0357659488"
}
}
],
"minimum_should_match": 1
}
}
}
Since I want to check the DB for more than 200 ISINS, this seems to be inefficient. Is there a way, in which I could just say "show me the trade if it contains one of the following 200 ISINs?".
I already googled and tried this, which did not work:
{
"query": {
"terms": {
"obdetails.isin": [ "CH0357659488", "CH0315622966"],
"boost": 1.0
}
}
}
The query works, but does not show any results.
To conclude. A field of type text is analyzed which basically converts the given data to a list of terms using given analyzers etc. rather than it being a single term.
Given behavior causes the terms query to not match these values.
Rather than changing the type of the field one may add an additional field of type keyword. That way a terms queries can be performed whilst still having the ability to match on the field.
{
"isin": {
"type" "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
The above example will add an extra field called obdetails.isin.keyword which can be used for terms. While still being able to use match queries on obdetails.isin
In Elastic search, i am using match_phrase_prefix to search data,
it is not working as expected,
{
"match_phrase_prefix": {
"title": {
"query": "microso"
}
}
}
Above sample returning all results with title Microsoft,
but for below query data is empty.
{
"match_phrase_prefix": {
"title": {
"query": "micro"
}
}
}
If prefix search word length is 50% then results are coming.
Please suggest how to fix it.
Mapping screenshot
Thanks
I had a nested field source in my index seems like this:
"source": [
{
"name": "source_c","type": "type_a"
},
{
"name": "source_c","type": "type_b"
}
]
I used query_string query and simple_query_string query to query type_a and got two different result.
query_string
{
"size" : 3,
"query" : {
"bool" : {
"filter" : {
"query_string" : {
"query" : "source:\"source.type:=\"type_a\"\""
}
}
}
}
}
I got 163459 hits in 294088 docs.
simple_query_string
{
"size": 3,
"query": {
"bool": {
"filter": {
"simple_query_string": {
"query": "source:\"source.type:=\"type_a\"\""
}
}
}
}
}
I got 163505 hits in 294088 docs.
I only made three different types type_a,type_b,type_c randomly. So I had to say 163459 and 163505 were very little difference in 294088 docs.
I noly got one info in Elasticsearch Reference [2.1]
Unlike the regular query_string query, the simple_query_string query will never throw an exception, and discards invalid parts of the query.
I don't think it's the reason to make the difference.
I want to know what make the little different results between query_string and simple_query_string?
As far as I know, nested query syntax is not supported for either query_string or simple_query_string. It is an open issue, and this is the PR regarding that issue.
Then how are you getting the result? Here Explain API will help you understand what is going on. This query
{
"size": 3,
"query": {
"bool": {
"filter": {
"simple_query_string": {
"query": "source:\"source.type:=\"type_a\"\""
}
}
}
}
}
have a look at the output, you will see
"description": "ConstantScore(QueryWrapperFilter(_all:source _all:source.type _all:type_a)),
so what is happening here is that ES looking for term source , source.type or type_a, it finds type_a and returns the result.
You will also find something similar with query_string using explain api
Also query_string and simple_query_string have different syntax, for e.g field_name:search_text is not supported in simple_query_string.
Correct way to query nested objects is using nested query
EDIT
This query will give you desired results.
{
"query": {
"nested": {
"path": "source",
"query": {
"term": {
"source.type": {
"value": "type_a"
}
}
}
}
}
}
Hope this helps!!
Acording to the documentation simple_query_string is meant to be used with unsafe input.
So that users can enter anything and it will not throw exception if input is invalid. Will simply discard invalid input.
I am having an issue with highlighting in Elastic 2.0 and 2.1 - it's returning more information than I think it should.
I am constructing a bool query (the filtered query keyword is deprecated in 2.0+ so I am trying to update my syntax). I am building a must section and a filter section within the query, followed by a request for highlighting information.
The documentation says to use the query either in a query context or a filter context, but the highlighter doesn't seem to denote such a distinction.
Here is my fully formed query:
GET /sample04/_search
{
"query": {
"bool": {
"must": [
{
"query": { "query_string": { "query": "east west" } }
}
],
"filter": [
{
"terms": {"OwnerId": ["1", "2","3"]}
}
]
}
},
"highlight": {
"fields": {
"*": { "require_field_match": "false" }
}
}
}
So this query works as expected - we are querying for terms east or west, and we are filtering documents on an Id field that is part of our security requirements, and then I ask for highlighting information.
The downside, however, is the highlighting information contains a hit every instance of every value I submitted in my filter (in this case 1, 2 or 3) that matched any value in any field in any part of my document, like this:
"highlight": {
"SomeTextField": [
"North <em>West</em>"
],
"OwnerId": [
"<em>3</em>"
],
"SerialNumber": [
"<em>3</em>-<em>3</em>"
],
"AssociatedValue": [
"<em>3</em>",
"<em>2</em>"
],
"RelatedValue": [
"<em>3</em>",
"<em>3</em>",
"<em>3</em>",
"<em>3</em>",
"<em>3</em>"
]
}
How do I get the highlighter to match my query in the must section, but ignore the filter? It is my belief that it should ignore highlighting matches that were part of the filter, notably when it's highlighting fields that contain values were requested to filter a SPECIFIC FIELD, but it's utilizing the value anywhere within my document. This seems wrong somehow, but perhaps it's my understanding.
As an FYI, if I set require_field_match to TRUE, then I ONLY get hits that match the filter, and NONE that match the query.
I cannot specify a field to generate highlighting information for, whereas we consume Elastic as a search once find anywhere model, so I don't know field my result will return from.
Can you see what I'm doing wrong? It would be greatly appreciated to understand this.
You can use highlight query for this purpose. change your highlight part to
"highlight": {
"fields": {
"*": {
"highlight_query": {
"query_string": {
"query": "east west"
}
}
}
}
}
I am trying to create a search that will return me exactly what i requested.
For instance let's say i have 2 documents with a field named 'Val'
First doc have a value of 'a - Copy', second document is 'a - Copy (2)'
My goal is to search exactly the value 'a - Copy' and find only the first document in my returned results and not both of them with different similarity rankings
When i try most of the usual queries like:
GET test/_search
{
"query": {
"match": {
"Val": {
"query": "a - copy",
"type": "phrase"
}
}
}
}
or:
GET /test/doc/_search
{
"query": {
"query_string": {
"default_field": "Val",
"query": "a - copy"
}
}
}
I get both documents all the time
There is a very good documentation for finding exact values in ES:
https://www.elastic.co/guide/en/elasticsearch/guide/current/_finding_exact_values.html
It shows you how to use the term filter and it mentions problems with analyzed fields, too.
To put it in a nutshell you need to run a term filter like this (I've put your values in):
GET /test/doc/_search
{
"query" : {
"filtered" : {
"query" : {
"match_all" : {}
},
"filter" : {
"term" : {
"Val" : "a - copy"
}
}
}
}
}
However, this doesn't work with analyzed fields. You won't get any results.
To prevent this from happening, we need to tell Elasticsearch that
this field contains an exact value by setting it to be not_analyzed.
There are multiple ways to achieve that. e.g. custom field mappings.
Yes, you are getting that because your field is, most likely, analyzed and split into tokens.
You need an analyzer similar to this one
"custom_keyword_analyzer": {
"type": "custom",
"tokenizer": "keyword",
"filter": "lowercase"
}
which uses the keyword tokenizer and the lowercase filter (I noticed you indexed upper case letters, but expect to search with lowercase letters).
And then use a term filter to search your documents.