I got 2 querys, the different between them is just 1 filter term.
The first query:
GET _search
{
"query": {
"filtered": {
"query": {
"query_string": {
"query": "*"
}
},
"filter": {
"and": {
"filters": [
{
"term": {
"type": "log"
}
},
{
"term": {
"context.blueprint_id": "adv1"
}
},
{
"term": {
"context.deployment_id": "deploy1"
}
}
]
}
}
}
}
}
return this result:
{
"_source": {
"level": "info",
"timestamp": "2014-03-24 10:12:41.925680",
"message_code": null,
"context": {
"blueprint_id": "Adv1",
"execution_id": "27efcba7-3a60-4270-bbe2-9f17d602dcef",
"deployment_id": "deploy1"
},
"type": "log",
"#version": "1",
"#timestamp": "2014-03-24T10:12:41.927Z"
}
}
The second query is:
{
"query": {
"filtered": {
"query": {
"query_string": {
"query": "*"
}
},
"filter": {
"and": {
"filters": [
{
"term": {
"type": "log"
}
},
{
"term": {
"context.blueprint_id": "adv1"
}
},
{
"term": {
"context.deployment_id": "deploy1"
}
},
{
"term": {
"context.execution_id": "27efcba7-3a60-4270-bbe2-9f17d602dcef"
}
}
]
}
}
}
}
}
return empty results.
the different between them is in the second query, i just add this term:
{
"term": {
"context.execution_id": "27efcba7-3a60-4270-bbe2-9f17d602dcef"
}
}
and in the result we can see that there is result match to that query, but it still not work.
what i'm doing wrong here?
thanks.
By default, ElasticSearch will treat string fields as text and will analyze them (i.e. tokenize, stem etc. before indexing). This means you might not be able to find them when searching for their exact content.
You should make sure the mapping for the execution_id field is not analyzed. Start with GET /_mappings and work from there. :)
Related
I have the following query that works as expected:
GET <index_name>/_search
{
"sort": [
{
"irFileCreateTime": {
"order": "desc"
}
}
],
"query": {
"bool": {
"should": [
{
"match": {
"fileId": 46704
}
},
{
"match": {
"fileId": 46706
}
},
{
"match": {
"fileId": 46719
}
}
]
}
}
}
The problem is that I need to further filter the data, but the field I need to filter on is a text field. I have tried many different ways of putting a must match into my query but everything is either malformed or filters out all hits when I know it should only filter out half. How can I add a must match "irStatus":"COMPLETE" to this query? Thanks in advance.
What you're after is a term query on, preferably, the keyword of irStatus. That is to say:
GET index/_search
{
"sort": [
{
"irFileCreateTime": {
"order": "desc"
}
}
],
"query": {
"bool": {
"must": [
{
"term": {
"irStatus.keyword": {
"value": "COMPLETE"
}
}
}
],
"should": [
{
"match": {
"fileId": 46704
}
},
{
"match": {
"fileId": 46706
}
},
{
"match": {
"fileId": 46719
}
}
]
}
}
}
Assuming your mapping looks something like this:
{
"mappings": {
"properties": {
"irFileCreateTime": {
"type": "date"
},
"fileId": {
"type": "integer"
},
"irStatus": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword"
}
}
}
}
}
}
The reason it's apparently failing on your end is that "COMPLETE" has been lowercased due to standard analyzer.
Alternatively, you could do:
{
"must":[
{
"query_string":{
"query":"irStatus:COMPLETE AND (fileId:(46704 OR 46706 OR 46719))"
}
}
]
}
I have built two queries in ElasticSearch to get the counts for each error message. for example, the first query is to get how many error messages related to "was not found" error
GET /logstash*/_search
{
"query": {
"bool": {
"filter": {
"bool": {
"must": [
{
"match": {
"kubernetes.pod_name": "api"
}
},
{
"match": {
"log": "error"
}
},
{
"match": {
"log": {
"query": "was not found",
"operator": "and"
}
}
},
{
"range": {"#timestamp": {
"time_zone": "CET",
"gt": "now-7d",
"lte": "now"}}
}
]
}
}
}
},
"aggs" : {
"type_count" : {
"value_count" : {
"script" : {
"source" : "doc['log.keyword'].value"
}
}
}
}
}
The second query is to get the count of error messages related to "Duplicate Entry" error
GET /logstash*/_search
{
"query": {
"bool": {
"filter": {
"bool": {
"must": [
{
"match": {
"kubernetes.pod_name": "api"
}
},
{
"match": {
"log": "error"
}
},
{
"match": {
"log": {
"query": "Duplicate entry",
"operator": "and"
}
}
},
{
"range": {"#timestamp": {
"time_zone": "CET",
"gt": "now-7d",
"lte": "now"}}
}
]
}
}
}
},
"aggs" : {
"type_count" : {
"value_count" : {
"script" : {
"source" : "doc['log.keyword'].value"
}
}
}
}
}
My boss really wants me to combine these individual query into a one big query, then get the list of counts for each error messages in one output. Since we have a lot of error messages, which means we have to write each query for each error message, then we have to run each query to get the counts. Is there a way I can click one run to get the list of counts?
I have been trying use query string query and looking for solutions on either Stack Overflow and Documentation. However, there is no luck
You can use filter aggregation along with the value_count aggregation to combine these two queries. In both the queries, out of the 4 queries inside must clause only one differs. You can take this out and combine them with the two filter aggregations as below:
{
"query": {
"bool": {
"filter": {
"bool": {
"must": [
{
"match": {
"kubernetes.pod_name": "api"
}
},
{
"match": {
"log": "error"
}
},
{
"range": {
"#timestamp": {
"time_zone": "CET",
"gt": "now-7d",
"lte": "now"
}
}
}
]
}
}
}
},
"aggs": {
"not_found_count": {
"filter": {
"match": {
"log": {
"query": "was not found",
"operator": "and"
}
}
},
"aggs": {
"count": {
"value_count": {
"script": {
"source": "doc['log.keyword'].value"
}
}
}
}
},
"duplicate_entry_count": {
"filter": {
"match": {
"log": {
"query": "Duplicate entry",
"operator": "and"
}
}
},
"aggs": {
"count": {
"value_count": {
"script": {
"source": "doc['log.keyword'].value"
}
}
}
}
}
}
}
I am trying to restrict the return of a search query to only those indices that start with abc-* pattern.
I tried the following regex but it didn't work.
{
"sort": [
{
"timestamp": {
"order": "desc",
"unmapped_type": "boolean"
}
}
],
"query": {
"filtered": {
"query": {
"query_string": {
"regexp": {
"index": "abc-*"
}
}
},
"filter": {
"bool": {
"must": [
{
"range": {
"timestamp": {
"gte": "now-24h"
}
}
}
]
}
}
}
}
}
Is it possible to use the indices query and apply regex on it?
even the following doesn't filter appropriately:
{
"sort": [
{
"timestamp": {
"order": "desc",
"unmapped_type": "boolean"
}
}
],
"query": {
"filtered": {
"query": {
"indices" : {
"query" : { "regexp" : { "index" : "abc-.*" } }
}
},
"filter": {
"bool": {
"must": [
{
"range": {
"timestamp": {
"gte": "now-24h"
}
}
}
]
}
}
}
}
}
There's a much easier solution simply by means of specifying your index pattern in the URL directly:
POST /abc-*/_search
{
"sort": [
{
"timestamp": {
"order": "desc",
"unmapped_type": "boolean"
}
}
],
"query": {
"filtered": {
"filter": {
"bool": {
"must": [
{
"range": {
"timestamp": {
"gte": "now-24h"
}
}
}
]
}
}
}
}
}
Not sure but faced same problem in different case . I think problem with - in "abc-*" .
just replace - with space , it will work
"index": "abc *"
The index pattern in the URL only supports native expressions, not regex expressions. It does solve the problem though.
I have a nested documents as:
"someField": "hello",
"users": [
{
"name": "John",
"surname": "Doe",
"age": 2
}
]
according to this https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-exists-query.html, the above should match:
GET /_search
{
"query": {
"exists" : { "field" : "users" }
}
}
whereas the following should not,
"someField": "hello",
"users": []
but unfortunately both do not match. any ideas?
The example mentioned on the Elasticsearch blog refers to string and array of string types, not for nested types.
The following query should work for you:
{
"query": {
"nested": {
"path": "users",
"query": {
"bool": {
"must": [
{
"exists": {
"field": "users"
}
}
]
}
}
}
}
}
Also, you can refer to this issue for more info, which discusses this usage pattern.
This works for me
GET /type/_search?pretty=true
{
"query": {
"bool": {
"must": [
{
"nested": {
"path": "outcome",
"query": {
"exists": {
"field": "outcome.outcomeName"
}
}
}
}
]
}
}
}
With the following index mapping:
{
"index_name": {
"mappings": {
"object_name": {
"dynamic": "strict",
"properties": {
"nested_field_name": {
"type": "nested",
"properties": {
"some_property": {
"type": "keyword"
}
}
}
}
}
}
}
}
I needed to use this query:
GET /index_name/_search
{
"query": {
"nested": {
"path": "nested_field_name",
"query": {
"bool": {
"must": [
{
"exists": {
"field": "nested_field_name.some_property"
}
}
]
}
}
}
}
}
Elasticsearch version 5.4.3
The answer from user3775217 has worked for me but I needed to tweak it to work as expected for must_not. Essentially the bool/must needed to be wrapped around the nested portion of the query:
{
"query": {
"bool": {
"must": [
{
"nested": {
"path": "users",
"query": {
"exists": {
"field": "users"
}
}
}
}
}
]
}
}
I have two queries in ES. Both have different turnaround time on the same set of documents. Both are doing the same thing conceptually. I have few doubts
1- What is the difference between these two?
2- Which one is better to use?
3- If both are same why they are performing differently?
1. Filtered bool
{
"from": 0,
"size": 5,
"query": {
"filtered": {
"filter": {
"bool": {
"must": [
{
"term": {
"called_party_address_number": "1987112602"
}
},
{
"term": {
"original_sender_address_number": "6870340319"
}
},
{
"range": {
"x_event_timestamp": {
"gte": "2016-07-01T00:00:00.000Z",
"lte": "2016-07-30T00:00:00.000Z"
}
}
}
]
}
}
}
},
"sort": [
{
"x_event_timestamp": {
"order": "desc",
"ignore_unmapped": true
}
}
]
}
2. Simple Bool
{
"query": {
"bool": {
"must": [
{
"term": {
"called_party_address_number": "1277478699"
}
},
{
"term": {
"original_sender_address_number": "8020564722"
}
},
{
"term": {
"cause_code": "573"
}
},
{
"range": {
"x_event_timestamp": {
"gt": "2016-07-13T13:51:03.749Z",
"lt": "2016-07-16T13:51:03.749Z"
}
}
}
]
}
},
"from": 0,
"size": 10,
"sort": [
{
"x_event_timestamp": {
"order": "desc",
"ignore_unmapped": true
}
}
]
}
Mapping:
{
"ccp": {
"mappings": {
"type1": {
"properties": {
"original_sender_address_number": {
"type": "string"
},
"called_party_address_number": {
"type": "string"
},
"cause_code": {
"type": "string"
},
"x_event_timestamp": {
"type": "date",
"format": "strict_date_optional_time||epoch_millis"
},
.
.
.
}
}
}
}
}
Update 1:
I tried bool/must query and bool/filter query on same set of data,but I found the strange behaviour
1-
bool/must query is able to search the desired document
{
"query": {
"bool": {
"must": [
{
"term": {
"called_party_address_number": "8701662243"
}
},
{
"term": {
"cause_code": "401"
}
}
]
}
}
}
2-
While bool/filter is not able to search the document. If I remove the second field condition it searches the same record with field2's value as 401.
{
"query": {
"bool": {
"filter": [
{
"term": {
"called_party_address_number": "8701662243"
}
},
{
"term": {
"cause_code": "401"
}
}
]
}
}
}
Update2:
Found a solution of suppressing scoring phase with bool/must query by wrapping it within "constant_score".
{
"query": {
"constant_score": {
"filter": {
"bool": {
"must": [
{
"term": {
"called_party_address_number": "1235235757"
}
},
{
"term": {
"cause_code": "304"
}
}
]
}
}
}
}
}
Record we are trying to match have "called_party_address_number": "1235235757" and "cause_code": "304".
The first one uses the old 1.x query/filter syntax (i.e. filtered queries have been deprecated in favor of bool/filter).
The second one uses the new 2.x syntax but not in a filter context (i.e. you're using bool/must instead of bool/filter). The query with 2.x syntax which is equivalent to your first query (i.e. which runs in a filter context without score calculation = faster) would be this one:
{
"query": {
"bool": {
"filter": [
{
"term": {
"called_party_address_number": "1277478699"
}
},
{
"term": {
"original_sender_address_number": "8020564722"
}
},
{
"term": {
"cause_code": "573"
}
},
{
"range": {
"x_event_timestamp": {
"gt": "2016-07-13T13:51:03.749Z",
"lt": "2016-07-16T13:51:03.749Z"
}
}
}
]
}
},
"from": 0,
"size": 10,
"sort": [
{
"x_event_timestamp": {
"order": "desc",
"ignore_unmapped": true
}
}
]
}