I'm trying to create a query to count instances where two conditions are met over the last day.
This query shows the count for the two conditions, but when I try to add a range in, it seems to match all documents:
GET logstash-*/_count
{
"query": {
"bool": {
"should": [
{
"match": {
"rawmsg": {
"query": "Could not send Message.",
"type": "phrase"
}
}
},
{
"match": {
"stack_trace": {
"query": "*WebServiceException*",
"type": "phrase"
}
}
}
]
}
}
}
Here's how I'm trying to add the date range:
GET logstash-*/_count
{
"query": {
"bool": {
"should": [
{
"match": {
"rawmsg": {
"query": "Could not send Message.",
"type": "phrase"
}
}
},
{
"match": {
"stack_trace": {
"query": "*WebServiceException*",
"type": "phrase"
}
}
},
{
"range" : {
"#timestamp" : {
"gte" : "now-1d/d",
"lt" : "now/d"
}
}
}
]
}
}
}
I ended up finding two ways of accomplishing what I needed:
GET logstash-*/tcp_input/_count?q=stack_trace: *WebServiceException* AND rawmsg: "Could not send Message" AND #timestamp: [ now-30d TO now ]
and
GET logstash-*/_count
{
"query": {
"query_string": {
"query": """stack_trace: *WebServiceException* AND rawmsg: "Could not send Message" AND #timestamp: [ now-3d TO now]""",
"analyze_wildcard": true
}
}
}
You are using a should clause instead of a must clause, which effectively combines your conditions with an OR instead of an AND.
Also the timestamp should be now/d-1d for the range query.
Related
Query: Get employee name "Mahesh" whose id is "200" and joining datetime is in a given date range and his epf status must be either 'NOK' or 'WRN'. (Possible values of epf_status are {OK,NOK,WRN,CANCELLED}.
I have written the following query, that matches epf_status also with OK, CANCELLED, but it must only match when epf_status is either 'NOK' or 'WRN'. What else do I need to change to make it work, as required?
GET myindex01/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"empname": { "query": "Mahesh", "operator": "AND" }
}
},
{
"match": {
"empid": { "query": "200", "operator": "AND" }
}
},
{
"range": {
"joining_datetime": {
"gte": "2020-01-01T00:00:00",
"lte": "2022-06-24T23:59:59"
}
}
}
],
"should": [
{ "match": { "epf_status": "NOK" } },
{ "match": { "epf_status": "WRN" } }
]
}
}
}
SAMPLE DATA:
{"Mahesh","200","2022-04-01","OK"}
{"Mahesh","200","2022-04-01","NOK"}
{"Mahesh","200","2022-04-01","WRN"}
{"Mahesh","200","2022-04-01","CANCELLED"}
REQUIRED OUTPUT:
{"Mahesh","200","2022-04-01","NOK"}
{"Mahesh","200","2022-04-01","WRN"}
Tldr;
You could be using the terms query for that I believe.
Returns documents that contain one or more exact terms in a provided field.
To solve
GET myindex01/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"empname": { "query": "Mahesh", "operator": "AND" }
}
},
{
"match": {
"empid": { "query": "200", "operator": "AND" }
}
},
{
"range": {
"joining_datetime": {
"gte": "2020-01-01T00:00:00",
"lte": "2022-06-24T23:59:59"
}
}
}
],
"should": [
{ "terms": { "epf_status": ["NOK", "WRN"] } }
]
}
}
}
I have built two queries in ElasticSearch to get the counts for each error message. for example, the first query is to get how many error messages related to "was not found" error
GET /logstash*/_search
{
"query": {
"bool": {
"filter": {
"bool": {
"must": [
{
"match": {
"kubernetes.pod_name": "api"
}
},
{
"match": {
"log": "error"
}
},
{
"match": {
"log": {
"query": "was not found",
"operator": "and"
}
}
},
{
"range": {"#timestamp": {
"time_zone": "CET",
"gt": "now-7d",
"lte": "now"}}
}
]
}
}
}
},
"aggs" : {
"type_count" : {
"value_count" : {
"script" : {
"source" : "doc['log.keyword'].value"
}
}
}
}
}
The second query is to get the count of error messages related to "Duplicate Entry" error
GET /logstash*/_search
{
"query": {
"bool": {
"filter": {
"bool": {
"must": [
{
"match": {
"kubernetes.pod_name": "api"
}
},
{
"match": {
"log": "error"
}
},
{
"match": {
"log": {
"query": "Duplicate entry",
"operator": "and"
}
}
},
{
"range": {"#timestamp": {
"time_zone": "CET",
"gt": "now-7d",
"lte": "now"}}
}
]
}
}
}
},
"aggs" : {
"type_count" : {
"value_count" : {
"script" : {
"source" : "doc['log.keyword'].value"
}
}
}
}
}
My boss really wants me to combine these individual query into a one big query, then get the list of counts for each error messages in one output. Since we have a lot of error messages, which means we have to write each query for each error message, then we have to run each query to get the counts. Is there a way I can click one run to get the list of counts?
I have been trying use query string query and looking for solutions on either Stack Overflow and Documentation. However, there is no luck
You can use filter aggregation along with the value_count aggregation to combine these two queries. In both the queries, out of the 4 queries inside must clause only one differs. You can take this out and combine them with the two filter aggregations as below:
{
"query": {
"bool": {
"filter": {
"bool": {
"must": [
{
"match": {
"kubernetes.pod_name": "api"
}
},
{
"match": {
"log": "error"
}
},
{
"range": {
"#timestamp": {
"time_zone": "CET",
"gt": "now-7d",
"lte": "now"
}
}
}
]
}
}
}
},
"aggs": {
"not_found_count": {
"filter": {
"match": {
"log": {
"query": "was not found",
"operator": "and"
}
}
},
"aggs": {
"count": {
"value_count": {
"script": {
"source": "doc['log.keyword'].value"
}
}
}
}
},
"duplicate_entry_count": {
"filter": {
"match": {
"log": {
"query": "Duplicate entry",
"operator": "and"
}
}
},
"aggs": {
"count": {
"value_count": {
"script": {
"source": "doc['log.keyword'].value"
}
}
}
}
}
}
}
Hi I want to join two queries in one in elasticsearch, but I don't know how to do it: I think I should do an aggregation but I don't know very clear how to do it. Could you help me? My ES version is 5.1.2.
First filter by status and name:
POST test_lite/_search
{
"aggs": {
"filtered": {
"filter": {
"bool": {
"must": [
{
"match": {
"STATUS": "Now"
}
},
{
"match": {
"NAME": "PRUDENTL"
}
}
]
}
}
}
}
}
Look for in the filtered records for the word filtered in description:
POST /test_lite/_search
{
"query": {
"wildcard" : { "DESCRIPTION" : "*english*" }
}
}
The only query needed is:
POST test_lite/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"STATUS": "Now"
}
},
{
"match": {
"NAME": "PRUDENTL"
}
},
{"wildcard" : { "DESCRIPTION" : "*english*" }}
]
}
}
}
I'm trying to run an exact match query in ES
in MYSQL my query would be:
SELECT * WHERE `content_state`='active' AND `author`='bob' AND `title` != 'Beer';
I looked at the ES docs here:
https://www.elastic.co/guide/en/elasticsearch/guide/current/_finding_exact_values.html
and came up with this:
{
"from" : '.$offset.', "size" : '.$limit.',
"filter": {
"and": [
{
"and": [
{
"term": {
"content_state": "active"
}
},
{
"term": {
"author": "bob"
}
},
{
"not": {
"filter": {
"term": {
"title": "Beer"
}
}
}
}
]
}
]
}
}
but my results are still coming back with the title = Beer, it doesn't seem to be excluding the titles that = Beer.
did I do something wrong?
I'm pretty new to ES
I figured it out, I used this instead...
{
"from" : '.$offset.', "size" : '.$limit.',
"query": {
"bool": {
"must": [
{
"query_string": {
"default_field": "content_state",
"query": "active"
}
},
{
"query_string": {
"default_field": "author",
"query": "bob"
}
}
],
"must_not": [
{
"query_string": {
"default_field": "title",
"query": "Beer"
}
}
]
}
}
}
Query String Query is a pretty good concept to handle various relationship between search criteria. Have a quick look into Query string query syntax to understand in detail about this concept
{
"query": {
"query_string": {
"query": "(content_state:active AND author:bob) AND NOT (title:Beer)"
}
}
}
Filters are supposed to work on exact values, if you had defined your mapping in a manner where title was a non-analyzed field, your previous attempt ( with filters) would have worked as well.
{
"mappings": {
"test": {
"_all": {
"enabled": false
},
"properties": {
"content_state": {
"type": "string"
},
"author": {
"type": "string"
},
"title": {
"type": "string",
"index": "not_analyzed"
}
}
}
}
}
How can i create Elasticsearch curl query to get the field value which are not null and not empty(""),
Here is the mysql query:
select field1 from mytable where field1!=null and field1!="";
A null value and an empty string both result in no value being indexed, in which case you can use the exists filter
curl -XGET 'http://127.0.0.1:9200/test/test/_search?pretty=1' -d '
{
"query" : {
"constant_score" : {
"filter" : {
"exists" : {
"field" : "myfield"
}
}
}
}
}
'
Or in combination with (eg) a full text search on the title field:
curl -XGET 'http://127.0.0.1:9200/test/test/_search?pretty=1' -d '
{
"query" : {
"filtered" : {
"filter" : {
"exists" : {
"field" : "myfield"
}
},
"query" : {
"match" : {
"title" : "search keywords"
}
}
}
}
}
'
As #luqmaan pointed out in the comments, the documentation says that the filter exists doesn't filter out empty strings as they are considered non-null values.
So adding to #DrTech's answer, to effectively filter null and empty string values out, you should use something like this:
{
"query" : {
"constant_score" : {
"filter" : {
"bool": {
"must": {"exists": {"field": "<your_field_name_here>"}},
"must_not": {"term": {"<your_field_name_here>": ""}}
}
}
}
}
}
On elasticsearch 5.6, I have to use command below to filter out empty string:
GET /_search
{
"query" : {
"regexp":{
"<your_field_name_here>": ".+"
}
}
}
Wrap a Missing Filter in the Must-Not section of a Bool Filter. It will only return documents where the field exists, and if you set the "null_value" property to true, values that are explicitly not null.
{
"query":{
"filtered":{
"query":{
"match_all":{}
},
"filter":{
"bool":{
"must":{},
"should":{},
"must_not":{
"missing":{
"field":"field1",
"existence":true,
"null_value":true
}
}
}
}
}
}
}
You can do that with bool query and combination of must and must_not like this:
GET index/_search
{
"query": {
"bool": {
"must": [
{"exists": {"field": "field1"}}
],
"must_not": [
{"term": {"field1": ""}}
]
}
}
}
I tested this with Elasticsearch 5.6.5 in Kibana.
The only solution here that worked for me in 5.6.5 was bigstone1998's regex answer. I'd prefer not to use a regex search though for performance reasons. I believe the reason the other solutions don't work is because a standard field will be analyzed and as a result have no empty string token to negate against. The exists query won't help on it's own either since an empty string is considered non-null.
If you can't change the index the regex approach may be your only option, but if you can change the index then adding a keyword subfield will solve the problem.
In the mappings for the index:
"myfield": {
"type": "text",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
}
Then you can simply use the query:
{
"query": {
"bool": {
"must": {
"exists": {
"field": "myfield"
}
},
"must_not": {
"term": {
"myfield.keyword": ""
}
}
}
}
}
Note the .keyword in the must_not component.
You can use not filter on top of missing.
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"not": {
"filter": {
"missing": {
"field": "searchField"
}
}
}
}
}
}
Here's the query example to check the existence of multiple fields:
{
"query": {
"bool": {
"filter": [
{
"exists": {
"field": "field_1"
}
},
{
"exists": {
"field": "field_2"
}
},
{
"exists": {
"field": "field_n"
}
}
]
}
}
}
You can use a bool combination query with must/must_not which gives great performance and returns all records where the field is not null and not empty.
bool must_not is like "NOT AND" which means field!="", bool must exist means its !=null.
so effectively enabling: where field1!=null and field1!=""
GET IndexName/IndexType/_search
{
"query": {
"bool": {
"must": [{
"bool": {
"must_not": [{
"term": { "YourFieldName": ""}
}]
}
}, {
"bool": {
"must": [{
"exists" : { "field" : "YourFieldName" }
}]
}
}]
}
}
}
ElasticSearch Version:
"version": {
"number": "5.6.10",
"lucene_version": "6.6.1"
}
ES 7.x
{
"_source": "field",
"query": {
"bool": {
"must": [
{
"exists": {
"field":"field"
}
}
],
"must_not": [
{
"term": {
"field.keyword": {
"value": ""
}
}
}
]
}
}
}
We are using Elasticsearch version 1.6 and I used this query from a co-worker to cover not null and not empty for a field:
{
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"bool": {
"must": [
{
"exists": {
"field": "myfieldName"
}
},
{
"not": {
"filter": {
"term": {
"myfieldName": ""
}
}
}
}
]
}
}
}
}
}
You need to use bool query with must/must_not and exists
To get where place is null
{
"query": {
"bool": {
"must_not": {
"exists": {
"field": "place"
}
}
}
}
}
To get where place is not null
{
"query": {
"bool": {
"must": {
"exists": {
"field": "place"
}
}
}
}
}
Elastic search Get all record where condition not empty.
const searchQuery = {
body: {
query: {
query_string: {
default_field: '*.*',
query: 'feildName: ?*',
},
},
},
index: 'IndexName'
};