I am using logstash to store logs in elasticsearch database. I want to get logs having a particular severitylabel and are between certain time stamps and matches to some specific message. The curl query I wrote is :
curl -XPOST 'localhost:9200/logstash-2015.06.19/_search/?pretty' -d '{
"query": {
"filtered": {
"query": {
"bool": {
"must": [
{
"match": {
"#message": "session"
}
}
]
}
},
"filter": {
"bool": {
"must": [
{
"range": {
"#timestamp": {
"gte": "2015-06-19T10:11:44.000Z",
"lte": "2015-06-19T11:11:44.000Z"
}
}
},
{
"term": {
"#app": "sparta"
}
},
{
"terms": {
"#severityLabel": [
"INFO",
"WARN",
"ERROR",
"FATAL",
"OFF"
]
}
}
]
}
}
}
} } '
It always shows zero documents, matched. I am using term filter as a sibling of terms filter, is that a problem?
Related
I have ES documents structured in a flat data structure using the nested data type, as they accept arbitrary JSON that we don't control, and we need to avoid a mapping explosion. Here's an example document:
{
"doc_flat":[
{
"key":"timestamp",
"type":"date",
"key_type":"timestamp.date",
"value_date":[
"2023-01-20T12:00:00Z"
]
},
{
"key":"status",
"type":"string",
"key_type":"status.string",
"value_string":[
"warning"
]
},
... more arbitrary fields ...
],
}
I've figured out how to query this nested data set to find matches on this arbitrary nested data, using a query such as:
{
"query": {
"nested": {
"path": "doc_flat",
"query": {
"bool": {
"must": [
{"term": {"doc_flat.key": "status"}},
{"term": {"doc_flat.value_string": "warning"}}
]
}
}
}
}
}
And I figured out how to find documents matching a particular date range:
{
"query": {
"nested": {
"path": "doc_flat",
"query": {
"bool": {
"must": [
{"term": {"doc_flat.key": "timestamp"}},
{
"range": {
"doc_flat.value_date": {
"gte": "2023-01-20T00:00:00Z",
"lte": "2023-01-21T00:00:00Z"
}
}
}
]
}
}
}
}
}
But I'm struggling to combine these two queries together, in order to search for documents that have a nested documents which match these two conditions:
a doc_flat.key of status, and a doc_flat.value_string of warning
a doc_flat.key of timestamp, and a doc_flat.value_date in a range
Obviously I can't just shove the second set of query filters into the same must array, because then no documents will match. I think I need to go "one level higher" in my query and wrap it in another bool query? But I can't get my head around how that would look.
You tried two nested inside Bool query?
{
"query": {
"bool": {
"filter": [
{
"nested": {
"path": "doc_flat",
"query": {
"bool": {
"must": [
{
"term": {
"doc_flat.key": "timestamp"
}
},
{
"range": {
"doc_flat.value_date": {
"gte": "2023-01-20T00:00:00Z",
"lte": "2023-01-21T00:00:00Z"
}
}
}
]
}
}
}
}
],
"must": [
{
"nested": {
"path": "doc_flat",
"query": {
"bool": {
"must": [
{
"term": {
"doc_flat.key": "status"
}
},
{
"term": {
"doc_flat.value_string": "warning"
}
}
]
}
}
}
}
]
}
}
}
I have to combine two filters to match requirements:
- a specific list of values in r.status field
- one of the multiple text fields contains the value.
Result query (with using Nest, but it doesn't matter) looks like:
{
"query": {
"bool": {
"filter": [
{
"bool": {
"must": [
{
"term": {
"isActive": {
"value": true
}
}
},
{
"nested": {
"query": {
"bool": {
"must": [
{
"terms": {
"r.status": [
"VALUE_1",
"VALUE_2",
"VALUE_3"
]
}
},
{
"bool": {
"should": [
{
"match": {
"r.g.firstName": {
"type": "phrase",
"query": "SUBSTRING_VALUE"
}
}
},
{
"match": {
"r.g.lastName": {
"type": "phrase",
"query": "SUBSTRING_VALUE"
}
}
}
]
}
}
]
}
},
"path": "r"
}
}
]
}
}
]
}
}
}
Also tried with multi_match query:
{
"query": {
"bool": {
"filter": [
{
"bool": {
"must": [
{
"term": {
"isActive": {
"value": true
}
}
},
{
"nested": {
"query": {
"bool": {
"must": [
{
"terms": {
"r.status": [
"VALUE_1",
"VALUE_2",
"VALUE_3"
]
}
},
{
"multi_match": {
"query": "SUBSTRING_VALUE",
"fields": [
"r.g.firstName",
"r.g.lastName"
]
}
}
]
}
},
"path": "r"
}
}
]
}
}
]
}
}
}
FirstName and LastName are configured in index mappings as text:
"firstName": {
"type": "text"
},
"lastName": {
"type": "text"
}
Elastic gives a lot of full-text search options: multi_match, phrase, wildcards etc. But all of them fail in my case looking a sub-string in my text fields. (terms query and isActive one work well, I just tried to run only them).
What options do I have also or maybe where I made a mistake?
UPD: Combined wildcards worked for me, but such query looks ugly. Looking for a more elegant solution.
The elasticsearch way is to use ngram tokenizer.
The ngram analyzer will split your terms with a sliding window. For example, the input "Hello World" will generate the following terms:
Hel
Hell
Hello
ell
ello
...
Wor
World
orl
...
You can configure the minimum and maximum size of the sliding window (in the example the minimum size is 3). Once the sub terms are generated you can use a match query an the subfield.
Another point, it is weird to use must within a filter. If you are interested in the score, you should always use must otherwise use filter. Read this article for a good understanding.
Hi I want to join two queries in one in elasticsearch, but I don't know how to do it: I think I should do an aggregation but I don't know very clear how to do it. Could you help me? My ES version is 5.1.2.
First filter by status and name:
POST test_lite/_search
{
"aggs": {
"filtered": {
"filter": {
"bool": {
"must": [
{
"match": {
"STATUS": "Now"
}
},
{
"match": {
"NAME": "PRUDENTL"
}
}
]
}
}
}
}
}
Look for in the filtered records for the word filtered in description:
POST /test_lite/_search
{
"query": {
"wildcard" : { "DESCRIPTION" : "*english*" }
}
}
The only query needed is:
POST test_lite/_search
{
"query": {
"bool": {
"must": [
{
"match": {
"STATUS": "Now"
}
},
{
"match": {
"NAME": "PRUDENTL"
}
},
{"wildcard" : { "DESCRIPTION" : "*english*" }}
]
}
}
}
I have an index with some documents having a field named "access_type" . It can have 2 values, either "faculty" or "students".
For the documents with "faculty" as the value for "access_type", there will be another field called "faculties" which is a list of faculty name.
So an example document would look like below:
{
"access_type": "faculty",
"faculties": [
"facultyId1",
"facultyId2",
"facultyId3"
]
}
Now if we have two inputs say one is for the access_type and another is for the faculties.
If I get the following input "faculty" and "facultyId4" . First I need to filter out all the documents matching the access type "faculty" and then in the resulting results the "facuultyId4" should search against the field "faculties". Since the "facultyId4" is not in the above document,it should not be considered a hit.
How can I implement this as an elasticsearch query?
POST http://your.elastic.host:9200/index/type/_search
{
"query": {
"filtered": {
"query": {
"match_all": {}
},
"filter": {
"bool": {
"must": [
{
"term": {
"access_type": "faculty"
}
},
{
"term": {
"faculties": "facultyId4"
}
}
]
}
}
}
}
}
Hope this will for work.
GET index/type/_search
{
"query": {
"filtered": {
"filter": {
"and": {
"filters": [
{
"query": {
"match": {
"access_type": "faculty"
}
}
},
{
"query": {
"match": {
"faculties": "facultyId4"
}
}
}
]
}
}
}
}
}
I am trying to query elasticsearch for all logs that have a #timestamp within a certain range. This is being done by a $.get() request using jQuery. My attempts so far have resulted in queries not within the time range. I am listing the logs on my own HTML page and although I have tried different query messages I always get one of two responses, one with times starting at 4:25 and the other with times starting at 6:39. I don't know if this is useless information but I thought it could be useful in understanding the nature of the problem.
So this is the message I have written myself.
{
"query" : {
"filtered": {
"query" : { "match_all" : {}},
"filter": {
"range" : {
"#timestamp": {
"from" : 1393580102248,
"to" : 1393583702248
}
}
}
}
}
}
We are also using Kibana (working) and I have tried copying and pasting the same exact query Kibana uses, with the SAME results as my query above.
{
"query": {
"filtered": {
"query": {
"bool": {
"should": [
{
"query_string": {
"query": "*"
}
}
]
}
},
"filter": {
"bool": {
"must": [
{
"match_all": {}
},
{
"range": {
"#timestamp": {
"from": 1393580102248,
"to": 1393583702248
}
}
},
{
"bool": {
"must": [
{
"match_all": {}
}
]
}
}
]
}
}
}
},
"highlight": {
"fields": {},
"fragment_size": 2147483647,
"pre_tags": [
"#start-highlight#"
],
"post_tags": [
"#end-highlight#"
]
},
"size": 500,
"sort": [
{
"#timestamp": {
"order": "desc"
}
}
]
}
What am I doing wrong? I can change the time range as well and I still get the same results.