I try to make a query with filtering but it fails, Bad Request comes as reponse
{
"query":{
"filtered":{
"query":{
"logdate":{
"gte":"01-01-2014"
}
}
}
}
}
I search documentation online and see it works as same part of my code but something in there is not true that I cant figure out
you seem "query" tag in filter comment in online documentation of elasticsearch or elsewhere ? lol never go there. Use "filter" tag in filtered query and also you must add "range" field. here This is the true form of your query
{
"query":{
"filtered":{
"filter":{
"range":{
"logdate":{
"gte":"01-01-2014"
}
}
}
}
}
}
Related
The problem:
Our log data has 27-34 million entries for a /event-heartbeat.
I need to filter those entries out to see just viable log messages in Kibana.
Using Kibana filters with wildcards does not work. Thus, I think I will have to write QueryDSL to do it in version 6.4.2 Elasticsearch to get it to filter out the event heart beats.
I have been looking and I can't find any good explanations on how to do an anti-pattern match so to search for all entries that don't have /event-heartbeat in the message.
Here is the log message:
#timestamp:
June 14th 2019, 12:39:09.225
host.name:
iislogs-production
source:
C:\inetpub\logs\LogFiles\W3SVC5\u_ex19061412.log
offset:
83,944,181
message:
2019-06-14 19:39:06 0.0.0.0 GET /event-heartbeat id=Budrug2UDw 443 - 0.0.0.0 - - 200 0 0 31
prospector.type:
log
input.type:
log
beat.name:
iislogs-production
beat.hostname:
MYHOSTNAME
beat.version:
6.4.2
_id:
yg6AV2sB0_n
_type:
doc
_index:
iislogs-production-6.4.2-2019.06.14
_score:
-
Message is a keyword field so I can do painless scripting on it.
I've used Lucene syntax
NOT message: "*/event-heartbeat*"
This is the anti pattern the kibana filter generates.
{
"query": {
"bool": {
"should": [
{
"match_phrase": {
"message": "*event-heartbeat*"
}
}
],
"minimum_should_match": 1
}
}
}
I've tried the proposed solution below by huglap. I also adjusted my query based on his comment and tried two ways. I adjust it with the term word instead of match and tried both ways because the field technically is a keyword so I could do painless scripting on it. The query still returns event heartbeat log entries.
Here are the two queries I tried from the below proposed solution:
GET /iislogs-production-*/_search
{
"query":{
"bool":{
"must":{
"match_all":{
}
},
"filter":{
"bool":{
"must_not":[
{
"term":{
"message.whitespace":"event-heartbeat"
}
}
]
}
}
}
}
}
GET /iislogs-production-*/_search
{
"query":{
"bool":{
"must":{
"match_all":{
}
},
"filter":{
"bool":{
"must_not":[
{
"match":{
"message.whitespace":"event-heartbeat"
}
}
]
}
}
}
}
}
Index Mapping:
https://gist.github.com/zukeru/907a9b2fa2f0d6f91a532b0865131988
Have you thought about a 'must_not' bool query?
Since your going for the whole set and not really caring about shaping the relevancy function, I suggest the use of a filter instead of a query. You'll get better performance.
{
"query":{
"bool":{
"must":{
"match_all":{
}
},
"filter":{
"bool":{
"must_not":[
{
"match":{
"message.whitespace":"event-heartbeat"
}
}
]
}
}
}
}
}
This example assumes you are querying against a text field, thus the use of a 'match' query instead of a 'term' one.
You also need to make sure that the field is analyzed (really tokenized) according to your goals. The fact that you have a dash in your query term will create problems if you're using a simple or even a standard analyser. Elasticsearch would break the term in two words. You could try the whitespace analyser on that one or just remove the dash from the query.
GET qnaindexfinal/_search
{
"query":{
"bool":{
"must":[
{
"common":{
"question.questionText":{
"query":"showrroom",
"cutoff_frequency":0.001
}
}
}
],
"filter":[
{
"term":{
"modelId":{
"value":78
}
}
}
]
}
}
}
Please help me with the above dsl.
With queries in elasticsearch it's best to break things down a little.
common same as a term but for more than one keyword
term value == 78
must in this context is checking that the only documents returned match both common and term.
I want to access a document in my index directly using its _id field and I want to highlight a word in messageTextfield, for this, I created below query but highlight attribute does not appear in result response.
{
"query":{
"term":{
"_id": "1006382869737"
}
},
"highlight" : {
"tags_schema" : "styled",
"fields" : {
"messageText" : {
"highlight_query":{
"term": {
"messageText":"car"
}
}
}
}
}
}
I'm sure that car is occurred in messageText field for document by Id 1006382869737. so I'm sure highlights must exist in response, but it's not.
if it is important I'm using 2.3.4 version of elasticsearch. and query has been created according to this documentation. I'm not sure what rescore_query is in this documentation, if it is important please tell me how to edit my query, else give me another suggestion.
tnx :)
Additional Information
I also try this below query:
{
"query":{
"bool": {
"must":[
{
"term":{
"_id": "1006382869737"
}
},
{
"term": {
"messageText":"car"
}
}
]
}
},
"highlight" : {
"fields" : {
"messageText" : {}
}
}
}
but it causes no document hits.
I found solution:
My data comes from a server that I haven't access to its code. After many debugging I figure out that in some cases server send the text data in another field named caption.
Unfortunately The server has not a good documentation, that causes this problem.
Finally I find problem and add a more highlight_query for caption field and it's work fine now.
I am very new to Elasticsearch and I have to perform the following query:
GET book-lists/book-list/_search
{
"query":{
"filtered":{
"filter":{
"bool":{
"must":[
{
"term":{
"title":"Sociology"
}
},
{
"term":{
"idOwner":"17xxxxxxxxxxxx45"
}
}
]
}
}
}
}
}
According to the Elasticsearch API, it is equivalent to pseudo-SQL:
SELECT document
FROM book-lists
WHERE title = "Sociology"
AND idOwner = 17xxxxxxxxxxxx45
The problem is that my document looks like this:
{
"_index":"book-lists",
"_type":"book-list",
"_id":"AVBRSvHIXb7carZwcePS",
"_version":1,
"_score":1,
"_source":{
"title":"Sociology",
"books":[
{
"title":"The Tipping Point: How Little Things Can Make a Big Difference",
"isRead":true,
"summary":"lorem ipsum",
"rating":3.5
}
],
"numberViews":0,
"idOwner":"17xxxxxxxxxxxx45"
}
}
And the Elasticsearch query above doesn't return anything.
Whereas, this query returns the document above:
GET book-lists/book-list/_search
{
"query":{
"filtered":{
"filter":{
"bool":{
"must":[
{
"term":{
"numberViews":"0"
}
},
{
"term":{
"idOwner":"17xxxxxxxxxxxx45"
}
}
]
}
}
}
}
}
This makes me suspect that the fact that "title" is the same name for the two fields is for something.
Is there a way to fix this without having to rename any of the fields. Or am I missing it somewhere else?
Thanks for anyone trying to help.
Your problem is described in the documentation.
I suspect that you don't have any explicit mapping on your index, which means elasticsearch will use dynamic mapping.
For string fields, it will pass the string through the standard analyzer which lowercases it (among other things). This is why your query doesn't work.
Your options are:
Specify an explicit mapping on the field so that it isn't analyzed before storing in the index (index: not_analyzed).
Clean your term query before sending it to elasticsearch (in this specific query lowercasing will work, but note that the standard analyzer also does other things like remove stop words, so depending on the title you may still have issues).
Use a different query type (e.g., query_string instead of term which will analyze the query before running it).
Looking at the sort of data you are storing you probably need to specify an explicit not_analyzed mapping.
For option three your query would look something like this:
{
"query":{
"filtered":{
"filter":{
"bool":{
"must":[
{
"query_string":{
"fields": ["title"],
"analyzer": "standard",
"query": "Sociology"
}
},
{
"term":{
"idOwner":"17xxxxxxxxxxxx45"
}
}
]
}
}
}
}
}
Note that the query_string query has special syntax (e.g., OR and AND are not treated as literals) which means you have to be careful what you give it. For this reason explicit mapping with a term filter is probably more appropriate for your use case.
I have described this issue in this blog.
The issue is coming due to default tokenization in Elasticsearch.
In the same , I have outlined 2 solutions.
One is enabling not_analyzed flag on the required field and other is to use keyword tokenizer.
To expand on solarissmoke's solution, while the contents of that field will be passed through the standard analyzer, your query will not. If you refer to the Elasticsearch documentation on the term query, you will see that term queries are not analyzed.
The match query is probably more appropriate for your case. What you query will be analyzed in the same way as the contents of the title field by default. The query_string query brings a lot more to the table and you should review the documentation if you plan on using that.
So again pretty much what you had with the small tweak:
GET book-lists/book-list/_search
{
"query":{
"filtered":{
"filter":{
"bool":{
"must":[
{
"match":{
"title":"Sociology"
}
},
{
"term":{
"idOwner":"17xxxxxxxxxxxx45"
}
}
]
}
}
}
}
}
It is important to note passing lowercase version of the terms to the term query (hack - does not seem like a good idea given what solarissmoke describe about the other features of the Standard analyzer like the stop filter), using the query_string query, or using the match query is still very different from the SQL query you described:
SELECT document
FROM book-lists
WHERE title = "Sociology"
AND idOwner = 17xxxxxxxxxxxx45
With those Elasticsearch queries, you can match records where idOwner might be the same but title might be something like "Another Sociology Title" which is different from what you would expect with that SQL. Here is some great stuff from the documentation and another stackoverflow post that will elaborate on what was going on, where term queries and filters are appropriate, and getting exact matches:
Elasticsearch : Finding Exact Values
Stackoverflow : Exact (not substring) matching in Elasticsearch
The title says it all: Is there a difference between using a type filter(s) vs specifying doc_type(s) in the url/api, ie
...
"filter" : {
'type': {
'value': 'my_doc_type'
}
}
....
vs
GET /my_index/my_doc_type/_search
EDIT:
I meant as a filter in a filtered query; however, I'm going to leave the question as is, so that there's more information to be gleaned from the responses.
Yes.
Specifying the type in the URL will apply before the query is run.
Specifying the type as a filter will apply after the query is run.
As a side note, using a filtered query instead of a filter will function similarly to specifying the type in the URL, as it will apply before the query is run.
Do be careful with the functionality of Filters vs Filtered queries if you are using facets as they function differently on facets.
Filtered query:
{
"query":{
"filtered":{
"query":{
"term":{"name":"franky"}
},
"filter":{
"term":{"age":73}
}
}
}
}
Filter:
{
"query":{
"term":{"name":"franky"}
},
"filter":{
"term":{"age":73}
}
}