In my elastic search index, each document will have two date fields createdDate and modifiedDate. I'm trying to add a filter in kibana to fetch the documents where the modifiedDate is greater than createdDate. How to create this filter in kibana?
Tried Using below query instead of greater than it is considering as gte and fetching all records
GET index/_search
{
"query": {
"bool": {
"filter": {
"script": {
"script" : {
"inline" : "doc['modifiedTime'].value.getMillis() > doc['createdTime'].value.getMillis()",
"lang" : "painless"
}
}
}
}
}
}
There are a few options.
Option A: The easiest and most performant one is to store the difference of the two fields inside a new field of your document, e.g.
{
"createDate": "2022-01-11T12:34:56Z",
"modifiedDate": "2022-01-11T12:34:56Z",
"diffMillis": 0
}
{
"createDate": "2022-01-11T12:34:56Z",
"modifiedDate": "2022-01-11T12:35:58",
"diffMillis": 62000
}
Then, in Kibana you can query on diffMillis > 0 and figure out all documents that have been modified after their creation.
Option B: You can use a script query
GET index/_search
{
"query": {
"bool": {
"filter": {
"script": {
"script": """
return doc['createdDate'].value.millis < doc['modifiedDate'].value.millis;
"""
}
}
}
}
}
Note: depending on the amount of data you have, this option can potentially have disastrous performance, because it needs to be evaluated on ALL of your documents.
Option C: If you're using ES 7.11+, you can use runtime fields directly from the Kibana Discover view.
You can use the following script in order to add a new runtime field (e.g. name it diffMillis) to your index pattern:
emit(doc['modifiedDate'].value.millis - doc['createdDate'].value.millis)
And then you can add the following query into your search bar
diffMillis > 0
Related
We are using elastic version 7.10.2. I want to compare two fields from a same document.Scripting is disabled in my organization.
Kindly help in building below query without using script.
Here my query is : nickname is null or nickname is empty or nickname is equal to firstname.
Hard part is how to build query to get the records which have nickname is equal to firstname
Relevant script query to be converted to normal query :
{
"query": {
"bool": {
"must": [{
"script": {
"script": {
"inline": "doc['nickname.keyword'].value==null || doc['nickname.keyword'].value =='' || doc['nickname.keyword'].value == doc['firstname.keyword'].value",
"lang": "painless",
}
}
}]
}
}
}
I see you are already comparing the nickname.keyword to your firstname also mentioned this is the hard part, for this why you need a script, you can simply use the search query on this keyword field and get the result you want.
You can use below term query for it.
{
"query": {
"term": {
"nickname.keyword": {
"value": "your-nickname", // provide your nickname as value
}
}
}
}
I'm trying to write query with custom script in elasticsearch:
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-script-filter.html#query-dsl-script-filter
https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting.html.
This is useful when you need to compare two document fields.
Everything worked fine, until I decide to use special document field (ex: _id, _uid, etc). The query always returns empty results and there is no errors if I use it like this: doc['_id'].value.
So how to use, for example, "_id" field of a document in a custom script?
The _id is indexed in the uid field, using this format: type#id.
So, your script should look like this (for a type called my_type and an ID of 1):
{
"query": {
"filtered": {
"filter": {
"script" : {
"script" : "doc['_uid'].value == 'my_type#1'"
}
}
}
}
}
A more elaborate solution, to take out the id ES-way is like this:
{
"query": {
"filtered": {
"filter": {
"script": {
"script": "org.elasticsearch.index.mapper.Uid.splitUidIntoTypeAndId(new org.apache.lucene.util.BytesRef(doc['_uid'].value))[1].utf8ToString() == '1'"
}
}
}
}
}
where org.elasticsearch.index.mapper.Uid.splitUidIntoTypeAndId(new org.apache.lucene.util.BytesRef(doc['_uid'].value))[1] is the id and org.elasticsearch.index.mapper.Uid.splitUidIntoTypeAndId(new org.apache.lucene.util.BytesRef(doc['_uid'].value))[0] is the type.
My data has a field localities which is an array of strings.
"localities": [
"Mayur Vihar Phase 1",
"Paschim Vihar",
"Rohini",
"",
"Laxmi Nagar",
"Vasant Vihar",
"Dwarka",
"Karol Bagh",
"Inderlok" ]
What query should I write to filter the documents by a specific locality such as "Rohini"?
A simple match query will be enough (if you don't know the mapping of your localities field).
POST <your index>/_search
{
"query": {
"match": {
"localities": "Rohini"
}
}
}
If the localities field is set as a string type and index as not_analyzed, the best way to query this is to use a term filter, wrapped in a filtered query (you can't use directly filters) :
POST <your index>/_search
{
"query": {
"filtered": {
"filter": {
"term": {
"localities": "Rohini"
}
}
}
}
}
If you doesn't need the score, the second solution is the way to go as filters doesn't compute score, are faster and cached.
Check the documentation for information about analysis which is a very important subject in ElasticSearch, heavily influencing the way you query.
POST /_search
{
"query": {
"match": {
"localities": "Rohini"
}
}
}
Or you can simply query:
GET /_search?q=localities:Rohini
How can I query for, or filter for, one field doesn't equal another field? i.e. where document1.city1.name not equal document1.city2.name.
Some version of this?
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-script-fields.html
Yes , you will need to use script filter to achieve this
{
"bool": {
"filter": {
"script": {
"script": "doc['field1'].value != doc['field2'].value"
}
}
}
}
You can find more information here
I'm using Elasticsearch and Nest to create a query for documents within a specific time range as well as doing some filter facets. The query looks like this:
{
"facets": {
"notfound": {
"query": {
"term": {
"statusCode": {
"value": 404
}
}
}
}
},
"filter": {
"bool": {
"must": [
{
"range": {
"time": {
"from": "2014-04-05T05:25:37",
"to": "2014-04-07T05:25:37"
}
}
}
]
}
}
}
In the specific case, the total hits of the search is 21 documents, which fits the documents within that time range in Elasticsearch. But the "notfound" facet returns 38, which fits the total number of ErrorDocuments with a StatusCode value of 404.
As I understand the documentation, facets collects data from withing the search. In this case, the "notfound" facet should never be able to return a count higher that 21.
What am I doing wrong here?
There's a distinct difference between filter/query/filtered_query/facet filter which is good to know.
Top level filter
{
filter: {}
}
This acts as a post-filter, meaning it will filter the results after the query phase has ended. Since facets are part of the query phase filters do not influence the documents that are facetted over. Filters do not alter score and are therefor very cacheable.
Top level query
{
query: {}
}
Queries influence the score of a document and are therefor less cacheable than filters. Queries run in the query phase and thus also influence the documents that are facetted over.
Filtered query
{
query: {
filtered: {
filter: {}
query: {}
}
}
}
This allows you to run filters in the query phase taking advantage of their better cacheability and have them influence the documents that are facetted over.
Facet filter
"facets" : {
"<FACET NAME>" : {
"<FACET TYPE>" : {
...
},
"facet_filter" : {
"term" : { "user" : "kimchy"}
}
}
}
this allows you to apply a filter to the documents that the facet is run over. Remember that the it'll be a combination of the queryphase/facetfilter unless you also specify global:true on the facet as well.
Query Facet/Filter Facet
{
"facets" : {
"wow_facet" : {
"query" : {
"term" : { "tag" : "wow" }
}
}
}
}
Which is the one that #thomasardal is using in this case which is perfectly fine, it's a facet type which returns a single value: the query hit count.
The fact that your Query Facet returns 38 and not 21 is because you use a filter for your time range.
You can fix this by either doing the filter in a filtered_query in the query phase or apply a facet filter(not a filter_facet) to your query_facet although because filters are cached better you better use facet filter inside you filter facet.
Confusingly Filter Facets are specified using .FacetFilter() on the search object. I will change this in 1.0 to avoid future confusion.
Sadly: .FacetFilter() and .FacetQuery() in NEST do not allow you to specify a facet filter like you can with other facets:
var results = typedClient.Search<object>(s => s
.FacetTerm(ft=>ft
.OnField("myfield")
.FacetFilter(f=>f.Term("filter_facet_on_this_field", "value"))
)
);
You issue here is that you are performing a Filter Facet and not a normal facet on your query (which will follow the restrictions applied via the query filter). In the JSON, the issue is because of the "query" between the facet name "notfound" and the "terms" entry. This is telling Elasticsearch to run this as a separate query and facet on the results of this separate query and not your main query with the date range filter. So your JSON should look like the following:
{
"facets": {
"notfound": {
"term": {
"statusCode": {
"value": 404
}
}
}
},
"filter": {
"bool": {
"must": [
{
"range": {
"time": {
"from": "2014-04-05T05:25:37",
"to": "2014-04-07T05:25:37"
}
}
}
]
}
}
}
Since I see you have this tagged with NEST as well, in your call using NEST, you are probably using FacetFilter on your search request, switch this to just Facet to get the desired result.