I'm working with ES and I need a query that returns the difference between two datetime (mysql timediff), but have not found any function of ES to do that. Someone who can help me?
MySQL Query
SELECT SEC_TO_TIME(
AVG(
TIME_TO_SEC(
TIMEDIFF(r.acctstoptime,r.acctstarttime)
)
)
) as average_access
FROM radacct
Thanks!
Your best best is scripted fields. The above search query should work , provided you have enabled dynamic scripting and these date fields are defined as date in the mapping.
{
"script_fields": {
"test1": {
"script": "doc['acctstoptime'].value - doc['acctstarttime'].value"
}
}
}
Note that you would be getting result in epoch , which you need to convert to your denomination.
You can read about scripted field here and some of its examples here.
Here is another example using script fields. It converts dates to milli seconds since epoch, subtracts the two and converts the results into number of days between the two dates.
{
"query": {
"bool": {
"must": [
{
"exists": {
"field": "priorTransactionDate"
}
},
{
"script": {
"script": "(doc['transactionDate'].date.millis - doc['priorTransactionDate'].date.millis)/1000/86400 < 365"
}
}
]
}
}
}
Related
I'm storing articles of blogs in ElasticSearch in this format:
{
blog_id: keyword,
blog_article_id: keyword,
timestamp: date,
article_text: text
}
Suppose I want to find all blogs with articles that mention X at least twice within the last 30 days. Is there a simple query to find all blog_ids that have articles with the same word at least n times within a date range?
Is this the right way to model the problem or should I use a nested objects for an easier query?
Can this be made into a report in Kibana?
The simplest query that comes to mind is
{
"_source": "blog_id",
"query": {
"bool": {
"must": [
{
"match": {
"article_text": "xyz"
}
},
{
"range": {
"timestamp": {
"gte": "now-30d"
}
}
}
]
}
}
}
nested objects are most probably not going to simplify anything -- on the contrary.
Can it be made into a Kibana report?
Sure. Just apply the filters either in KQL (Kib. query lang) or using the dropdowns & choose a metric that you want to track (total blog_id count, timeseries frequency etc.)
EDIT re # of occurrences:
I know of 2 ways:
there's the term_vector API which gives you the word frequency information but it's a standalone API and cannot be used at query time.
Then there's the scripted approach whereby you look at the whole article text, treat is as a case-sensitive keyword, and count the # of substrings, thereby eliminating the articles with non-sufficient word frequency. Note that you don't have to use function_score as I did -- a simple script query will do. it may take a non-trivial amount of time to resolve if you have non-trivial # of docs.
In your case it could look like this:
{
"query": {
"bool": {
"must": [
{
"script": {
"script": {
"source": """
def word = 'xyz';
def docval = doc['article_text.keyword'].value;
String temp = docval.replace(word, "");
def no_of_occurences = ((docval.length() - temp.length()) / word.length());
return no_of_occurences >= 2;
"""
}
}
}
]
}
}
}
Hi i am kinda new to elasticsearch. I need to get an aggregation with date comparison and a dynamic range filter.
Like i need to get documents count where created_at document is 1 week earlier than their identification_date.
So i tried something like this but my date param seems unused, actually changing it never changes my results.
"aggs": {
"identified": {
"terms": {
"script": "doc['created_at'].value > (doc['identification_date'].value - diff_date)
&& doc['created_at'].value < doc['identification_date'].value",
"params": {
"diff_date": 604800
}
}
}
}
Thank you for taking time helping.
since on the abstracted level you just want count of documents created 7 days before on some date field.you don't need aggregation for this if you don't want to group the results sets further on some fields.Simply you can a)use range filter query on your date field
{
"range" : {
"date" : {
"gte" : "now-7d/d",
"lt" : "now"
}
}
}
https://www.elastic.co/guide/en/elasticsearch/reference/current/query-dsl-range-query.html
How can I query for, or filter for, one field doesn't equal another field? i.e. where document1.city1.name not equal document1.city2.name.
Some version of this?
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-request-script-fields.html
Yes , you will need to use script filter to achieve this
{
"bool": {
"filter": {
"script": {
"script": "doc['field1'].value != doc['field2'].value"
}
}
}
}
You can find more information here
I have a query that properly parses date ranges. However, my database has a default value that all dates have a timestamp of 00:00:00. This means that items that are still valid today are shown as expired even if they should still be valid. How can I adjust the following to look at just the date and not the time of the item (expirationDate).
{
"range": {
"expirationDate": {
"gte": "now"
}
}
}
An example of the data is:
"expirationDate": "2014-06-24T00:00:00.000Z",
Did you look into the different format options for dates stored in ElasticSearch? If this does not work for you or you don't want to store dates without the time you can try this query, which will work for your exact use case I guess:
{
"range": {
"expirationDate": {
"gt": "now-1d"
}
}
}
You can also round down the time so that your query returns anything that occurred since the beginning of the day:
Assuming that
now is 2017-03-07T07:00:00.000,
now/d is 2017-03-07T00:00:00.000
Your query would be:
{
"range": {
"expirationDate": {
"gte": "now/d"
}
}
}
elastic search documentation on rounding times
I'm storing something like the following information in elastic search:
{ "timeslot_start_at" : "2013-02-01", "timeslot_end_at" : "2013-02-03" }
Given that I have another date range (given from user input for example) I am wanting to search for an intersecting time range. Similar to this: Determine Whether Two Date Ranges Overlap Which outlines that the following logic is what i'm after:
(StartDate1 <= EndDate2) and (StartDate2 <= EndDate1)
But I'm unsure of how to fit this into an elastic search query, would I use a range filter and only set the 'to' values, leaving the from blank? Or is there a more efficient way of doing this?
Update: It is now possible to use date_range data type that was added in elasticsearch v5.2. For an earlier version of elasticsearch the following solution still applies.
To test for intersection, you should combine two range queries into a single query using bool query:
{
"bool": {
"must": [
{
"range": {
"timeslot_start_at": {
"lte": "2013-02-28"
}
}
},
{
"range": {
"timeslot_end_at": {
"gte": "2013-02-03"
}
}
}
]
}
}