what I have:
24h logged data in elasticsearch with a number field containing the byte size of transmitted messages (microsecond granularity).
Via a date histogram I can easily drill down to ms-intervals to determine network traffic spikes.
what I need:
a deterministic way to find the maximum traffic spike within the 24 hours based on a fixed size 100 ms interval.
find( max( sum(bytessize) of X ms interval)) over Yh range)
I'm new to the ELK-Stack, so any help how and where (elasticsearch or kibana) to solve such a problem is appreciated.
If i understand correctly, then something very similar to what you want can be done, and you are very close.
Click on the 'pencil' icon in the top right corder of you graph to edit the visualization. In your 'Y-Axis' aggregation choose 'Max' as the aggregation type.
In the 'X-Axis' aggregation section, I see you're already using a 'Date Histogram', so just define the interval to be 'Second' (That's the lowest possible interval in kibana, currently available)
Related
I am trying to create a Kibana TSVB visualization that displays an “events per second (EPS)” metric for the last created elasticsearch index of a particular index pattern. Currently I’m using a Count aggregator that pipes to the Math aggregator with the formula params.Count / (params._interval / 1000).
But this calculation is only accurate if the chart’s timerange is set to exactly the first and last timestamps in the index. Otherwise the empty data sets (both before and after the index’s timeframe) is being included in calculating the EPS. Currently I have to manually query the min/max timestamps of the index and then manually set the chart’s timeframe in the upper right corner to match that, only then it calculates the EPS correctly.
So my question… is there a way to automatically do this? Such as having the chart’s Start and End timerange as variables equal to the Min and Max timestamps of the particular index I’m looking at? Or have it ignore the out of bounds time range?
Thanks
Given a series incoming events like say:
#timestamp1: a,b,c,d,e
#timestamp2: a,b,c,d,e
(all numbers)
I need to perform some calculation which would be of the form
(max (a) - min (a) )* (max (b) - min (b)) / (max © - min © ).
I know how to show it as a time series graph (using Visual Builder). But I also want to show it as a simple number for the overall duration that has been selected.
I tried lucene expression numeric APIs (doc[‘field_name’].max(), min()) but that doesn’t work. I didn’t see any such API within painless.
I also looked at “Scripted Metric Aggregation”, but couldn’t quite understand, where in Kibana to specify those expressions.
Same is the case with “Metrics Aggregation”, how do I make use of it within Kibana?
How can displaying aggregated number be so difficult as compared to a time-series chart? Any help is appreciated. Thanks.
In kibana when you create visualisation, you have in the bucket section in the bottom "advance" link, click on it, text area will open, there you can code agg that will inject to kibana aggregation, try write there your metric aggregation
I have a query with respect to building visualizations in Kibana.
I have already performed aggregations like uniq value over a period of time(say every 1 minute for 30 days) and saved it in Elastic Search. Now I want to plot this graph in Kibana
The problem I am facing is that Kibana is asking me to select a aggregation(like sum, uniq count,etc.) for Y-axis and select a field to apply the same.
On X-axis, I am having timestamp.
But since as mentioned above, I have already done the Uniq Count of my data set over timeperiod of 1 minute interval for a period of 30 days and stored it in Elastic Search, Hence I donot want Kibana to aggregate the data for me, rather just plot the data which I have in my Elastic Search.
Is it possible to just plot the data which I have in Elastic Search to Visualize in Kibana, but not aggregating them over there in my Y-axis?
If Yes How can I do so. Please advise on this problem of Kibana.
Thanks
I don't know if you are still struggling with this or not...
If you already have the data "summed up", then as you say, it is not necessary to tell Kibana (Actually Elasticsearch) to do the summing for you... this is true, if you are going to be plotting ONE minute time intervals. Over a period of 30 days is around (...) 43200 intervals... it is a LOT for charting. So kibana is going to suggest (strongly, and with a lot of sense) to use say, 30 minutes or even hourly intervals.
Now, if for instance you are grouping together events for a whole hour, then Elasticsearch is going to either SUM/AVG/MIN/MAX this 60 measurements and get a single value for that hour's plot.
So, my advice is that you select a Average Metric, and use common sense for the time lapse selection. That way you are going to have your chart, and if you really need to go to the minute by minute detail, you are go
I'm facing a following problem. In Kibana 4 I've created a line chart based on my input from elasticeasrch but I can only display average, min, max instead of an actual value of the field per time, e.g. sent bytes.
Most answears to that question on stackoverflow are about Kibana 3 (How to create value over time chart with Kibana 3?) and seem to include a Histogram on a X axis, yet I can't seem to find one which will enable me to apply them to Kibana 4. I was unable to find the histogram panel and once I click on the discover tab there is the constant Searching loading.
If I have the following fields in my _source:
{"timestamp":"2015-06-02T10:16:44.0855","time":587,"threadName":"Thread Group 1-957","byte":1372,"status":"false","latence":306,"registerCall":"404"}
and I would like to have the number of bytes on the Y-axis and on the X-axis my timestamp.
Any help in the right direction will be appreciated :)
To create a value over time line chart in Kibana, follow these steps:
Go to visualize tab and select line chart
In the X-axis, select X-axis, Aggregation as Date Histogram and then select your timestamp field as the date field.
Next for the Y-Axis, select Sum as the aggregation and then bytes as the field.
For the X axis, what Alcanzar said is good, but as you notice, the Y axis is problematic.
Sum (suggested by "Limit") works, but since it's aggregated, it shows the total used in each aggregated bucket, but that may be meaningless depending on what you are trying to show. Your question isn't clear on what you want, so I'm just guessing here. One hour of requests, each of which ran for one minute and sent 1 megabyte is indeed 60 megabytes-minutes, if you are trying to show total capacity used over than hour (maybe you are paying a bill based on usage per time). On the other hand, if you are trying to show peak usage in each time, it would be wrong.
You said you already looked and Max and Min and they don't meet your needs. I don't suppose Standard Deviation would be any better?
I have the same concern. The best I've been able to do so far is
display Min and Max simultaneously in the Y axis. When they diverge, I know I'm zoomed out too far, so I zoom in until they align.
This is how I know I'm seeing individual events.
In any case, I share your frustration. I too would like to be able to show time series as easily as I can in, say, Excel.
I have documents under two daily indexes. Both have count field which is >=1.
I want to create a graph which shows trend of ratio of these two fields aggregated over time.
Data will be sampled based on time duration selected in dashboard ex : for one day each sample would be be 10 min which will sum these two fields separately and calculate ratio and then show as one data point. So for 24 hours it would be 24*60 point in the graph.
How can I achieve same in Kibana 4 ?
We tried something similar but turns out it is not possible in Kibana.
As of now you can not plot a calculated field based on two different fields in Kibana.
To workaround this, we implemented a plugin that modifies data before it is pumped to elastic search. So we carried out calculations in that plugin. Also, the plugin periodically pumps data to elastic search so kibana gets the latest values