I'm facing a following problem. In Kibana 4 I've created a line chart based on my input from elasticeasrch but I can only display average, min, max instead of an actual value of the field per time, e.g. sent bytes.
Most answears to that question on stackoverflow are about Kibana 3 (How to create value over time chart with Kibana 3?) and seem to include a Histogram on a X axis, yet I can't seem to find one which will enable me to apply them to Kibana 4. I was unable to find the histogram panel and once I click on the discover tab there is the constant Searching loading.
If I have the following fields in my _source:
{"timestamp":"2015-06-02T10:16:44.0855","time":587,"threadName":"Thread Group 1-957","byte":1372,"status":"false","latence":306,"registerCall":"404"}
and I would like to have the number of bytes on the Y-axis and on the X-axis my timestamp.
Any help in the right direction will be appreciated :)
To create a value over time line chart in Kibana, follow these steps:
Go to visualize tab and select line chart
In the X-axis, select X-axis, Aggregation as Date Histogram and then select your timestamp field as the date field.
Next for the Y-Axis, select Sum as the aggregation and then bytes as the field.
For the X axis, what Alcanzar said is good, but as you notice, the Y axis is problematic.
Sum (suggested by "Limit") works, but since it's aggregated, it shows the total used in each aggregated bucket, but that may be meaningless depending on what you are trying to show. Your question isn't clear on what you want, so I'm just guessing here. One hour of requests, each of which ran for one minute and sent 1 megabyte is indeed 60 megabytes-minutes, if you are trying to show total capacity used over than hour (maybe you are paying a bill based on usage per time). On the other hand, if you are trying to show peak usage in each time, it would be wrong.
You said you already looked and Max and Min and they don't meet your needs. I don't suppose Standard Deviation would be any better?
I have the same concern. The best I've been able to do so far is
display Min and Max simultaneously in the Y axis. When they diverge, I know I'm zoomed out too far, so I zoom in until they align.
This is how I know I'm seeing individual events.
In any case, I share your frustration. I too would like to be able to show time series as easily as I can in, say, Excel.
Related
We have a counter metric in one our micro services which pushes data to DataDog. I want to display the total count for given time frame, and also the count per day (X axis would have the date and Y axis would have count). How do we achive this?
I tried using sum by and diff with Query value representation. It gives the total number of the count for given time frame. But I would like to get a bar graph with the X axis as the date and the Y axis as the count. Is this possible in DataDog?
It seems like there are 2 main questions here:
display the total count for a given time frame.
the count per day.
I think the rollup method is going to be your friend for both questions.
For #1 you need to pass in the time frame you want a total over: sum:<metric_name>.rollup(sum, <time_frame>) and the single value can be displayed using the Query Value visualization.
For #2 the datadog docs say you can get metrics per a day by
graphed using a day-long rollup with .rollup(avg,86400)
So this would look something like sum:<metric_name>.rollup(sum, 86400) and can be displayed a Timeseries with bars.
I want to visualize the amount of correct auto-responses my system sent in regards to the percentage of questions it has already learned.
So my idea was to filter all my test-results where a boolean field didSendCorrectAutoResponse is true, make the bucket on the x axis over a field called learnPercentage and on the y axis simply take the count as a metric.
The only problem with this is that the values on the y-axis are absolute and only count the number of responses sent but I want it to show it as a percentage of the total number of tests per percentage learned.
Here is how I defined my chart:
I can calculate the total number of test-cases for each percentage learned with this learnPercentage: 100 && strategy.keyword: "sum" (it only counts them for 100% questions learned, but the number of tests for each percentage is the same).
So what I want on the y-axis is not the plain count but count / totalNumberOfTestCases
edit:
In order for you to better understand what I need here is what I do with my system:
Lets say I have 100 known questions my system can learn. And I have 2500 test questions. Now I do the following:
Let my system learn none of the known questions
Ask the 2500 test questions
Save how many questions have been correctly answered (let's say 600)
Save this test result in elastic
Repeat with 10 questions learned:
Let my system learn 10% of the known questions
Ask the 2500 test questions
Save how many questions have been correctly answered (let's say 590)
Save this result in elastic
Repeat with 20 questions learned...
Now I want to plot how many questions have been correctly answered in each learning step:
600 at 0%
590 at 10%
900 at 20%
...
But instead of showing these absolute numbers I want 600/2500, 590/2500 etc on the y-axis.
For Visualizing your Y axis in percentage if it is not already in, You should first create a scripted field for your favorite column and then visualize that scripted field in kibana.
check the photos; in scripted field code, the removed part is your column name.
my kibana version is 4.5.
my elastic version is 2.3.1.
see the pic1 .the uv is 7665.
but see the pic2.the uv is 7845.
why diffrent ?
kibana unique count seeing not correct.
If these charts are based on live data, then I doubt both the graphs cannot show the same count since you're having two different time-range in both the graphs.
In the first one your time range is yesterday, where as in the second one your trying to have an auto-refresh every minute which shows as paused. I'm assuming that you're dealing with live data so that some records might have slipped through, by the time you paused. If not I cannot see any chances of these two showing two different values.
Just being curious, how do you know that the correct count for uv should be 7665 since I can't see the exact value of uv from the snapshot of the graph? Did you double check from your ES indice through a query?
EDIT:
Interestingly Unique counts are based on the cardinality aggregation, which is designed to work efficiently across very large amounts of data and delivers an approximate result, which may why your results vary. You can maybe try increasing the precision_threshold.
To get a more correct value, add a something like: {"precision_threshold": 1000} to the "JSON Input" box for the aggregation.
Hope this helps!
I was just wondering if anybody knew of a way to be able to show a graph of the difference of metrics like system.network.in.bytes -
If you look at this graph you can just see that the value continuously gets bigger (at around the same speed) - but I just want to graph the difference between each value not the total.
Example
Anyone have any ideas?
Try a timeseries visualization or timelion.
Assuming your field name is 'bytesIn' (for simplicity) and taking 1 minute intervals (as IMO 30s isn't possible in timelion), your timelion expression should look something like:
.es(*,metric='avg:bytesIn').subtract(.es(*,metric='avg:bytesIn',offset='-1m'))
Explanation
.es(*,metric='avg:bytesIn') gives average of bytesIn over a time interval (here I'm assuming 1m)
Adding offset='-1m', offsets the series retrieval by -1m as if they are happening now
.subtract just subtracts value of one series from another
I am ingesting data to elasticsearch using flume, I want to create a time-series graph in kibana to show the events collected over time. BUT I also want to to the average per that time unit so the user knows if the current flow is around the average or not.
To create a timeline I am using line graph with #timestamp as X-axis and count as Y-axis.
The question is how to create the average line? and how to make this average dynamic e.g. as we zoom in average changes from average per day to average per hour.
While creating a visualization you can choose the type of y-axis metric. The default is "count". You can click on the icon to choose other type of metrics you want. It will have various options like average, sum, percentile etc.
As for the time range of average calculation, the the x-axis metrics, under buckets when you choose date histogram the default interval is auto.This means that the time range of average will chage automatically depending on overall time range selected.
You can change it to a fixed interval like per second, minute, hourly daily etc.
It's a bit odd, you would expect count to appear alongside field as something you can average. In reality you have to do it another way:
For the Y axis, instead of selecting count, select "Average Bucket"
then set up your bucket aggregation that you would like e.g. Date Histogram with second interval.
Below this you have another box for metric, e.g. the thing you're averaging, set this to count