Problem
I am trying to build a dashboard in elastic with a table to monitor job runs.
I want to have per run the minimum timestamp (ie. job start) and the number of processed messages. The minimum timestamp is my problem, I can't seem to get it.
What I have done
All my log lines have as (relevant) fields: #timestamp, nb_messages, run_id. run_id is unique per run, and a run creates multiple log lines.
I create a dashboard, add a TSVB panel, and select Table.
I use run_id as the field to group by.
I can use max(nb_message) in my table without issue.
But if I use min(#timestamp), or any other aggregation than count, I just get a -.
I first tried with a lens instead of a TSVB panel, and I had the same issue, but with as message: To use this function, select a different field.
I can confirm in the index that logging.timestamp has date for type.
Question
Is there a way to use the timestamp as metric?
I would use a "normal" data table visualization (navigate through Aggregation based option in the Visualization menu if you're using the latest version of Kibana) instead of the TSVB. There, the default metric is count representing the amount of events of the index pattern in the selected time range. You can use the min metric on the #timestamp field and aggregate/group your data as you want.
The preliminary is of course that the selected index pattern contains an #timestamp field.
I hope I could help you.
Related
*We are trying to utilize the time range selector in grafana which works only for time series representation as of now. Can you please let me know any way to utilize that time range selector for bar-chart and bar-graph in Grafana representation.*
Adding the Schema, query and representation in grafana for reference.
Schema:[ https://i.stack.imgur.com/n2srW.png]
query: [https://i.stack.imgur.com/HwWRJ.png]
Bar-chart rep: [https://i.stack.imgur.com/1MQbA.png]
Not totally sure what you are asking but hopefully this is right.
Your query has no where clause to filter by the selected time in Grafana. You can use a Grafana macro to add a time range filter. The MySQL macro documentation is here. And there are some examples further down on the page.
You haven't specified what the table schema looks like or which column you want to use for filtering. So I made a guess and just added a where clause with the $__timeFilter macro and filtering on the lastupdated column as it is the only datetime column in your example:
SELECT
lastupdated as time_sec,
TIME_FORMAT(sec_to_time(time), "%H:%i") AS total_time,
mrid as metric
,time/3600 as time_in_hours
FROM margetomerge.mergedetails
WHERE $__timeFilter(lastupdated)
order by mrid desc
I have simple data analytics to display in AWS QuickSight: some date fields and amounts. QS aggregates by default all date-fields and the lowest granularity is aggregate by minute. But my need is to display all data without any aggregation at all. I have searched but not found how it could be possible disable aggregation at all? Any ideas?
First contribution to Stack Overflow, feels good man.
I ran into the same problem and the solution is to convert the field selected for the y-axis to a dimension rather than a measure.
Converting fields from measure to dimension
Aggregations will automatically be applied to 'measure' fields and therefore they need to be converted to 'dimension'
Kibana provides time filter shortcuts like Today, Yesterday, last 10 days etc in the dashboard. I want to know how do they work. When I click Today in Kibana, Which field is used in the query? How can i configure these links to take custom timestamp fields?
When you create your index the first time you choose after that Time filter field name:
this filter will be used when you choose last 15 minutes or another time filter.
By default it is the timestamp but if you have a time field in your fields list you can use it.
I have a request which sum the core and memory of specify jobs, the origin data about core and memory have been put in the elk logs. Like as below picture.
First red column is core data, the second column is memory data, third red column is job name. Now I want to sum the "core" or "memory" by filtering specify jobs and create a dashboard in kibana. As the dashboard, x-axis is 'time', y-axis is sum of 'core' or 'memory'.
I don't know how to sum the column in elk and create dashboard, someone who can help me, thank you so much.
You need to add a scripted field which will add a field at runtime in every document by operations on your current field.
After creating an index pattern , go to that index pattern in settings tab in got to scripted fields tab and a new scripted field.
Name the field which will appear while making the visualisation and add the script as follows.
Here is how to make a script
https://www.elastic.co/guide/en/elasticsearch/reference/6.x/search-request-script-fields.html
And here is a link of how you can do that in kibana
https://www.elastic.co/guide/en/kibana/current/scripted-fields.html
In Kibana, I have an index that looks like as follows
type (String)
value (String)
timestamp (Date)
I would like to have a visualization that shows the most recent value field where the type is equal to "battery", for example.
I would like the visualization to be similar to the "Metric" one, but displaying a string of text instead of a number, of course.
Is this possible with Kibana? If not, how can I get a similar result?
You can use a Data Table visualization.
In the search query you would specify type: "Battery"
In the metric section you would specify Max timestamp
In the Split Rows section you would specify Aggregation=Terms, Field=value, OrderBy=metric:Max timestamp, Order=descending, Size=1
You will have a result that is a table with 1 row and 2 columns, one of which being a value and the other a timestamp
If this does not satisfy your needs, you may look into available Kibana plugins that allow new visualizations (see the list of known plugins) or modify one of them to suite your needs.