Log records of the following shape produced by a multi-threaded application are pushed to my elasticsearch index 'log':
[2015-10-09T09:52:18.928] [Debug] [00000x2934 0x000026c0] [Visualization]: updated rendering
[2015-10-09T09:52:19.966] [Debug] [00000x2934 0x000013a0] [Database]: Query request accepted
...
The two hexadecimal entries are process and thread ID.
A document in the elasticsearch index looks like this:
{
"_index": "log",
"_type": "record",
"_id": "AVBXUnCah58TK-z65dea",
"_score": 1,
"_source": {
"process": "00000x2934",
"severity": "Debug",
"thread": "0x000026c0",
"recordId": 1,
"timestamp": "2015-10-09T09:52:18.928",
"message": "updated rendering",
"channel": "Visualization"
}
}
How can I create a Kibana visualization that has a time range as X axis and on its Y axis it has entries for the different thread IDs (in my application there is a thread pool with a fixed number of threads, <= 10). There should be a point (X/Y) for such an event with appropriate information (message or channel).
To make it short: How can I visualize the chronology of a multithreaded application by its events using this search index and Kibana?
One additional note: If there is an easy solution without Kibana, I am also okay with that. It doesn't need to be real-time.
I feel Gantt Chart might be useful to you. You can have one entry in Y axis per thread and visualize its function in each time frame. This is not available in Kibana4 as of now , but we can expect this in the future.
Related
I have multiple indexes to store user tracking log. In which there is 1 index is index-pageview. How can I query out the list of users who viewed the page 10 times between 2021-12-11 and 2021-12-13 using IOS operating system?
Log example:
index: index-pageview
[
{
"user_id": 1,
"session_id": "xxx",
"timestamp": "2021-12-11 hh:mm:ss",
"platform": "IOS"
},
{
"user_id": 1,
"session_id": "yyy",
"timestamp": "2021-12-13 hh:mm:ss",
"platform": "Android"
}
]
You can try building a normal bool query on timestamp and platform and then either terms aggregation (possibly with min_doc_count: 10) or collapse on user_id. Both ways will have some limitations though:
aggregation might be slower (needs benchmarking)
aggregation bucket number is limited (at 10k by default)
collapse will work on at most size docs at a time (capped at 10k as well) so you might need scrolling and app-side processing
Though performance of these might be pretty poor. If you need to run queries like those very often I would consider using another storage (SQL? Something more fancy?)
I have an index which contains information about some objects. I want to display some of the information on my Kibana's dasboard. Lets assume an object looks as follows:
{
"_index": "obj",
"_type": "_doc",
"_id": "KwDPAHABfo5V345r4IYV",
"_version": 1,
"_score": 0,
"_source": {
"value_1": "some value",
"value_2": "some_other value",
"owner": "jason",
"modified_date": "2020-02-01T12:53:08.210317+00:00",
"created_date": "2020-02-01T12:53:08.243980+00:00"
}
}
I need to show (live) number of objects that has owner: 'UNKNOWN'. Thing is, that this value changes in time. Each change is a new document - they are not being updated. I need to track how many UNKNOWN owners currently I see. Updates (new documents) are being sent to elk in fixed intervals.
When I try to set up a metric, it sometimes shows 0, during the window between one update and another - when there is no documents flowing into elk. How can I make Kibana display only last documents with owner: 'UNKNOWN'?
How can I make Kibana display only last documents with owner: 'UNKNOWN'?
You could set up a data table visualization for that as an alternative to the one-dimensional metric visualization.
This is how I personally would configure the data table:
Set a filter with 'owner(.keyword) is UNKNOWN'.
Use the metric 'Top Hit' on the field created_date (or #timestamp, thats up to you) instead of the count metric.
Set the order to descending based on the timestamp field.
Split the rows (Term Aggregations) for every field you want to display in the rows. This will create 'columns' in your table.
Go to the options tab and enable count on the sum of all rows.
Set an appropriate time interval, e.g. last 1 hour.
This will display all the relevant data of your documents that have the field owner equal to UNKNOWN. Also, you see the ingestion/creation date timestamp of these documents in a descending order. Furthermore, you see the number of documents that match (configured via the options tab as described above).
I hope I could help you.
I'm have an elasticsearch database with documents stored the following way(, seperates the documents):
{
"path":"path/to/data"
"kind": "type1"
},
{
"path":"path/to/data/values1"
"kind": "type2"
"x": 2
},
{
"path":"path/to/data/values2"
"kind": "type2"
"x": 2
},
{
"path":"path/to/data/datasub"
"kind": "type1"
},
{
"path":"path/to/data/datasub/values1"
"kind": "type2"
"x": 1
}
Now I want the create table view/chart show all type2's with all the sum of x of all their childeren.
So I expect the total of path/to/data to be 5 and the total of path/to/data/datasub 1.
To consider: the depth of this structure could theoretically be unlimited
I'm running Elastichsearch 7 and Kibana 7 and I want to use the table visualisation to start with but I would like to be able to use this kind of aggregation throughout multiple visualisations. I have Googles a lot and found all kinds of Elastichsearch queries but nothing on how to achieve this in Kibana.
All help is much appreciated
For those who run into the same question:
The solution I ended up using is to split the path in to tokens prior to importing it into Elasticsearch. So consider a document having a path like "/this/is/a/path". This becomes the following array in the document:
[
"/this",
"/this/is",
"/this/is/a",
"/this/is/a/path"
]
You can then use a terms aggregation on it with various metrics to calculate your desired measurements.
I'm a newbie to kibana.
I have following data stored in ES:
{
"_index": "test",
"_type": "impressions",
"_id": "AVZ4QLgkLqvQLIzbvF4e",
"_version": 1,
"_score": 1,
"_source": {
"campaign_id": "1011",
"count": 691,
"played_dt": "2016-01-02"
}
}
So, basically I have counts per campaign_id which is already aggregated data.
I want a simple bar chart which plots counts per campaign_id where X axis is campaign_id and Y axis is it's count.
I'm getting hits for that specific campaign_id as unique count rather than the actual value in count field.
Thanks in advance!
Go to "Visualize" tab, select "Vertical bar chart":
Choose new search and select appropriate index. Now you probably want to visualize your data in time. So, on X axis use "Date histogram" and select your time filed (played_dt).
Now you can use e.g. "Split bars", use splitting by terms and select campaign_id field.
I use logstash to store log files containing the speed of vehicles over time.
In Kibana 3, how can I generate a panel which displays a value over time, i.e. the x axis displays the time and the y axis the related value, e.g. vehicle speed.
Most panels I found count the occurrence of events in a given time span and display it on the y axis. My goal however is to directly print a value from the json log entry (wheelSpeed_m_s), which looks as follows:
{
"_index": "logstash-2013.05.07",
"_type": "vehicle_odometry",
"_id": "Q3b58Pi7RUKuPon0s_ihlA",
"_score": null,
"_source": {
"message": " ",
"wheelSpeed_m_s": 0.91,
"#timestamp": "2013-05-07T17:50:04.099+02:00",
"angularVelocity_rad_s": 0,
"type": "vehicle_odometry",
"#version": "1",
"ts_ms": 1367934604099
},
}
Any help is highly appreciated.
In the histogram panel, click the "Configure" (gear) icon, then select the "Panel" tab.
On that tab, you can select the "Chart value". This defaults to count, but can be any of the basic math set functions (mean, max, min, total). Select the function, and you'll be asked to enter the field to which the function should be applied:
OP: please don't accept this answer (rutter deserves the points for getting you straight). I leave the info here to complete the question so it's not marked as 'unanswered'.