I am extracting info from logfiles, but I want to map them together for aggregations, here's a sample logfile:
2017-01-01 07:53:44 [monitor_utils.py] INFO: Crawled iteration for merchant ariika started
2017-01-01 07:53:44 [utils.py] INFO: UpdateCrawlIteration._start_crawl_iteration function took 0.127 s
2017-01-01 07:57:22 [statscollectors.py] INFO: Dumping Scrapy stats:
{'item_scraped_count': 22,
'invalid_items_count': 84}
I am extracting the merchant name from the first line ariika and items_scraped_counts, invalid_items_count from the last two lines, I have different logfiles for each merchant, and I want to know items scraped count per logfile for each merchant using Kibana.
How to filter between one merchant and another in my case?
if I understand you in a correct way, I believe the source field in kibana can help you.
The source field indicates the name of log file.
using the bar char, you can choose the metric that you want based in your tems scraped count field, and then create a bucket aggregation using the source field. And filter based in merchant field in the kibana search bar.
Or on top of the first aggregation you can create another aggregation using the merchant field and choose the split bars. without using the kibana search bar.
Related
Problem
I am trying to build a dashboard in elastic with a table to monitor job runs.
I want to have per run the minimum timestamp (ie. job start) and the number of processed messages. The minimum timestamp is my problem, I can't seem to get it.
What I have done
All my log lines have as (relevant) fields: #timestamp, nb_messages, run_id. run_id is unique per run, and a run creates multiple log lines.
I create a dashboard, add a TSVB panel, and select Table.
I use run_id as the field to group by.
I can use max(nb_message) in my table without issue.
But if I use min(#timestamp), or any other aggregation than count, I just get a -.
I first tried with a lens instead of a TSVB panel, and I had the same issue, but with as message: To use this function, select a different field.
I can confirm in the index that logging.timestamp has date for type.
Question
Is there a way to use the timestamp as metric?
I would use a "normal" data table visualization (navigate through Aggregation based option in the Visualization menu if you're using the latest version of Kibana) instead of the TSVB. There, the default metric is count representing the amount of events of the index pattern in the selected time range. You can use the min metric on the #timestamp field and aggregate/group your data as you want.
The preliminary is of course that the selected index pattern contains an #timestamp field.
I hope I could help you.
I want to create a Kibana metric for the unique users visiting my site.
I have an index collecting logs from a service in format
<date> <user1#gmail.com> - <log message> <client>
and I want to count unique user emails ignoring the rest of the fields.
Is it possible to do such a regex via some of the aggregations? Currently I was able to find only unique count based on some specific field which is not an option for me.
You can create a separate field first:
Either by using kibana scripted fields.
Or by using logstash mutate filter plugin.
And then you can apply terms aggregation on data table visualization to achieve this.
I have a request which sum the core and memory of specify jobs, the origin data about core and memory have been put in the elk logs. Like as below picture.
First red column is core data, the second column is memory data, third red column is job name. Now I want to sum the "core" or "memory" by filtering specify jobs and create a dashboard in kibana. As the dashboard, x-axis is 'time', y-axis is sum of 'core' or 'memory'.
I don't know how to sum the column in elk and create dashboard, someone who can help me, thank you so much.
You need to add a scripted field which will add a field at runtime in every document by operations on your current field.
After creating an index pattern , go to that index pattern in settings tab in got to scripted fields tab and a new scripted field.
Name the field which will appear while making the visualisation and add the script as follows.
Here is how to make a script
https://www.elastic.co/guide/en/elasticsearch/reference/6.x/search-request-script-fields.html
And here is a link of how you can do that in kibana
https://www.elastic.co/guide/en/kibana/current/scripted-fields.html
I have records in Elasticsearch as vlabs/usage - 13000 records . I was able to draw dashboards from this.
To the same index I added a new doc-type (feedback) as vlabs/feedback - 1000 records.
How can I draw a separate Kibana dashboard for feedback records.
I can see records of both doc_type in single section in Discover section.
You can see records from different type of same index in single Discover section.
Using can use _type: doc_typeA AND _type: doc_typeB to get results but they've to be from same index.
reference link: https://www.elastic.co/guide/en/beats/packetbeat/current/kibana-queries-filters.html
Document type in elasticsearch is identified by a filed named _type. Say that you have an index designation with multiple document types Manager, Engineer, SupportEngineer as shown below -
designation/Manager
designation/Engineer
designation/SupportEngineer
Just write _type:Manager in kibana query box and you get all records listed in the Discovery section.
I am building visualizations in Kibana for AWS CloudWatch metrics, and have run into a bit of an issue creating Metric Tables.. Kibana is splitting my fields that contain dashes (instance ID, region, etc..)
Rather than having an individual row with an instance ID, for example, i-7bb06dzz, it is creating 2 rows: i & 7bb06dzz. "i" displays the aggregate count of all other fields. If I add a second split with the region, this is duplicated for every set of characters in the region name separated by dashes as well. (us , east, and 1 instead of just us-east-1).
I tried to post a screenshot, but my reputation is not high enough to do so..
Here are my visualization settings:
Metrics: Metric (Count)
Aggregations:
Split Rows: Terms: InstanceID: Top 5: Order by metric:Count
Split Rows: Terms: Region: Top 5: Order by metric:Count
No Advanced Settings have been specified. I was able to get a reasonable looking list by only specifying InstanceID, and excluding the pattern "i"... however, it doesn't do me a lot of good when I can't display the region next to it. Both values are indexed as strings and were recorded in ElasticSearch with double quotes around them.
Any recommendations on how to display the fields as intended would be much appreciated.
This is because Elasticsearch "analyzes" the field for the individual tokens in it. Logstash will store fields in both the fieldname and fieldname.raw fields - the latter is unanalyzed and will behave as you expect.