So I am trying to automate the scrape of our internal Kibana Dashboards from within our environments for information gathering purposes. I have looked through the following link, but Elasticsearch doesn't seem to really provide good examples of what I am trying to do or accomplish here. Several constraints I have: 1. the commands must be in BASH, 2. I cannot use any compiler such as Python and the Requests and/or Beautifulsoup modules to grab the information and parse it.
Here is my Dilemma:
I log in to the Kibana Dashboard:
Some http://<IP_ADDRESS>:5601/app/kibana#/dashboards?_g=(refreshInterval:(pause:!t,value:0),time:(from:now-1h,mode:quick,to:now))
It will look like this if I am properly tunneled into the environment.
There are three dashboards that I want to collect:
API RESPONSES
Logs
Notifications
The example curl command I am using is as follows to scrape the dashboards:
curl -s http://<IP_ADDRESS>:5601/app/kibana#/dashboard/API\ RESPONSES
curl -s http://<IP_ADDRESS>:5601/app/kibana#/dashboard/logs
curl -s http://<IP_ADDRESS>:5601/app/kibana#/dashboard/notifications
Now the Elasticsearch Documentation mentions something of a Dashboard ID, to which I cannot see it unless I open a webpage and use the inspect tool on a particular element I am sending the GET request to. I am trying to accomplish that by curling the main dashboard page:
curl -s http://<IP_ADDRESS>:5601/app/kibana#/dashboard/_search?pretty
My output will return an HTML output, but it doesn't seem to change and I cannot properly acquire the Dashboards without knowing the Dashboard ID. Furthermore, I am trying to see what dashboards are available and scrape all of them depending on what a person has set up within the environment so it's important that this process is dynamic. My eventual and ultimate goals are to get:
Dashboard IDs Available
Scrape the Dashboards by IDs
Basically I want to curl this output to get the return JSON.
Any thoughts would be greatly appreciated.
So apparently, I was curling the wrong location.
I needed to curl the VIP and port 9200 for the Kibana Index to pull in the available dashboards.
rbarrett#cfg01:~$ curl -s http://<IP_ADDRESS>:9200/.kibana/dashboard/_search?pretty
{
"took" : 15,
"timed_out" : false,
"_shards" : {
"total" : 1,
"successful" : 1,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 2,
"max_score" : 1.0,
"hits" : [
{
"_index" : ".kibana",
"_type" : "dashboard",
"_id" : "logs",
"_score" : 1.0,
"_source" : {
"description" : "",
"hits" : 0,
"kibanaSavedObjectMeta" : {
"searchSourceJSON" : "{\"filter\":[{\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"*\"}}}]}"
},
"optionsJSON" : "{\"darkTheme\":true}",
"panelsJSON" : "[{\"col\":1,\"columns\":[\"Hostname\",\"Logger\",\"programname\",\"severity_label\",\"Payload\",\"environment_label\"],\"id\":\"search-logs\",\"panelIndex\":5,\"row\":13,\"size_x\":12,\"size_y\":12,\"sort\":[\"Timestamp\",\"desc\"],\"type\":\"search\"},{\"col\":1,\"id\":\"NUMBER-OF-LOG-MESSAGES-PER-SEVERITY\",\"panelIndex\":7,\"row\":9,\"size_x\":6,\"size_y\":4,\"type\":\"visualization\"},{\"col\":7,\"id\":\"TOP-10-PROGRAMS\",\"panelIndex\":9,\"row\":5,\"size_x\":6,\"size_y\":4,\"type\":\"visualization\"},{\"col\":1,\"id\":\"LOG-MESSAGES-OVER-TIME-PER-SOURCE\",\"panelIndex\":10,\"row\":1,\"size_x\":6,\"size_y\":4,\"type\":\"visualization\"},{\"col\":7,\"id\":\"TOP-10-HOSTS\",\"panelIndex\":11,\"row\":9,\"size_x\":6,\"size_y\":4,\"type\":\"visualization\"},{\"col\":1,\"id\":\"TOP-10-SOURCES\",\"panelIndex\":14,\"row\":5,\"size_x\":6,\"size_y\":4,\"type\":\"visualization\"},{\"col\":7,\"id\":\"LOG-MESSAGES-OVER-TIME-PER-SEVERITY\",\"panelIndex\":16,\"row\":1,\"size_x\":6,\"size_y\":4,\"type\":\"visualization\"}]",
"timeFrom" : "now-1h",
"timeRestore" : true,
"timeTo" : "now",
"title" : "Logs",
"uiStateJSON" : "{\"P-10\":{\"vis\":{\"legendOpen\":true}},\"P-11\":{\"vis\":{\"colors\":{\"Count\":\"#629E51\"},\"legendOpen\":true}},\"P-12\":{\"spy\":{\"mode\":{\"fill\":false,\"name\":null}},\"vis\":{\"colors\":{\"Count\":\"#2F575E\"},\"legendOpen\":false}},\"P-14\":{\"vis\":{\"legendOpen\":true}},\"P-7\":{\"vis\":{\"legendOpen\":false}},\"P-9\":{\"vis\":{\"colors\":{\"Count\":\"#99440A\"},\"legendOpen\":true}}}",
"version" : 1
}
},
{
"_index" : ".kibana",
"_type" : "dashboard",
"_id" : "notifications",
"_score" : 1.0,
"_source" : {
"description" : "",
"hits" : 0,
"kibanaSavedObjectMeta" : {
"searchSourceJSON" : "{\"filter\":[{\"query\":{\"query_string\":{\"analyze_wildcard\":true,\"query\":\"*\"}}}]}"
},
"optionsJSON" : "{\"darkTheme\":true}",
"panelsJSON" : "[{\"col\":1,\"columns\":[\"Logger\",\"publisher\",\"severity_label\",\"event_type\",\"old_state\",\"old_task_state\",\"state\",\"new_task_state\",\"environment_label\",\"display_name\"],\"id\":\"search-notifications\",\"panelIndex\":1,\"row\":14,\"size_x\":12,\"size_y\":11,\"sort\":[\"Timestamp\",\"desc\"],\"type\":\"search\"},{\"col\":1,\"id\":\"NOTIFICATIONS-OVER-TIME-PER-SOURCE\",\"panelIndex\":2,\"row\":1,\"size_x\":6,\"size_y\":4,\"type\":\"visualization\"},{\"col\":7,\"id\":\"NOTIFICATIONS-OVER-TIME-PER-SEVERITY\",\"panelIndex\":3,\"row\":1,\"size_x\":6,\"size_y\":4,\"type\":\"visualization\"},{\"col\":7,\"id\":\"EVENT-TYPE-BREAKDOWN\",\"panelIndex\":4,\"row\":5,\"size_x\":6,\"size_y\":5,\"type\":\"visualization\"},{\"col\":1,\"id\":\"SOURCE-BREAKDOWN\",\"panelIndex\":5,\"row\":5,\"size_x\":6,\"size_y\":5,\"type\":\"visualization\"},{\"col\":1,\"id\":\"HOST-BREAKDOWN\",\"panelIndex\":6,\"row\":10,\"size_x\":6,\"size_y\":4,\"type\":\"visualization\"},{\"col\":7,\"id\":\"NOTIFICATIONS-PER-SEVERITY\",\"panelIndex\":7,\"row\":10,\"size_x\":6,\"size_y\":4,\"type\":\"visualization\"}]",
"timeFrom" : "now-1h",
"timeRestore" : true,
"timeTo" : "now",
"title" : "Notifications",
"uiStateJSON" : "{\"P-4\":{\"vis\":{\"legendOpen\":true}},\"P-7\":{\"vis\":{\"legendOpen\":false}}}",
"version" : 1
}
}
]
}
}
Afterwhich I was able to pull out the existing IDs with JQ:
I have a static mapping in Elasticsearch index. When a message doesn't match this mapping, it is discarded. Is there a way to route it to a default index for wrong messages?
To give you example, I have some fields with integer type:
"status_code": {
"type": "integer"
},
When a message contains a number
"status_code": 123,
it's ok. But when it is
"status_code": "abc"
it fails.
You can have ES do this triage pretty easily using ingest nodes/processors.
The main idea is to create an ingest pipeline with a convert processor for the status_code field and if the conversion doesn't work, you can add an on_failure condition which will direct the document at another index that you can later process.
So create the failures ingest pipeline:
PUT _ingest/pipeline/failures
{
"processors": [
{
"convert": {
"field": "status_code",
"type": "integer"
}
}
],
"on_failure": [
{
"set": {
"field": "_index",
"value": "failed-{{ _index }}"
}
}
]
}
Then when you index a document, you can simply specify the pipeline in parameter. Indexing a document with correct status code will succeed:
PUT test/doc/1?pipeline=failures
{
"status_code": 123
}
However, trying to index a document with a bad status code, will actually also succeed, but your document will be indexed in the failed-test index and not the test one:
PUT test/doc/2?pipeline=failures
{
"status_code": "abc"
}
After running these two commands, you'll see this:
GET failed-test/_search
{
"took" : 3,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 1,
"max_score" : 1.0,
"hits" : [
{
"_index" : "failed-test",
"_type" : "doc",
"_id" : "2",
"_score" : 1.0,
"_source" : {
"status_code" : "abc"
}
}
]
}
}
To sum up, you didn't have to handle that exceptional case in your client code and could fully leverage ES ingest nodes to achieve the same task.
You can set the parameter ignore malformed to ignore just the field with the type mismatch and not the whole document.
And you can try to combine it with multi-fields, that allows you to map the same value in different ways.
You will probably need something like this:
"status_code": {
"type": "integer",
"fields": {
"as_string": {
"type": "keyword"
}
}
}
This way you will have a field named status_code as an intenger and the same value in a field named status_code.as_string as a keyword, but you should test to see if really does what you want.
Use Strict mapping and you will be able to catch the exception raised by Elastic.
Below is the excerpt from Elastic docs:
By default, when a previously unseen field is found in a document, Elasticsearch will add the new field to the type mapping. This behaviour can be disabled, both at the document and at the object level, by setting the dynamic parameter to false (to ignore new fields) or to strict (to throw an exception if an unknown field is encountered).
As a part of Exception handling, you can push the message to some other index where dynamic mapping is enabled.
I'm new to ES and am struggling with nested aggregations.
Here is my dummy data object ([Here is my data object][1][1]: https://i.stack.imgur.com/X7oaM.png). I'm just trying to get the minimum cost out of the "modern" field.
I have read the following posts in regards to the problem I'm trying to solve. None of them have helped me solve the problem
- Elastic Search 6 Nested Query Aggregations
- https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-nested-aggregation.html
- https://madewithlove.be/elasticsearch-aggregations/
- https://iridakos.com/tutorials/2018/10/22/elasticsearch-bucket-aggregations.html
- https://github.com/elastic/elasticsearch/issues/9317
Moreover, I've searched all over stackoverflow and have had not had success (yes, I've tried just about every solution I've come across with no success).
According to the docs and the above posts and more, a nested aggregation should be ran as follows:
GET /loquesea/_search
{
"size": 0,
"aggs": {
"modern_costs": {
"nested": {
"path": "modern"
},
"aggs": {
"min_cost": {
"min": {
"field": "modern.cost1"
}
}
}
}
}
}
However, upon completion, this is what I get:
{
"took" : 1,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"skipped" : 0,
"failed" : 0
},
"hits" : {
"total" : 3,
"max_score" : 0.0,
"hits" : [ ]
},
"aggregations" : {
"modern_costs" : {
"doc_count" : 0
}
}
}
I've spent hours trying to make just a basic nested aggregation work. What am I doing wrong?
problem is solved. Turns out that since I didn't declare in the mapping the type of "cars" as "nested", the nested aggregation will not work. Reason being is that in Elastic Search, since the type was not declared as "nested", ES will treat "cars" as an object.
As stated in the ElasticSearch documentation:
In Elasticsearch, you have the ability to execute searches returning hits and at the same time return aggregated results separate from the hits all in one response. This is very powerful and efficient in the sense that you can run queries and multiple aggregations and get the results back of both (or either) operations in one shot avoiding network roundtrips using a concise and simplified API.
I want to execute searches returning hits when i have queries out for the aggregation. But i am not sure how can i achieve the above?
I am using the following query:
curl -XPOST 'localhost:9200/employee/_search?pretty' -d '
{
"size": 0,
"aggs": {
"group_by_domain": {
"terms": {
"field": "domain"
}
}
}
}'
and here is the result which i am getting,
{
"took" : 92,
"timed_out" : false,
"_shards" : {
"total" : 5,
"successful" : 5,
"failed" : 0
},
"hits" : {
"total" : 1000,
"max_score" : 0.0,
"hits" : [ ]
},
"aggregations" : {
"group_by_domain" : {
"doc_count_error_upper_bound" : 5,
"sum_other_doc_count" : 744,
"buckets" : [ {
"key" : "finance",
"doc_count" : 30
}]
}
}
}
As we can see that the hits array is empty. I am not sure how to get those hits array. Any suggestion?
the hits are empty because you have set the size of the returning query to 0 when you specify:
"size": 0,
you can remove size completely and in this case you'll get 10 hits that is the default or you can set the size you want, for instance if you specify 100 you'll get 100 hits in response. This is related to the search results.
Now, if you also want to get results in the aggregation you can use Top Hits Aggregation for that.
I want to count number of document returned as a result of a query with size limit. For example, I run following query:
curl -XGET http://localhost:9200/logs_-*/a_logs/_search?pretty=true -d '
{
"query" : {
"match_all" : { }
},
"size" : 5,
"from" : 8318
}'
and I get:
{
"took" : 5,
"timed_out" : false,
"_shards" : {
"total" : 159,
"successful" : 159,
"failed" : 0
},
"hits" : {
"total" : 8319,
"max_score" : 1.0,
"hits" : [ {
....
Total documents matching my query are 8319, but I fetched at max 5. Only 1 document was returned since I queried "from" 8318.
In the response, I do not know how many documents are returned. I want to write a query such that the number of documents being returned are also present in some field. Maybe some facet may help, but I could not figure out. Kindly help.
Your query :
{
"query" : {
"match_all" : { }
},
=> Means that you ask all your data
"size" : 5,
=> You want to display only 5 results
"from" : 8318
=> You start from the 8318 records
ElasticSearch respons :
....
"hits" : {
"total" : 8319,
...
=> Elastic search told you that there is 8319 results in his index.
You ask him all the result and you start from the 8318.
8319 - 8318 = 1 So you have 1 result.
Try by removing the from.
Looking through the documentation, it's not clear how to make the query return this -- if indeed the API supports it. If you just want to have the count of the returned hits, the easiest way seems to be to actually count them yourself after parsing the response.