We have several different indices in ElasticSearch:
myindex1.messages.ttl60-${date:format=yyyy.MM.dd}
myindex2.messages.ttl60-${date:format=yyyy.MM.dd}
myindex3.messages.ttl60-${date:format=yyyy.MM.dd}
All of them shares the same scheme and is used to log events.
Now I want to create ONE alert in BOSUN for all listed indices, but I don't want to write their names explicitly.
Can I have some kind of pattern matching for indices just like we have in Kibana: *messages*?
I tried esindices expression, but it requires literal names of indices.
Maybe you can just combine all the indices with a template into an alias on elastic side.
Related
We're using ElasticSearch and we have two different indexes with different data. Recently, we wanted to make a query that needs data from both indexes. ES allows to search through multiple indexes: /index1,index2/_search. The problem is that both indexes have properties with the same name and there could be collisions because ES doesn't know on which index to search.
How can we tell ES to look up a property from concrete index?
For example: index1.myProperty and index2.otherProperty
For Kibana server decommissioning purposes, I want to get a list of index patterns which never had any single document and had documents.
How to achieve this using Kibana only?
I tried this but it doesn't give the list based on the document count.
GET /_cat/indices
Also in individual level getting the count to check the documents are there is time consuming .
GET index-pattern*/_count
You can try this. V is for verbose and s stands for sort.
GET /_cat/indices?v&s=store.size:desc
From the docs :
These metrics are retrieved directly from Lucene, which {es} uses internally to power indexing and search. As a result, all document counts include hidden nested documents.
I have Kibana and ES. I have many indexes. I am using message field in ElasticSearch. My goal is to mask all IP addresses, which I already do using Logstash.
Now, given the fact there are many different indexes, and also different log types, I would like to run either Kibana or ES query for any occurence of IP. Just in case, that I missed any of them. Also, I would like to do it for email format as well.
Question is, how can I run IP/email regex search on ElasticSearch or Kibana?
Message field is string type, and is indexed.
I have found what I was looking for. In my case this approach is valid, since I do not care about performance. This was just a test to make sure I don't 'leak' information.
ElasticSearch regex query.
I have imported weblogs into Elasticsearch via Logstash. This has completed successfully.
I have a field in the log file (clientip) that is always populated and another field that is sometimes populated (trueclientip). I want to aggregate based on the coalescing of the two; e.g. if trueclientip is not empty then use that otherwise use clientip.
How can I do this with the Visualisation in Kibana? Do I need to generate a scripted field or is there another approach?
Thanks.
Define a scripted field that should have this formula: doc['trueclientip'].value ? doc['trueclientip'].value : doc['clientip'].value and use this in your aggregations.
But, there is a downside to this scripted fields functionality AND the ip type: it seems what you get back from the script is the number itself (which is logic because the scripted fields in Kibana 4 only use Lucene expressions as a language), not the string representation. IPs internally are actually long numbers in Lucene.
For example, 127.0.0.1 is represented internally as 2130706433. And this is what you will see in Visualize.
Is not ideal, indeed, and it would be good to have a more advanced scripting language in scripted fields, but a github issue already exists.
I'm wondering how exactly do index aliases work behind the scenes in Elasticsearch?
Does an alias have a separate copy of the data for each index it is linked to? Or is it only aware of the index names, and the not the data within each index?
If this is the case, are aggregations much slower when performed on an alias with many linked indices?
From the Index Aliases Elasticsearch reference:
APIs in elasticsearch accept an index name when working against a specific index, and several indices when applicable. The index aliases API allow to alias an index with a name, with all APIs automatically converting the alias name to the actual index name. An alias can also be mapped to more than one index, and when specifying it, the alias will automatically expand to the aliases indices. An alias can also be associated with a filter that will automatically be applied when searching, and routing values.
So based on this it is only aware of the index names and not the data within each index. An aggregation could be slower when performed against an alias that spans multiple indexes. Because as far as I know in order to perform the aggregation action, Elasticsearch must collect the dataset to perform the aggregation function(s) against.