I have an index in Elasticsearch with data that looks like this:
"_source": {
"segments": [
{
"segmentType": "Indirect",
"segmentCount": 100
},
{
"segmentType": "Direct",
"segmentCount": 20
}
]
}
I want to create a pie chart in Kibana where it takes the sum of segment count of each segment type. Currently, I only have the data above. So the pie chart should be split about: 83% for indirect and about 17% for direct. However, when I try to create a pie chart it is being split by 50% each. For slice size, I'm doing a sum of segment count and for split slices I'm doing a terms aggregation by the segment count field.
How can I achieve the result I want?
Old post but in case someone is still looking.
As far as I know it is not possible with the current version of Kibana (7.11) to create a pie chart using "sum of field value" to split slices, which is my understanding of what you were trying to do here.
In recent kibana versions the "Percentage Bar" visualization seems to be the way to present the percentage split you seek.
There is a related but old post (2017) on the elastic forum discussing 2 workaround options.
https://discuss.elastic.co/t/simple-pie-chart-question/92749
Ingesting the data (in these case it would be Indirect/Direct) as separate documents to be able to use the available aggregations.
Creating an alternate visualization type, though I believe the more recent "Percentage bar" would fit the purpose.
Related
We are using Spring Data Elasticsearch to build a 'fan out on read' user content feed. Our first attempt is currently showing content based on keyword matching and latest content using NativeSearchQueryBuilder.
We want to further improve the relevancy order of what is shown to the user based on additional factors (e.g. user engagement, what currently the user is working on etc).
Can this custom ordering be done using NativeSearchQueryBuilder or do we get more control using a painless script? If it's a painless script, can we call this from Spring Data ElasticSearch?
Any examples, recommendations would be most welcome.
Elasticsearch orders it result by it relevance-score (which marks a result relevancy to your search query), think that each document in the result set includes a number which signifies how relevant the document is to the given query.
If the data you want to change your ordering upon is part of your indexed data (document fields for example), you can use QueryDSL, to boost the _score field, few options I can think on:
boost a search query dependent on it criteria: a user searches for a 3x room flat but 4x room in same price would be much better match, then we can: { "range": { "rooms": { "gte": 4, "boost": 1 }}}
field-value-factor you can favor results by it field value: more 'clicks' by users, more 'likes', etc..,
random-score if you want randomness in your results: different
result every time a user refreshes your page or you can mix with existing scoring.
decay functions (Gauss!) to boost/unboost results that are close/far to our central point. lets say we want to search apartments and our budget is set to 1700. { "gauss": { "price": { "origin": "1700", "scale": "300" } } } will give us a feeling on how close we are to our budget of 1,700. any flat with much higher prices (let's say 2,300) - would get much more penalized by the gauss function - as it is far from our origin. the decay and the behavior of gauss function - will separate our results accordingly to our origin.
I don't think this has any abstraction on spring-data-es and I would use FunctionScoreQueryBuilder with the NativeSearchQueryBuilder.
I have a project in kibana integrated with elastic search.
In Kibana page I am displaying a chart with X(months) and Y(Euro) values.
I want to show a line in the chart that will show the average Euro value of all data.
For the moment I add a manual value to show the horizontal line in the chart. Chart example I want to show
I want to get average value automatically from my data in elastic search. Is there any option to do this task?
Thank you
Considering it is timeseries data, timelion can be used.
I have created dummy data as follows:
POST /balance_new/doc?pretty
{
"#timestamp": "2018-01-14T12:32:50.548Z",
"amount":136.5
}
There are more entries present like this.
Timelion query:
.es(index='balance_new', timefield='#timestamp', metric=avg:amount).range(135,140).title('Average EUR Monthly').yaxis(label='Average EUR'),
.es(index='balance_new', timefield='#timestamp', metric=avg:amount).aggregate(function=avg)
Graph look like:
You can read more about timelion here: https://www.elastic.co/guide/en/kibana/current/timelion.html
In Kibana visualization (bar chart), i have created the bar chart for one of the index pattern (eg: Aircel), I have created the another index pattern (eg: Nationwide), having the similar columns for both index patterns with different data.
My Question is how can create a common visualization of Bar char (called as bar chart template) so that it can be used for both Aircel and Nationwide indexes.
can you please help me on this. Thanks in advance.
You can not use one visualization for two indexes. You can save them as same index, but two different types and then create visualization for both types on same bar chart. If you do not want to do this, you can create a dashboard and add both bar charts from different indexes there, so you will have them on the same page, but not on the same visualization
Hope that helps
You can actually accommodate data from multiple indices inside the same Kibana visualization.
Create an index that represents multiple indices having similar fields or the indices you want to compare by naming "*" or anything like that.
Create the Y axis using "counts" aggregation or "sum" of "total" or whatever aggregation used in your data.
Create the X axis first using "Terms" aggregation using your data specified fields. Then add a sub-bucket "Split Series" upon that using "Terms" aggregation with the Field as "_index" or "_type" according to your data.
I have an elasticsearch index (index1) in which I have one type (type1). I added documents to type1 and ran a search on it:
POST /index1/type1/_search
{
"query": {
"match": {
"keyword": "quick brown fox"
}
}
}
I get a result set back with scores that generally range between .03 and 1.
Then I add another type (type2) to index1 and add some documents to it. When I run the exact same search again, I get the same documents back, but they all have different scores, now ranging from 2 and 5. Ideally, the scores of these documents would not change even after adding documents to type2.
Any ideas as to why this happening? I am running a search on type1, yet adding documents to type2 seems to influence the scoring of the results. Is there anyway to stop this from happening?
I am using v1.1.2 of elasticsearch. I should also mention, I'm working with a pretty small dataset (less than 1000 docs).
Elasticsearch scoring is detailed here, but basically what you are running into is that the inverse document frequency of some of your terms is changing based on what you are indexing into type2 (which is still in the same INDEX as type1). The change in IDF changes the relevancy of your search terms.
The only way you could avoid it is to have separate indexes for type1 and type2 (and then if you need to search across both, your search would need to pass in both indexes).
The scores really have no deep meaning though and really should only be used as a relative indication that some results are better than others.
Is it possible to query for a distinct/unique count of a field using Kibana? I am using elastic search as my backend to Kibana.
If so, what is the syntax of the query? Heres a link to the Kibana interface I would like to make my query: http://demo.kibana.org/#/dashboard
I am parsing nginx access logs with logstash and storing the data into elastic search. Then, I use Kibana to run queries and visualize my data in charts. Specifically, I want to know the count of unique IP addresses for a specific time frame using Kibana.
For Kibana 4 go to this answer
This is easy to do with a terms panel:
If you want to select the count of distinct IP that are in your logs, you should specify in the field clientip, you should put a big enough number in length (otherwise, it will join different IP under the same group) and specify in the style table. After adding the panel, you will have a table with IP, and the count of that IP:
Now Kibana 4 allows you to use aggregations. Apart from building a panel like the one that was explained in this answer for Kibana 3, now we can see the number of unique IPs in different periods, that was (IMO) what the OP wanted at the first place.
To build a dashboard like this you should go to Visualize -> Select your Index -> Select a Vertical Bar chart and then in the visualize panel:
In the Y axis we want the unique count of IPs (select the field where you stored the IP) and in the X axis we want a date histogram with our timefield.
After pressing the Apply button, we should have a graph that shows the unique count of IP distributed on time. We can change the time interval on the X axis to see the unique IPs hourly/daily...
Just take into account that the unique counts are approximate. For more information check also this answer.
Be aware with Unique count you are using 'cardinality' metric, which does not always guarantee exact unique count. :-)
the cardinality metric is an approximate algorithm. It is based on the
HyperLogLog++ (HLL) algorithm. HLL works by hashing your input and
using the bits from the hash to make probabilistic estimations on the
cardinality.
Depending on amount of data I can get differences of 700+ entries missing in a 300k dataset via Unique Count in Elastic which are otherwise really unique.
Read more here: https://www.elastic.co/guide/en/elasticsearch/guide/current/cardinality.html
Create "topN" query on "clientip" and then histogram with count on "clientip" and set "topN" query as source. Then you will see count of different ips per time.
Unique counts of field values are achieved by using facets. See ES documentation for the full story, but the gist is that you will create a query and then ask ES to prepare facets on the results for counting values found in fields. It's up to you to customize the fields used and even describe how you want the values returned. The most basic of facet types is just to group by terms, which would be like an IP address above. You can get pretty complex with these, even requiring a query within your facet!
{
"query": {
"match_all": {}
},
"facets": {
"terms": {
"field": "ip_address"
}
}
}
Using Aggs u can easily do that.
Writing down query for now.
GET index/_search
{
"size":0,
"aggs": {
"source": {
"terms": {
"field": "field",
"size": 100000
}
}
}
}
This would return the different values of field with there doc counts.
For Kibana 7.x, Unique Count is available in most visualizations.
For example, in Lens:
In aggregation based visualizations:
And even in TSVB (supporting normal fields as well as Runtime Fields, Scripted Fields are not supported):