Doc stored in ES index is like this:
{
"product_id": "something_unique",
"price": 25
}
The price section can also be null at times. (This is just as assumed db structure, as I don't want to list all the actual attributes which are being used in elasticsearch).
Now when I try to visualize this in Kibana I get results like this:
The visualized data is only for 3 entries whereas we have data for 11 products in the ES system. The other 8 entries have price as null. I want to showcase this info in kibana graph so that the admin can take proper action for this.
So, is there any way I can show the details of docs which have price as null ?
I am new to Elasticsearch and couldn't find solution to this anywhere else therefore posting it here. Please don't come up with comments like "null values are not supposed to be visualized , I have implemented the /_xpack/sql api but its not very handy".
I am using Elasticsearch 6.5
Related
I'm currently trying to create a graph on Grafana to monitor the status of my servers, however, I can't seem to find a way to use the value of a field as the value to be displayed on the graph. (Datasource is ElasticSearch)
The following "document" is going to be sent to GrayLog (which saves to Elastic) every 1 minute for an array of regions.
{
"region_key": "some_key",
"region_name": "Some Name",
"region_count": 1610
}
By using the following settings, I can get Grafana to display the count of messages it received for each region, however, I want to display the number on the region_count field instead.
Result:
How can I accomplish this? is this even possible using Elastic as the datasource?
1) Make sure that your document includes a timestamp in ElasticSearch.
2) In the Query box, provide the Lucene query which narrows down the documents to only those related to this metric
3) In the Metric line, press "Count" and change that to one which takes a specific field: for example, "Average"
4) Next to the "Average" box will appear "select field", which is a dropdown of the available fields. If you see unexpected fieldnames here, it's probably because your Lucene query isn't specific enough. (Kibana can be useful for getting this query right)
I am using Elasticsearch 6.2.
I have an index products with index_type productA having data with following structure:
{
"id": 1,
"parts": ["part1", "part2",...]
.....
.....
}
Now during the query time, I want to add or project a field parts_count to the response which simply represents the number of parts i.e the length of parts array. Also, if possible, I would also like to sort the documents of productA based on the generated field parts_count.
I have gone through most of the docs but haven't found a way to achieve this.
Note:
I don't want to update the mapping and add dynamic fields. I am not sure if Elasticsearch allows it. I just wanted to mention it.
Did you read about Script Fields and on Script Based Sorting?
I think you should be able to achieve both things and this not require any mapping updates.
I was successfully able to get data from an individual index, but now I will have to get the data from 3 indexes, those 3 indexes are of different types (Student, Employee, School). With a single query trigger can I be able to get data from all the three index.
In es you can specify that by URL
POST /gb,us/user,tweet/_search
{
//YOur query
}
In your case if you have one type per index it could be
POST /Student,Employee,School/_search
More info here
In Java according to this you should have smth like this
QueryBuilders.indicesQuery(queryBuilder, "product-a", "product-b");
I have records in Elasticsearch as vlabs/usage - 13000 records . I was able to draw dashboards from this.
To the same index I added a new doc-type (feedback) as vlabs/feedback - 1000 records.
How can I draw a separate Kibana dashboard for feedback records.
I can see records of both doc_type in single section in Discover section.
You can see records from different type of same index in single Discover section.
Using can use _type: doc_typeA AND _type: doc_typeB to get results but they've to be from same index.
reference link: https://www.elastic.co/guide/en/beats/packetbeat/current/kibana-queries-filters.html
Document type in elasticsearch is identified by a filed named _type. Say that you have an index designation with multiple document types Manager, Engineer, SupportEngineer as shown below -
designation/Manager
designation/Engineer
designation/SupportEngineer
Just write _type:Manager in kibana query box and you get all records listed in the Discovery section.
Is it possible to query for a distinct/unique count of a field using Kibana? I am using elastic search as my backend to Kibana.
If so, what is the syntax of the query? Heres a link to the Kibana interface I would like to make my query: http://demo.kibana.org/#/dashboard
I am parsing nginx access logs with logstash and storing the data into elastic search. Then, I use Kibana to run queries and visualize my data in charts. Specifically, I want to know the count of unique IP addresses for a specific time frame using Kibana.
For Kibana 4 go to this answer
This is easy to do with a terms panel:
If you want to select the count of distinct IP that are in your logs, you should specify in the field clientip, you should put a big enough number in length (otherwise, it will join different IP under the same group) and specify in the style table. After adding the panel, you will have a table with IP, and the count of that IP:
Now Kibana 4 allows you to use aggregations. Apart from building a panel like the one that was explained in this answer for Kibana 3, now we can see the number of unique IPs in different periods, that was (IMO) what the OP wanted at the first place.
To build a dashboard like this you should go to Visualize -> Select your Index -> Select a Vertical Bar chart and then in the visualize panel:
In the Y axis we want the unique count of IPs (select the field where you stored the IP) and in the X axis we want a date histogram with our timefield.
After pressing the Apply button, we should have a graph that shows the unique count of IP distributed on time. We can change the time interval on the X axis to see the unique IPs hourly/daily...
Just take into account that the unique counts are approximate. For more information check also this answer.
Be aware with Unique count you are using 'cardinality' metric, which does not always guarantee exact unique count. :-)
the cardinality metric is an approximate algorithm. It is based on the
HyperLogLog++ (HLL) algorithm. HLL works by hashing your input and
using the bits from the hash to make probabilistic estimations on the
cardinality.
Depending on amount of data I can get differences of 700+ entries missing in a 300k dataset via Unique Count in Elastic which are otherwise really unique.
Read more here: https://www.elastic.co/guide/en/elasticsearch/guide/current/cardinality.html
Create "topN" query on "clientip" and then histogram with count on "clientip" and set "topN" query as source. Then you will see count of different ips per time.
Unique counts of field values are achieved by using facets. See ES documentation for the full story, but the gist is that you will create a query and then ask ES to prepare facets on the results for counting values found in fields. It's up to you to customize the fields used and even describe how you want the values returned. The most basic of facet types is just to group by terms, which would be like an IP address above. You can get pretty complex with these, even requiring a query within your facet!
{
"query": {
"match_all": {}
},
"facets": {
"terms": {
"field": "ip_address"
}
}
}
Using Aggs u can easily do that.
Writing down query for now.
GET index/_search
{
"size":0,
"aggs": {
"source": {
"terms": {
"field": "field",
"size": 100000
}
}
}
}
This would return the different values of field with there doc counts.
For Kibana 7.x, Unique Count is available in most visualizations.
For example, in Lens:
In aggregation based visualizations:
And even in TSVB (supporting normal fields as well as Runtime Fields, Scripted Fields are not supported):