I'm actually trying to dislay all number after comma in my kibana's datatable but even with json input format, it does display as expected ...
Do you have an idea how to do this ?
Here for example I have 2.521 but in can be 0.632, or 0.194 ...
I only see 0 in Min, Max, Avg columns
In my C# code is a double and indexed as a number in Kibana index:
How to do this plz ?
Thank a lot and best regards
This usually means that your field has been mapped as integer or long. If that's the case, 0.632 is stored as 0 and 2.521 as 2.
You need to make sure that those fields are mapped as float or double in your mapping.
PS: you cannot change the mapping type once the index has been created, you need to create a new index and reindex your data.
You need to pre-create your index with the right mapping types before sending the first document:
PUT webapi-myworkspace-test
{
"mappings": {
"properties": {
"GraphApiResponseTime" : {
"type" : "double"
}
}
}
}
Related
I am trying to use the metricbeat http module to monitor F5 pools.
I make a request to the f5 api and bring back json, which is saved to kibana. But the json contains an array of pool members and I want to count the number which are up.
The advice seems to be that this can be done with a scripted field. However, I can't get the script to retrieve the array. eg
doc['http.f5pools.items.monitor'].value.length()
returns in the preview results with the same 'Additional Field' added for comparison:
[
{
"_id": "rT7wdGsBXQSGm_pQoH6Y",
"http": {
"f5pools": {
"items": [
{
"monitor": "default"
},
{
"monitor": "default"
}
]
}
},
"pool.MemberCount": [
7
]
},
If I try
doc['http.f5pools.items']
Or similar I just get an error:
"reason": "No field found for [http.f5pools.items] in mapping with types []"
Googling suggests that the doc construct does not contain arrays?
Is it possible to make a scripted field which can access the set of values? ie is my code or the way I'm indexing the data wrong.
If not is there an alternative approach within metricbeats? I don't want to have to make a whole new api to do the calculation and add a separate field
-- update.
Weirdly it seems that the number values in the array do return the expected results. ie.
doc['http.f5pools.items.ratio']
returns
{
"_id": "BT6WdWsBXQSGm_pQBbCa",
"pool.MemberCount": [
1,
1
]
},
-- update 2
Ok, so if the strings in the field have different values then you get all the values. if they are the same you just get one. wtf?
I'm adding another answer instead of deleting my previous one which is not the actual question but still may be helpful for someone else in future.
I found a hint in the same documentation:
Doc values are a columnar field value store
Upon googling this further I found this Doc Value Intro which says that the doc values are essentially "uninverted index" useful for operations like sorting; my hypotheses is while sorting you essentially dont want same values repeated and hence the data structure they use removes those duplicates. That still did not answer as to why it works different for string than number. Numbers are preserved but strings are filters into unique.
This “uninverted” structure is often called a “column-store” in other
systems. Essentially, it stores all the values for a single field
together in a single column of data, which makes it very efficient for
operations like sorting.
In Elasticsearch, this column-store is known as doc values, and is
enabled by default. Doc values are created at index-time: when a field
is indexed, Elasticsearch adds the tokens to the inverted index for
search. But it also extracts the terms and adds them to the columnar
doc values.
Some more deep-dive into doc values revealed it a compression technique which actually de-deuplicates the values for efficient and memory-friendly operations.
Here's a NOTE given on the link above which answers the question:
You may be thinking "Well that’s great for numbers, but what about
strings?" Strings are encoded similarly, with the help of an ordinal
table. The strings are de-duplicated and sorted into a table, assigned
an ID, and then those ID’s are used as numeric doc values. Which means
strings enjoy many of the same compression benefits that numerics do.
The ordinal table itself has some compression tricks, such as using
fixed, variable or prefix-encoded strings.
Also, if you dont want this behavior then you can disable doc-values
OK, solved it.
https://discuss.elastic.co/t/problem-looping-through-array-in-each-doc-with-painless/90648
So as I discovered arrays are prefiltered to only return distinct values (except in the case of ints apparently?)
The solution is to use params._source instead of doc[]
The answer for why doc doesnt work
Quoting below:
Doc values are a columnar field value store, enabled by default on all
fields except for analyzed text fields.
Doc-values can only return "simple" field values like numbers, dates,
geo- points, terms, etc, or arrays of these values if the field is
multi-valued. It cannot return JSON objects
Also, important to add a null check as mentioned below:
Missing fields
The doc['field'] will throw an error if field is
missing from the mappings. In painless, a check can first be done with
doc.containsKey('field')* to guard accessing the doc map.
Unfortunately, there is no way to check for the existence of the field
in mappings in an expression script.
Also, here is why _source works
Quoting below:
The document _source, which is really just a special stored field, can
be accessed using the _source.field_name syntax. The _source is loaded
as a map-of-maps, so properties within object fields can be accessed
as, for example, _source.name.first.
.
Responding to your comment with an example:
The kyeword here is: It cannot return JSON objects. The field doc['http.f5pools.items'] is a JSON object
Try running below and see the mapping it creates:
PUT t5/doc/2
{
"items": [
{
"monitor": "default"
},
{
"monitor": "default"
}
]
}
GET t5/_mapping
{
"t5" : {
"mappings" : {
"doc" : {
"properties" : {
"items" : {
"properties" : {
"monitor" : { <-- monitor is a property of items property(Object)
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
}
}
}
}
Is there a way to add an Elasticsearch data field to an index mapping, such that it always returns a constant numeric value?
I know I can just add a numeric datatype, and then reindex everything with the constant, but I would like to avoid reindexing, and I'd also like to be able to change the constant dynamically without reindexing.
Motivation: Our cluster has a lot of different indexes. We routinely search multiple indexes at once for various reasons. However, when searching multiple indices, our search logic still needs to treat each index slightly differently. One way we could do this is by adding a constant numeric field to each index, and then use that field in our search query.
However, because this is a constant, it seems like we should not need to reindex everything (seems pointless to add a constant value to every record).
You could use the _meta field for that purpose:
PUT index1
{
"mappings": {
"_meta": {
"constant": 1
},
"properties": {
... your fields
}
}
}
PUT index2
{
"mappings": {
"_meta": {
"constant": 2
},
"properties": {
... your fields
}
}
}
You can change that constant value anytime, without any need for reindexing anything. The value is stored at the index level and can be retrieved anytime by simply retrieve the index mapping with GET index1,index2/_mapping
I'm parsing log files which for simplicity's sake let's say will have the following format :
{"message": "hello world", "size": 100, "forward-to": 127.0.0.1}
I'm indexing these lines into an Elasticsearch index, where I've defined a custom mapping such that message, size, and forward-to are of type text, integer, and ip respectively. However, some log lines will look like this :
{"message": "hello world", "size": "-", "forward-to": ""}
This leads to parsing errors when Elasticsearch tries to index these documents. For technical reasons, it's very much untrivial for me to pre-process these documents and change "-" and "" to null. Is there anyway to define which values my mapping should treat as null ? Is there perhaps an analyzer I can write which works on any field type whatsoever that I can add to all entries in my mapping ?
Basically I'm looking for somewhat of the opposite of the null_value option. Instead of telling Elasticsearch what to turn a null_value into, I'd like to tell it what it should turn into a null_value. Also acceptable would be a way to tell Elasticsearch to simply ignore fields that look a certain way but still parse the other fields in the document.
So this one's easy apparently. Add the following to your mapping settings :
{
"settings": {
"index": {
"mapping": {
"ignore_malformed": "true"
}
}
}
}
This will still index the field (contrary to what I've understood from the documentation...) but it will be ignored during aggregations (so if you have 3 entries in an integer field that are "1", 3, and "hello world", an averaging aggregation will yield 2).
Keep in mind that because of the way the option was implemented (and I would say this is a bug) this still fails for and object that is entered as a concrete value and vice versa. If you'd like to get around that you can set the field's enabled value to false like this :
{
"mappings": {
"my_mapping_name": {
"properties": {
"my_unpredictable_field": {
"enabled": false
}
}
}
}
}
This comes at a price though, since this means the field won't be indexed, but the values entered will be still be stored so you can still accessing them by searching for that document through another field. This usually shouldn't be an issue as you likely won't be filtering documents based on the value of such an unpredictable field, but that depends on your specific case use. See here for the official discussion of this issue.
I am not asking the count for the search response. what I am asking is, size of the result(_source) that took Elasticsearch's hard-disk memory. is it possible to find such?. Why I am asking is, I need to find which type of source takes maximum size for an index. thanks in advance.
You can enable the _size field in your mapping. So this is data is created at index time.
{
"tweet" : {
"_size" : {"enabled" : true, "store" : true }
}
}
Check out the size field documentation.
Then you can return this field by adding it to the fields list in the query.
See the Fields documentation for how to do that.
I made a mistake at the time of first index creation for one field. Instead of "integer" data type mistakenly I assigned as "string" for the field "rating". But the data stored in the field was integer only. When I'm trying to calculate the average rating aggregation it was throwing an error because of the string data type.
Is there a way to change the data type of the field with out reindexing?
If not possible without reindex, how can I remove the rating field and add a rating field with "integer" data type?
Help me on this issue to resolve.
Update
Deleted the type in the index by using below command
curl -XDELETE 'http://localhost:9300/feedbacks_16/responses'
Deleted the type and created the type with the same name and changed the data type for my rating field and re indexed the entire data. Everything goes fine until reindexing. But the avg query not working.
Below is the error I'm getting :
{ "error": "SearchPhaseExecutionException[Failed to execute phase [query], all shards failed; shardFailures {[2NsGsPYLR2eP9p2zYSnKGQ][feedbacks-16][0]: ClassCastException[org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData cannot be cast to org.elasticsearch.index.fielddata.IndexNumericFieldData]}{[2NsGsPYLR2eP9p2zYSnKGQ][feedbacks_16][1]: ClassCastException[org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData cannot be cast to org.elasticsearch.index.fielddata.IndexNumericFieldData]}{[pcwXn3X-TceUO0ub29ZFgA][feedbacks_16][2]: RemoteTransportException[[tsles02][inet[/localhost:9300]][indices:data/read/search[phase/query]]]; nested: ClassCastException; }{[pcwXn3X-TceUO0ub29ZFgA][feedbacks_16][3]: RemoteTransportException[[tsles02][inet[/localhost:9300]][indices:data/read/search[phase/query]]]; nested: ClassCastException; }{[2NsGsPYLR2eP9p2zYSnKGQ][feedbacks_16][4]: ClassCastException[org.elasticsearch.index.fielddata.plain.PagedBytesIndexFieldData cannot be cast to org.elasticsearch.index.fielddata.IndexNumericFieldData]}]", "status": 500 }
Aside from a few exceptions, a mapping cannot be updated. There are some exceptions:
you can add new properties
you can promote a simple field to a multi-field
you can disable doc_values (but not enable them)
you can update the ignore_above parameter
So if you wish to transform your rating field from string to integer without recreating a new index, the only solution you have is to create a sub-field (e.g. called rating.int) of type integer.
Note that you'll still have to reindex your data in order to populate that new sub-field, though. However, if you do so, you'd be much better off simply re-creating a clean index from scratch and re-populating it.
1) You can change data type of field without reindexing but the problem is that It wont affect your data i.e rating field will remain string as documents are stored in immutable segments but newly added field will be integer but again that wont solve your problem
2) You could delete all documents from your current index and then change the mapping with PUT API like this
$ curl -X PUT 'http://localhost:9200/your_index/your_type/_mapping?ignore_conflicts=true' -d
'{
"your_type": {
"properties": {
"rating": {
"type": "integer"
}
}
}
}'
and then reindex
but better would be to create new index with above mapping and reindex with zero downtime