convert field number to date on kibana, elastic search - elasticsearch

I've a field number on Elasticsearch via Kibana.
This is an example field:
"insert.datePeticio:1,546,185,770,733"
I need to convert this field to date, for visualize purpose.
How can I parse it to date mode?

As far as I understand, you have an integer field on Elasticsearch that stores date/time in milliseconds since the epoch. You would like to convert it to an appropriate date type to visualise on Kibana properly. In this case, I would recommend two solutions:
1) If you are able to define a mapping, use the the following for insert.datePeticio field:
"mappings": {
"_doc": {
"properties": {
"insert.datePeticio": {
"format": "epoch_millis",
"type": "date"
}
}
}
}
This allows Kibana to define and represent insert.datePeticio field as date even though the actual value is store in miliseconds as integer on Elasticsearch.
2) If not, so cannot make any changes on the original mapping, create a scripted field on Kibana as follows:
Go to Management > Index Patterns
Select the Index Pattern you want to modify
Select the index pattern’s Scripted Fields tab
Click Add Scripted Field
Fill the form referring to the image below, and Save it.
If you go to Kibana > Discover, you can see two fields together in different representations and types: insert.datePeticio: 1,546,185,770,733 and insert.datePeticio_UTC:December 30th 2018, 16:02:50.733. Since being a date type, the scripted field insert.datePeticio_UTC can be easily used to create visualisations based on date aggregations.
Note: Scripted fields compute data on the fly from the data in your Elasticsearch indices. Keep in mind that computing data on the fly with scripted fields can be very resource intensive and can have a direct impact on Kibana’s performance.

Related

elasticsearch date type mapping conflict

I have an index with a field I am storing date information in. The field is in conflict at the moment. As far as I can tell there are three kinds of values:
some documents don't have the field
some documents have the field like this in the JSON:
"timestamp": "2019-03-01T23:32:28Z"
other documents have the field like this in the JSON:
"timestamp": "1551206688760"
I want to fix the conflict.
I've tried doing a wildcard search and I get the following error:
failed to parse date field [*] with format [strict_date_optional_time||epoch_millis]
I have two questions, ultimately.
1) Is the core problem causing the conflict that when I tried to represent the timestamp in epoch_millis that I used a string rather than a number? IOW, "timestamp": 1551206688760 would have been good?
2) What is the proper way to fix this without simply tossing all the data out?
Unfortunately you will need to reindex.
Create new index with date mapping to provide multiple formats
"format": "yyyy-MM-dd HH:mm:ss||yyyy-MM-dd||epoch_millis"
Reindex your data
Use aliases

difference between a field and the field.keyword

If I add a document with several fields to an Elasticsearch index, when I view it in Kibana, I get each time the same field twice. One of them will be called
some_field
and the other one will be called
some_field.keyword
Where does this behaviour come from and what is the difference between both of them?
PS: one of them is aggregatable (not sure what that means) and the other (without keyword) is not.
Update : A short answer would be that type: text is analyzed, meaning it is broken up into distinct words when stored, and allows for free-text searches on one or more words in the field. The .keyword field takes the same input and keeps as one large string, meaning it can be aggregated on, and you can use wildcard searches on it. Aggregatable means you can use it in aggregations in elasticsearch, which resembles a sql group by if you are familiar with that. In Kibana you would probably use the .keyword field with aggregations to count distinct values etc.
Please take a look on this article about text vs. keyword.
Briefly: since Elasticsearch 5.0 string type was replaced by text and keyword types. Since then when you do not specify explicit mapping, for simple document with string:
{
"some_field": "string value"
}
below dynamic mapping will be created:
{
"some_field": {
"type" "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
As a consequence, it will both be possible to perform full-text search on some_field, and keyword search and aggregations using the some_field.keyword field.
I hope this answers your question.
Look at this issue. There is some explanation of your question in it. Roughly speaking some_field is analyzed and can be used for fulltext search. On the other hand some_field.keyword is not analyzed and can be used in term queries or in aggregation.
I will try to answer your questions one by one.
Where does this behavior come from?
It is introduced in Elastic 5.0.
What is the difference between the two?
some_field is used for full text search and some_field.keyword is used for keyword searching.
Full text searching is used when we want to include individual tokens of a field's value to be included in search. For instance, if you are searching for all the hotel names that has "farm" in it, such as hay farm house, Windy harbour farm house etc.
Keyword searching is used when we want to include the whole value of the field in search and not individual tokens from the value. For eg, suppose you are indexing documents based on city field. Aggregating based on this field will have separate count for "new" and "york" instead of "new york" which is usually the expected behavior.
From Elastic 5.0 onwards, strings now will be mapped both as keyword and text by default.

Elasticsearch aggregation by 7 fields

I need to do aggregation by 7 fields in elasticsearch, then retrieve data TopHits and do some calculations Sum and Avg. Is there any possibility to get latest buckets of hits and calculations without many loops/recursive?
According to Elasticsearch documentation:
"The terms aggregation does not support collecting terms from multiple fields in the same document. The reason is that the termsagg doesn’t collect the string term values themselves, but rather uses global ordinals to produce a list of all of the unique values in the field. Global ordinals results in an important performance boost which would not be possible across multiple fields.
There are two approaches that you can use to perform a terms agg across multiple fields:
Script
Use a script to retrieve terms from multiple fields. This disables the global ordinals optimization and will be slower than collecting terms from a single field, but it gives you the flexibility to implement this option at search time.
copy_to field
If you know ahead of time that you want to collect the terms from two or more fields, then use copy_to in your mapping to create a new dedicated field at index time which contains the values from both fields. You can aggregate on this single field, which will benefit from the global ordinals optimization."
Source: https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-bucket-terms-aggregation.html#_multi_field_terms_aggregation
EDIT: If you use copy_to field, there is not reason to index it since so you do not need to analyze it, for this you only have to change its mapping:
"metaFieldName" => [
"type" => "string",
"index" => "not_analyzed"
]

Range query for date of "long" field type in kibana

I have a field in ES type representing date with long type.
I am trying to filter data in kibana based on this field.
For example, get me the documents created in last seven days. But not able to come up with the solution.
In Kibana Visualization page, right now i have created like,
(events.time:[1458325800000 TO * ]) AND (event.type:"login")
But I need something like this
(events.time:[now-7d TO * ]) AND (event.type:"login")
I do not want to change the data type from 'long' to 'date' as it is not in my hands.

Field not searchable in ES?

I created an index myindex in elasticsearch, loaded a few documents into it. When I visit:
localhost:9200/myindex/mytype/1023
I noticed that my particular index has the following metadata for mappings:
mappings: {
mappinggroupname: {
properties: {
Aproperty: {
type: string
}
Bproperty: {
type: string
}
}
}
}
Is there some way to add "store:yes" and index: "analyzed" without having to reload/reindex all the documents?
Note that when i want to view a single document...
i.e. localhost:9200/myindex/mytype/1023
I can see the _source field contains all the fields of that document are and when I go to the "Browser" section of the head plugin it appears that all the columns are correct and corresponding to my fieldnames. So why is it that "stored" is not showing up in metadata? I can even perform a _search on them.
What is the difference between "stored":"true" versus the fact that I can see all my fields and values after indexing all my documents via the means I mention above?
Nope, no way! That's how your documents got indexed in the underlying lucene. The only way to change it is to reindex them all!
You see all those fields because you see the content of the special _source field in lucene, that's stored by default through elasticsearch. You are not storing all the fields separately but you do have the source document that you originally indexed through the _source, a single field that contains the whole document.
Generally the _source field is just enough, you don't usually need to configure every field as stored.
Also, the default is "index":"analyzed" if not specified for all the string fields. That means those fields are indexed and analyzed using the standard analyzer if not specified in the mapping. Therefore, as far as I can see from your mapping those two fields should be indexed, thus searchable.

Resources