I'm trying to use Visualize feature in Kibana to plot monthly date_histogram graph that counts # of messages in my system. Message type has a sent_at field that is stored as number since epoch time.
Although I can do that just fine with elasticsearch query
POST /_all/message/_search?size=0
{
"aggs" : {
"monthly_message" : {
"date_histogram" : {
"field" : "sent_at",
"interval" : "month"
}
}
}
}
I ran into a problem in Kibana saying No Compatible Fields: The "myindex" index pattern does not contain any of the following field types: date
Is there a way to get Kibana to use number field as date?
Not to my knowledge, Kibana will use the index mapping in order to find out date fields, if no date fields can be found, then Kibana won't be able to infer one from the other number fields.
What you can do is to add another field called sent_at_date to your mapping, then use the update-by-query API in order to copy the sent_at field to that new field and finally to recreate your index pattern in Kibana.
It goes basically like this:
# 1. add a new field to your mapping
PUT myindex/_mapping/message
{
"properties": {
"sent_at_date": {
"type": "date"
}
}
}
# 2. update all your documents
POST myindex/_update_by_query
{
"script": {
"source": "ctx._source.sent_at_date = ctx._source.sent_at"
}
}
And finally recreate your index pattern in Kibana. You should see a new field called sent_at_date of type date that you can use in Kibana.
Related
I have an index that contains a "createdAt" string field I would like to convert to date.
I'm trying to that via the UI and since scripted fields are deprecated I understand I should use runtime fields.
I've figuired out to convert a string to date object and while it works for actual runtime queries, If i set a field using Index Pattern settings, the values don't seem to be shown on Kibana.
Here's how I setup the field:
And while the same code works, if I try to visualize the data in Kibana I see "no results found".
I don't understand where the issue is as the following query presents the field just fine:
GET mails/_search
{
"runtime_mappings": {
"exampleColumn": {
"type": "date",
"script": {
"source":
"""emit(new SimpleDateFormat('yyyy-mm-dd HH:mm:ss').parse(doc['createdAt.keyword'].value).getTime())"""
}
}
},
"fields" : ["exampleColumn"]
}
Does someone know what I'm doing wrong?
Any help will be appritiated.
Imagine that I have a specific data string and a specific query. The simple way to check that the query matches the data is to load the data into the Elastic index and run the online query. But can I do it without putting it into the index?
Maybe there are some open-source libraries that implement the Elastic search functionality offline, so I can call something like getScore(data, query)? Or it's possible to implement by using specific API endpoints?
Thanks in advance!
What you can do is to leverage the percolator type.
What this allows you to do is to store the query instead of the document and then test whether a document would match the stored query.
For instance, you first create an index with a field of type percolator that will contain your query (you also need to add in the mapping any field used by the query so ES knows what their types are):
PUT my_index
{
"mappings": {
"properties": {
"query": {
"type": "percolator"
},
"message": {
"type": "text"
}
}
}
}
Then you can index a real query, like this:
PUT my_index/_doc/match_value
{
"query" : {
"match" : {
"message" : "bonsai tree"
}
}
}
Finally, you can check using the percolate query if the query you've just stored would match
GET /my_index/_search
{
"query" : {
"percolate" : {
"field" : "query",
"document" : {
"message" : "A new bonsai tree in the office"
}
}
}
}
So all you need to do is to only store the query (not the documents), and then you can use the percolate query to check if the documents would have been selected by the query you stored, without having to store the documents themselves.
When a time-based index is added to kibana, you have to pick the field that will act as a time field. If you want to switch from one field to another, normally I would delete the index and re-add it back. But you end up loosing scripted fields and filed formatting this way.
Is there any way to modify the existing index time field without loosing scripted fields/formatting?
It can probably be done by messing around directly with /.kibana/index-pattern/index_pattern_name but all my attempts with changing timeFieldName directly ended up dropping scripted fields.
This is what worked for me on Kibana 7.7.0 (Elastic Cloud):
find the {id} of the document whose title field corresponds to the index you want to make change
GET .kibana/_search
{
"query": {
"match": {
"index-pattern.title": "{NAME_OF_THE_INDEX}"
}
}
}
change the timefield with following code
POST .kibana/_update/{id}
{
"doc": {
"index-pattern": {
"timeFieldName" : "{NEW_TIME_FIELD_NAME}"
}
}
}
The easiest way seems to update the corresponding document:
POST /.kibana/index-pattern/YOUR_INDEX_NAME/_update
{
"doc": {
"timeFieldName": "NEW_TIME_FIELD_NAME"
}
}
It should preserve scripted fields.
This doesn't seem to work on Kibana 5.
Instead, following is the way worked on Kibana 5.
1. find the {id} of the document whose title field corresponds to the index you want to make change
GET .kibana/index-pattern/_search
{
"_source" : "_id",
"query" : {
"match" : {
"title": "{NAME_OF_THE_INDEX}"
}
}
}
2. change the timefield with following code
POST /.kibana/index-pattern/{id}/_update
{
"doc": {
"timeFieldName" : "{NEW_TIME_FIELD_NAME}"
}
}
This worked fine with me on Kibana 5.
I've inherited an ELK stack for logs and I'm still learning the ropes - I've been tasked with making two fields numeric on a certain type on our logstash indexes. Can't seem to figure out how to do this. Things I tried:
In the Kibana settings page, went to my logstash index and found the field. Went to edit on the controls tab, saw type listed as String (and it was immutable). Dropdown for format shows URL and String.
Went to one of my Elasticsearch hosts and found the grok rule for the document type, and found that they were indeed written to parse the field as a number. Example: %{NUMBER:response_code}
Ran out of ideas, since I don't know my way around the ELK stack.
Any help greatly appreciated, especially links to relevant documentation so I can understand what's going on. I'd be googling harder if I knew what to google.
Also note that %{NUMBER:response_code} doesn't make a number out of a string, it simply recognizes and parses a number present in a string, but the resulting response_code field is still a string, which you need to convert to number using a mutate/convert filter. grok will always parse a string into other smaller strings and it is your job to convert the resulting fields into the types you expect.
So you need to add this after your grok filter:
mutate {
convert => { "response_code" => "integer" }
}
From then on, the response_code in your event will be an integer and the logstash template used to create your daily logstash indices contains a specific dynamic template for integer fields. Note that the response_code field will be an integer only once the new logstash index is created, the existing indices will not change.
You will need to reindex your data. Because the Elasticsearch mapping (ie. schema) is already set to string for this field, you will not be able to index data as an integer within the same index.
A typical ELK setup will create rolling indices (per day or month), so it's possible to switch from string to interger between indices, but this is not recommended as it will interfere with long term aggregations and searches.
As you found out, changing the Grok rule will help with future data. Now, you need to pass all your existing data through Logstash again to apply the new ryles.
To do this, you can either pass the log files again, or have Logstash read from Elasticsearch using
input {
elasticsearch {
hosts => "localhost"
}
}
The newer versions of Elasticsearch should improve this by providing a native reindex API.
Try to view sample of documents:
curl -XGET 'localhost:9200/_search?q=opcode:userLessonComplexityPoll&pretty'
let say you see these docs:
{
"_index" : "myindex",
"_type" : "logs",
"_id" : "AWNoYI8pGmxxeL6jupEZ",
"_score" : 1.0,
"_source" : {
"production" : "0",
"lessonId" : "2144",
"opcode" : "userLessonComplexityPoll",
"courseId" : "45",
"lessonType" : "minitest",
...
So, try to convert in one document:
curl -XPOST 'localhost:9200/educa_stats-*/_update_by_query?pretty' -d '
{
"script": {
"lang": "painless",
"source": "if(ctx._source.lessonId instanceof String) { int lessonId = Integer.parseInt(ctx._source.lessonId); ctx._source.lessonId = (int)lessonId; }"
},
"query": {
"bool": {
"terms": {
"_id": ["AWNoYI8pGmxxeL6jupEZ", "AWMcRJYFGmxxeL6jucIZ"]
}
}
}
}'
success? Try to convert all documents by query:
curl -XPOST 'localhost:9200/educa_stats-*/_update_by_query?pretty' -d '
{
"script": {
"lang": "painless",
"source": "if(ctx._source.lessonId instanceof String) { int lessonId = Integer.parseInt(ctx._source.lessonId); ctx._source.lessonId = (int)lessonId; }"
},
"query": {
"bool": {
"must": [
{
"exists": {
"field": "lessonId"
}
}
]
}
}
}'
All fields lessonId will be converted from String type to int (-2^32 - 2^32) type. It's all.
My logstash service sends the logs to elasticsearch as daily indices.
elasticsearch {
hosts => [ "127.0.0.1:9200" ]
index => "%{type}-%{+YYYY.MM.dd}"
}
Does Elasticsearch provides the API to lookup the indices before specific date?
For example, how could I get the indices created before 2015-12-15 ?
The only time I really care about what indexes are created is when I want to close/delete them using curator. Curator has "age" type features built in, if that's also your use case.
I think you are looking for Indices Query have a look here
Here is an example:
GET /_search
{
"query": {
"indices" : {
"query": {
"term": {"description": "*"}
},
"indices" : ["2015-01-*", "2015-12-*"],
"no_match_query": "none"
}
}
}
Each index has a creation_date field.
Since the number of indices is supposed to be quite small there's no such feature as 'searching for indices'. So you just get their metadata and filter them inside your app. The creation_date is also available via _cat API.