How to make a field in Kibana numeric (from String) - elasticsearch

Question

I've inherited an ELK stack for logs and I'm still learning the ropes - I've been tasked with making two fields numeric on a certain type on our logstash indexes. Can't seem to figure out how to do this. Things I tried:
In the Kibana settings page, went to my logstash index and found the field. Went to edit on the controls tab, saw type listed as String (and it was immutable). Dropdown for format shows URL and String.
Went to one of my Elasticsearch hosts and found the grok rule for the document type, and found that they were indeed written to parse the field as a number. Example: %{NUMBER:response_code}
Ran out of ideas, since I don't know my way around the ELK stack.
Any help greatly appreciated, especially links to relevant documentation so I can understand what's going on. I'd be googling harder if I knew what to google.

Answer 1

Also note that %{NUMBER:response_code} doesn't make a number out of a string, it simply recognizes and parses a number present in a string, but the resulting response_code field is still a string, which you need to convert to number using a mutate/convert filter. grok will always parse a string into other smaller strings and it is your job to convert the resulting fields into the types you expect.
So you need to add this after your grok filter:
mutate {
convert => { "response_code" => "integer" }
}
From then on, the response_code in your event will be an integer and the logstash template used to create your daily logstash indices contains a specific dynamic template for integer fields. Note that the response_code field will be an integer only once the new logstash index is created, the existing indices will not change.

Answer 2

You will need to reindex your data. Because the Elasticsearch mapping (ie. schema) is already set to string for this field, you will not be able to index data as an integer within the same index.
A typical ELK setup will create rolling indices (per day or month), so it's possible to switch from string to interger between indices, but this is not recommended as it will interfere with long term aggregations and searches.
As you found out, changing the Grok rule will help with future data. Now, you need to pass all your existing data through Logstash again to apply the new ryles.
To do this, you can either pass the log files again, or have Logstash read from Elasticsearch using
input {
elasticsearch {
hosts => "localhost"
}
}
The newer versions of Elasticsearch should improve this by providing a native reindex API.

Answer 3

Try to view sample of documents:
curl -XGET 'localhost:9200/_search?q=opcode:userLessonComplexityPoll&pretty'
let say you see these docs:
{
"_index" : "myindex",
"_type" : "logs",
"_id" : "AWNoYI8pGmxxeL6jupEZ",
"_score" : 1.0,
"_source" : {
"production" : "0",
"lessonId" : "2144",
"opcode" : "userLessonComplexityPoll",
"courseId" : "45",
"lessonType" : "minitest",
...
So, try to convert in one document:
curl -XPOST 'localhost:9200/educa_stats-*/_update_by_query?pretty' -d '
{
"script": {
"lang": "painless",
"source": "if(ctx._source.lessonId instanceof String) { int lessonId = Integer.parseInt(ctx._source.lessonId); ctx._source.lessonId = (int)lessonId; }"
},
"query": {
"bool": {
"terms": {
"_id": ["AWNoYI8pGmxxeL6jupEZ", "AWMcRJYFGmxxeL6jucIZ"]
}
}
}
}'
success? Try to convert all documents by query:
curl -XPOST 'localhost:9200/educa_stats-*/_update_by_query?pretty' -d '
{
"script": {
"lang": "painless",
"source": "if(ctx._source.lessonId instanceof String) { int lessonId = Integer.parseInt(ctx._source.lessonId); ctx._source.lessonId = (int)lessonId; }"
},
"query": {
"bool": {
"must": [
{
"exists": {
"field": "lessonId"
}
}
]
}
}
}'
All fields lessonId will be converted from String type to int (-2^32 - 2^32) type. It's all.