We are using ElasticSearch to save and manage information on complex transactions. We might need to add more information for every transaction, on the near future.
How about including JSON doc version?
Is it possible for elastic search, to include different versions of JSON docs, to save and to search?
How does this affects performance on ElasticSearch?
It's completely possible, By default elastic uses the dynamic mappings for every new documents such as your JSON documents to index them. For each field in your documents elastic creates a table called inverted_index and the search queries executed against them so regardless of your field variation as long as you know which field you want to execute query the data throughput and performance will not be affected.
Is it possible to configure Elasticsearch to retrieve synonyms list from database/sql instead of file?
Unfortunately it is not possible.
ES supports file or online synonyms. It supports solr synonyms format..
doc
But you can add all the synonyms along with original field as an array in ES. While querying, you can use that array field and retrieve the original field.
The json data is stored in the elasticsearch. However, the type of the specific field of the data is not one but two or more types. So, if save it, I got a type error because of the stored other type data first.
I want to store the raw data without changing the data type. Is it possible to save except for the indexing of certain fields?
You can disable the indexing of specific fields with the enabled option. But this option can only be used at the root of the mapping or on "object" fields.
An other way is to set dynamic mapping to false for this index (documentation here), and manually create the mapping for the only fields you want to index.
Kibana's UI allows the user to create a scripted field which is stored as part of the index (screenshot below). How can that be done programatically? In particular, using either the NEST client or the Elasticsearch low level client.
Kibana UI for the Indice with the Scripted Fields tab highlighted
Note that I am not asking how to create add an expression/script field as part of a query, I'm specifically looking for how to add it as part of the Index when the mapping is created so that queries can reference it without having to explicitly include it.
Kibana dashboards are stored in the .kibana index. To export dashboards, you can query the Kibana index as you would any other index. For example, curl -XGET http://localhost:9200/.kibana/_search?type=dashboard&pretty would show the JSON for your dashboards. You could export the template, add the scripted field to the JSON, and then POST it again. Since Kibana uses a standard Elasticsearch index, the normal Elasticsearch API would apply to modifying Kibana dashboards. This may provide a little more clarification.
At the time of writing, current version 5.2 does not have an official way to do this.
This is how I do it:
Get index fields: GET /.kibana/index-pattern/YOUR_INDEX
Add your scripted field to _source.fields (as string, notice scaped quotation marks)
"fields":"[...,{\"name\":\"test\",\"type\":\"number\",\"count\":0,\"scripted\":true,\"script\":\"doc['area_id'].value\",\"lang\":\"painless\",\"indexed\":false,\"analyzed\":false,\"doc_values\":false,\"searchable\":true,\"aggregatable\":true}]"
Post back _source json to /.kibana/index-pattern/YOUR_INDEX
{
"title":"YOUR_INDEX",
"timeFieldName":"time",
"fields":"[...,{\"name\":\"test\",...}]"
}
I'm using java API to index data into ElasticSearch and generate graphs in Kibana.
I have a field named "Event_TS" which holds values of type long (time at which event was created in milliseconds). I could generate Date Histograms using it.
(I'm getting JSON document from a separate method.)
But, when I finally reindexed the whole data, Kibana is not identifying "Event_TS" as time-based anymore and hence I can't generate Date Histograms. How do I resolve this?