For example, a JSON file with key-value pair where the key name has a dot in between it. When this file is uploaded, the dot is treated as next line \n and the name will split into two properties. I tried to use mapper.allow_dots_in_name=True in the setting but no effect.
Similar question posted by someone else, but no reply https://discuss.elastic.co/t/disable-expansion-of-field-names-with-dots-in-mapping/84761
Appreciate if anyone could help.
Elasticsearch 2.4 includes a property where the field names can include a dot. And the field will not get converted to object style mapping.
This setting can be enabled by
export ES_JAVA_OPTS="-Dmapper.allow_dots_in_name=true"
But from 5.x, it is not possible to get a field value with dots without converting it into object mapping. If you index a field like abc.foo.bar (with no explicit mapping). This will get converted to
{
"mappings": {
"properties": {
"abc": {
"properties": {
"foo": {
"properties": {
"bar": {
"type": "long"
}
}
}
}
}
}
}
}
It is best to avoid dots in the field names. You can refer to this documentation, to know more about this
Related
I have successfully ingested data using the XML filter plugin from Logstash to Elasticsearch, however all the field types are of the type "text."
Is there a way to manually or automatically specify the correct type?
I found the following technique good for my use:
Logstash would filter the data and change a field from the default - text to whatever form you want. The documentation would be found here. The example given in the documentation is:
filter {
mutate {
convert => { "fieldname" => "integer" }
}
}
This you add in the /etc/logstash/conf.d/02-... file in the body part. I believe the downside of this practice is that from my understanding it is less recommended to alter data entering the ES.
After you do this you will probably get the this problem. If you have this problem and your DB is a test DB that you can erase all old data just DELETE the index until now that there would not be a conflict (for example you have a field that was until now text and now it is received as date there would be a conflict between old and new data). If you can't just erase the old data then read into the answer in the link I linked.
What you want to do is specify a mapping template.
PUT _template/template_1
{
"index_patterns": ["te*", "bar*"],
"settings": {
"number_of_shards": 1
},
"mappings": {
"type1": {
"_source": {
"enabled": false
},
"properties": {
"host_name": {
"type": "keyword"
},
"created_at": {
"type": "date",
"format": "EEE MMM dd HH:mm:ss Z YYYY"
}
}
}
}
}
Change the settings to match your needs such as listing the properties to map what you want them to map to.
Setting index_patterns is especially important because it tells elastic how to apply this template. You can set an array of index patterns and can use * as appropriate for wildcards. i.e logstash's default is to rotate by date. They will look like logstash-2018.04.23 so your pattern could be logstash-* and any that match the pattern will receive the template.
If you want to match based on some pattern, then you can use dynamic templates.
Edit: Adding a little update here, if you want logstash to apply the template for you, here is a link to the settings you'll want to be aware of.
I've already created an index, and it contains data from my MySQL database. I've got few fields which are string in my table, where I need them as different types (integer & double) in Elasticsearch.
So I'm aware that I could do it through mapping as follows:
{
"mappings": {
"my_type": {
"properties": {
"userid": {
"type": "text",
"fielddata": true
},
"responsecode": {
"type": "integer"
},
"chargeamount": {
"type": "double"
}
}
}
}
}
But I've tried this when I'm creating the index as a new one. What I wanted to know is how can I update an existing field (ie: chargeamount in this scenario) using mapping as a PUT?
Is this possible? Any help could be appreciated.
Once a mapping type has been created, you're very constrained on what you can update. According to the official documentation, the only changes you can make to an existing mapping after it's been created are the following, but changing a field's type is not one of them:
In general, the mapping for existing fields cannot be updated. There
are some exceptions to this rule. For instance:
new properties can be added to Object datatype fields.
new multi-fields can be added to existing fields.
doc_values can be disabled, but not enabled.
the ignore_above parameter can be updated.
i need help to correct kibana field. when I try to visualizing the fields, shown me the following warning:
Careful! The field contains Analyzed selected strings. Analyzed
strings are highly unique and can use a lot of memory to visualize.
Values: such as bar will be foo-foo and bar broken into. See Core
Mapping Types for more information on setting esta field Analyzed as
not
Elasticsearch default dynamic mapping is to analyze any string field (break the field into tokens, for instance: aaa_bbb_ccc will be break down into aaa,bbb and ccc).
If you do not want such behavior you must change the mapping settings
before any document was pushed into the index.
You have two options to do that:
Change the mapping for a particular index using mapping API, in a static way or dynamic way (dynamic means that the mapping will be applies also to fields that still does not exist in the index)
You can change the behavior of any index according to a pattern, using the template API
This example shows a template that changes the mapping for any index that starts with "app", applying "not analyze" to any field in any type and make sure "timestamp" is a date (good for cases in with the timestamp is represented as a number of seconds from 1970):
{
"template": "myindciesprefix*",
"mappings": {
"_default_": {
"dynamic_templates": [
{
"strings": {
"match_mapping_type": "string",
"mapping": {
"type": "string",
"index": "not_analyzed"
}
}
},
{
"timestamp_field": {
"match": "timestamp",
"mapping": {
"type": "date"
}
}
}
]
}
}
}
Really you dont have any problem is only a message of info, but if you dont want analyzed fields when you build your index in elasticsearch you must indicate that one field is a not analyzed field.
I have a type of document that can have lots of nested type objects, how can I map all of these as nested without actually having to tediously specify a mapping for every single field in the document?
Have you seen Dynamic templates?
Dynamic templates allow you to define custom mappings that can be
applied to dynamically added fields based on:
the datatype detected by Elasticsearch, with match_mapping_type.
the name of the field, with match and unmatch or match_pattern.
the full dotted path to the field, with path_match and path_unmatch.
The original field name {name} and the detected datatype
{dynamic_type} template variables can be used in the mapping
specification as placeholders.
So you could potentially use this example by adding some sort of special pattern to your field, so template would recognize it and map it as a nested object.
PUT my_index
{
"mappings": {
"my_type": {
"dynamic_templates": [
{
"nested_objects": {
"match": "nested_*",
"mapping": {
"type": "nested"
}
}
}
]
}
}
}
P.S. I haven't tested this myself. Let me know if this helps you.
Is it possible to setup alias for field names in elasticsearch? (Just like how index names can be aliased)
For example: i have a document {'firstname': 'John', 'lastname': 'smith'}
I would like to alias 'firstname' to 'fn'...
Just a quick update, Elasticsearch 6.4 came up with feature called Alias Datatype. Check the below mapping and query as sample.
Note that the type of the field is alias in the below mapping for fieldname fn
Sample Mapping:
PUT myindex
{
"mappings": {
"_doc": {
"properties": {
"firstname": {
"type": "text"
},
"fn": {
"type": "alias",
"path": "firstname"
}
}
}
}
}
Sample Query:
GET myindex/_search
{
"query": {
"match" : {
"fn" : "Steve"
}
}
}
The idea is to use the alias for actual field on which inverted index is created. Note that fields with alias datatype aren't meant for write operations and its only meant for querying purpose.
Although you can refer to the link I've mentioned for more details, below are just some of the important points.
Field alias is only meant to be used when your index has a single mapping. Index has to be created post 6.xx version or be created in older version with the setting index.mapping.single_type: true
Can be used in querying, aggregations, sorting, highlighting and suggestion operations
Target field must be actual field on which inverted index is created
Cannot create alias of another alias field
Cannot use alias on multiple fields. Single alias, Single field.
Cannot be used as part of source filtering using _source.
There is no direct field alias functionality. However, you could rename the fields upon indexing using the index_name property in your mappings.
index_name : The name of the field that will be stored in the index.
Defaults to the property/field name.
See here for more information: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-core-types.html
Adding alias fn for existing field firstname
PUT myindex/_mapping
{
"properties": {
"fn": {
"type": "alias",
"path": "firstname"
}
}
}
Should work this way as of Elasticsearch 7.
Probably you can try creating an alias on your index with filter on the desired field. Your filter must be written in such a way that it selects all the entries from your field. Please refer Filtered aliases section in here. But I am interested in knowing your use case. Why you want to create alias on particular field.