we have an elastic search index with following mapping (showing only partial mapping relevant to this question)
"instFields": {
"properties": {
"_index": {
"type": "object"
},
"fieldValue": {
"fields": {
"raw": {
"index": "not_analyzed",
"type": "string"
}
},
"type": "string"
},
"sourceFieldId": {
"type": "integer"
}
},
"type": "nested"
}
as you can see fieldValue type is string: in original data in the database for that fieldValue column is stored in a JSON type column (in Postgresql). use case is such that when this data is stored fieldValue can be valid JsNumber, JsString,JsBoolean (any valid [JsValue][1] now question is that when storing this fieldValue in ES - it'll have to be a definite type - so we convert fieldValue to String while pushing data into ElasticSearch.
Following is a sample data from Elastic search
"instFields": [
{
"sourceFieldId": 1233,
"fieldValue": "Demo Logistics LLC"
},
{
"sourceFieldId": 1236,
"fieldValue": "169451"
}
]
this is where it gets interesting where now we want to run various metrics aggregations on fieldValue - for e.g. if sourceFieldId = 1236 then run [avg][3] on fieldValue - problem is fieldValue had to be stored as string in ES - due to originally fieldValue being JsValue type field in the application. what's the best way to create mapping in elastic search such that fieldValue can be stored with an appropriate type vs string type so various metrics aggregation can be run of fieldValue which are of type long (though encoded as string in ES)
One of the ways to achieve this is create different fields in elastic search with all possible type of JsValue (e.g. JsNumber, JsBoolean,JsString etc). now while indexing - application can derive proper type of JsValue field to find out whether it's JsString, JsNumber, JsBoolean etc.
on application side I can decode proper type of fieldValue being indexed
value match{
case JsString(s) =>
case JsNumber(n) =>
case JsBoolean(b)
}
now modify mapping in elastic search and add more fields - each with proper type - as shown below
"instFields": {
"properties": {
"_index": {
"type": "object"
},
"fieldBoolean": {
"type": "boolean"
},
"fieldDate": {
"fields": {
"raw": {
"format": "dateOptionalTime",
"type": "date"
}
},
"format": "dateOptionalTime",
"type": "date"
},
"fieldDouble": {
"fields": {
"raw": {
"type": "double"
}
},
"type": "double"
},
"fieldLong": {
"fields": {
"raw": {
"type": "long"
}
},
"type": "long"
},
"fieldString": {
"fields": {
"raw": {
"index": "not_analyzed",
"type": "string"
}
},
"type": "string"
},
"fieldValue": {
"fields": {
"raw": {
"index": "not_analyzed",
"type": "string"
}
},
"type": "string"
}
now at the time of indexing
value match{
case JsString(s) => //populate fieldString
case JsNumber(n) => //populate fieldDouble (there is also fieldLong)
case JsBoolean(b) //populate fieldBoolean
}
this way now boolean value is stored in fieldBoolean, number is stored in long etc. now running metrics aggregation becomes a normal business by going against fieldLong or fieldDouble field (depending on the query use case). notice fieldValue field is still there in ES mapping and index as before. Application will continue to convert value to string and store it in fieldValue as before - this way queries which don't care about types can only query fieldValue field in the index.
It sounds like you should have two separate fields, one for the case when the value is a string and one for when it is an instance of a number.
Depending on how you're indexing this data, it can be easy or hard. However, its a bit strange that you have a fields that could be a string or a number.
Regardless, elasticsearch is not going to be able to do both in a single field
Related
I wonder that if the type of destination field of copy_to must be text type.
I have not find the description in the official document.
https://www.elastic.co/guide/en/elasticsearch/reference/current/copy-to.html
In other words,can the full_name's type be other type but not text? Thanks.
PUT my-index-000001
{
"mappings": {
"properties": {
"first_name": {
"type": "text",
"copy_to": "full_name"
},
"last_name": {
"type": "text",
"copy_to": "full_name"
},
"full_name": {
"type": "text"
}
}
}
}
You can try it yourself, I tried it creating with keyword and integer types and it worked perfectly fine.
However if you define it to be integer and then try to index text like the one given in example it will throw mapper_parsing_exception.
I have a nested field my index, value, numberValue, and type are some of its properties
"metadata": {
"type": "nested",
"properties": {
"numbervalue": {
"type": "double"
},
"type": {
"type": "keyword"
},
"value": {
"type": "keyword"
}...
}
I want to sort on value if the type is STRING and sort on numberValue if the type is NUMBER. I read some examples about script sort we can choose a field with script but in all of them, type was fixed. is it possible to have conditional type in script sort?
I am implementing Completion Suggester in my application, and here goes my requirement:
I am using Elasticsearch 5.5.3 (which support multiple types). I have around 10 types in my Elasticsearch and each type has around 10 string fields. What I want to do is make a single search box, that would complete the phrase (of any fields of those 10 types) as user starts typing using completion suggester. What could be the best approach to it? Is using _all field a good idea?
Yes, that's perfectly doable using a "custom all field" field of type completion
First, create the index with all the types and make sure to copy each field in a custom field of type completion:
PUT my_index
{
"mappings": {
"type1": {
"properties": {
"field1": {
"type": "text",
"copy_to": "my_all"
},
"field2": {
"type": "text",
"copy_to": "my_all"
},
"my_all": {
"type": "completion"
}
}
},
"type1": {
"properties": {
"field1": {
"type": "text",
"copy_to": "my_all"
},
"field2": {
"type": "text",
"copy_to": "my_all"
},
"my_all": {
"type": "completion"
}
}
}
}
}
Then, you'd query the completion data like this (i.e. without specifying any mapping type and using the common my_all field):
POST my_index/_search
{
"suggest": {
"my-suggest": {
"prefix": "bla",
"completion": {
"field": "my_all"
}
}
}
}
I want to have an elasticsearch schema that has some pre defined fields including an object type field. I want to have all the fields inside that object type field to be string by default.
I have the following mappings and dynamic templates while creating the index.
PUT myindex
{
"mappings": {
"doc": {
"dynamic_templates": [
{
"default_string": {
"path_match": "myObj.*",
"match_mapping_type": "*",
"mapping": {
"type": "text"
}
}
}],
"properties": {
"dummy_field_name": { "type": "text" },
"timestamp": {
"type": "date",
"format": "epoch_second"
},
"myObj": {
"type": "object"
}
}
}
}
}
But when I submit a field inside the object with a numeric value, it is not mapping that field to string.
curl -XPOST "http://elastic-url:8080/myindex/test" -d
'{"dummy_field_name": "dymmy_value", "myObj":{ "filed_1": 123 ,
"field_2": "some value"}, "timestamp": 1522196333}'
"filed_1" is identified as a number field. But I want it to be stored as a string type.
Field types detected
Your mapping is defined for type "doc" but you are indexing to type "test", try matching them
I'm using olivere's elastic v.5 Go library (https://godoc.org/gopkg.in/olivere/elastic.v5) for my Elastic queries. If I have an elastic mapping like this:
"mappings": {
"boxes": {
"properties": {
"field1": {
"type": "string"
},
"field2": {
"type": "string"
},
"field3": {
"type": "date"
},
"field4": {
"type": "date"
}
}
}
And I want to get a list of all fields that have type 'date'.
I've looked into GetFieldMapping but that doesn't seem to have an option to filter the fields based on type.
I tried this:
elasticclient.GetFieldMapping().Index("someindex").Type("boxes").Field().Type("date").Pretty(true).Do(ctx)
That just gives me all the fields and their types. Is there a different syntax to do this? Thanks!