I couldn't find any documentation regarding the following issue:
We are creating a template file for all the fields we are indexing to the Elasticsearch. The question is regarding fields which are not defined in the template:
What is the default Elastic value they are index with?
What are the limitations (if there are any) for indexing those fields?
I was trying to index a field which it's value is a list of JSON and
I got the exception: "Can't get text on a START_OBJECT at 1:311",
what does it mean?
string fields are indexed with a text field with a standard analyzer, and a subfield .keyword with a keyword datatype, with option ignore_above setted to 256. date field are tried to parse into iso 8601 format - this one yyyy-MM-dd HH:mm:ss . long is the default for numerical and double for for decimals. you could modify this default behaviour with dynamic templates. For example, if we wanted to map all integer fields as short instead of long, and all string fields as keyword, we could use the following template:
PUT my_index
{
"mappings": {
"dynamic_templates": [
{
"integers": {
"match_mapping_type": "long",
"mapping": {
"type": "short"
}
}
},
{
"strings": {
"match_mapping_type": "string",
"mapping": {
"type": "keyword",
"ignore_above" :256
}
}
}
]
}
}
There are no limitation to index fields not defined in templates
this error means that there is an error in json with es syntax, could you share this json?
Related
I have data and and i want to create index and want it to make searchable and aggregatable both.If I use datatype keyword I can't search any string but can aggregate but if I use datatype text then I can't aggregate but can search any string.
So please tell me how to resolve this problem. I am using elasticsearch 6
The solution is to create a text field with a keyword sub-field so that you can do both, search text and aggregate values:
Your field mapping should look like this:
{
"my_field": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword"
}
}
}
}
Using the above mapping, you can search on the my_field field and aggregate and my_field.keyword.
I am trying to copy two fields onto a third field, which should have the type 'keyword' (because I want to be able to aggregate by it, and do not need to perform a full-text search)
PUT /test/_mapping/_doc
{
"properties": {
"first": {
"copy_to": "full_name",
"type": "keyword"
},
"last": {
"copy_to": "full_name",
"type": "keyword"
},
"full_name": {
"type": "keyword"
}
}
}
I then post a new document:
POST /test/_doc
{
"first": "Bar",
"last": "Foo"
}
And query it using the composite field full_name:
GET /test2/_search
{
"query": {
"match": {
"full_name": "Bar Foo"
}
}
}
And no hits are returned.
If the type of the composite field full_name were text then it works as expected and described in the docs:
https://www.elastic.co/guide/en/elasticsearch/reference/current/copy-to.html
Is it not possible to copy onto a keyword-type field?
The problem is that you use match query - When you index your docs you use keyword type which according to the ES documentation are "...only searchable by their exact value."
However when you query that field you use match query which is using the standard analyzer which, among other stuff, also does lower-casing which causes your terms to not match nothing.
You have few options I can think of in this case:
Change the field type to text which will perform the same analysis as the match query.
Create a custom field type with custom analyzer which will perform lower casing
Don't query more than a single term at a time and use term query instead of match
It seems that the type of destination field of copy_to must be text type.
I have json data that has a "product_ref" field that can take these values as an example:
"product_ref": "N/A"
"product_ref": "90323"
"product_ref": "SN3005"
"product_ref": "2015-05-23"
When pushing the data to the index i get a mapping error:
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"mapper [root.product_ref] of different type, current_type [date], merged_type [text]"}],"type":"illegal_argument_exception","reason":"mapper [root.product_ref] of different type, current_type [date], merged_type [text]"},"status":400}
Any idea?
There is something called date detection, and by default, it is enabled.
If date_detection is enabled (default), then new string fields are checked to see whether their contents match any of the date patterns specified in dynamic_date_formats. If a match is found, a new date field is added with the corresponding format.
You just need to disable it by modifying your mappings:
PUT /products
{
"mappings": {
"doc": {
"date_detection": false,
"properties": {
"product_ref": { "type": "keyword" },
}
}
}
}
This is happening because ElasticSearch assumed you're indexing dates of a particular format, and a value which doesn't match that was attempted to be indexed. i.e. after indexing date, you index wrong format.
Make sure all the values are dates and none are empty,perhaps remove these in your ingestion layer.
EDIT: If you don't care to lose the date value you can use the dynamic mapping.
{
"dynamic_templates": [
{
"integers": {
"match_mapping_type": "date",
"mapping": {
"type": "text"
}
}
}
]
}
Using elastic 2.3.5. Is there a way to make a field filterable, but not searchable? For example, I have a language field, with values like en-US. Setting several filters in query->bool->filter->term, I'm able to filter the result set without affecting the score, for example, searching for only documents that have en-US in the language field.
However, I want a query searching for the term en-US to return no results, since this is not really an indexed field for searching, but just so I can filter.
Can I do this?
ElasticSearch use an _all field to allow fast full-text search on entire documents. This is why searching for en-US in all fields of all documents return you the one containing 'language':'en-US'.
https://www.elastic.co/guide/en/elasticsearch/reference/current/mapping-all-field.html
You can specify "include_in_all": false in the mapping to deactivate include of a field into _all.
PUT my_index
{
"mappings": {
"my_type": {
"properties": {
"title": {
"type": "string"
},
"country": {
"type": "string"
},
"language": {
"type": "string",
"include_in_all": false
}
}
}
}
}
In this example, searching for 'US' in all field will return only document containing US in title or country. But you still be able to filter your query using the language field.
https://www.elastic.co/guide/en/elasticsearch/reference/current/include-in-all.html
Ok, in my elastisearch I am using the following mapping for an index:
{
"mappings": {
"mytype": {
"type":"object",
"dynamic" : "false",
"properties": {
"name": {
"type": "string"
},
"address": {
"type": "string"
},
"published": {
"type": "date"
}
}
}
}
}
it works. In fact if I put a malformed date in the field "published" it complains and fails.
Also I've the following configuration:
...
node.name : node1
index.mapper.dynamic : false
index.mapper.dynamic.strict : true
...
And without the mapping, I can't really use the type. The problem is that if I insert something like:
{
"name":"boh58585",
"address": "hiohio",
"published": "2014-4-4",
"test": "hophiophop"
}
it will happily accept it. Which is not the behaviour that I expect, because the field test is not in the mapping. How can I restrict the fields of the document to only those that are in the mapping???
The use of "dynamic": false tells Elasticsearch to never allow the mapping of an index to be changed. If you want an error thrown when you try to index new documents with fields outside of the defined mapping, use "dynamic": "strict" instead.
From the docs:
"The dynamic parameter can also be set to strict, meaning that not only new fields will not be introduced into the mapping, parsing (indexing) docs with such new fields will fail."
Since you've defined this in the settings, I would guess that leaving out the dynamic from the mapping definition completely will default to "dynamic": "strict".
Is your problem with the malformed date field?
I would fix the date issue and continue to use dynamic: false.
You can read about the ways to set up the date field mapping for a custom format here.
Stick the date format string in a {type: date, format: ?} mapping.