Elasticsearch. Can not find custom analyzer - elasticsearch

I have model like this:
#Getter
#Setter
#Document(indexName = "indexName", type = "typeName")
#Setting(settingPath = "/elastic/elastic-setting.json")
public class Model extends BaseModel {
#Field(type = FieldType.String, index = FieldIndex.analyzed, analyzer = "customAnalyzer")
private String name;
}
And i have elastic-setting.json inside ../resources/elastic/elastic-setting.json:
{
"index": {
"number_of_shards": "1",
"number_of_replicas": "0",
"analysis": {
"analyzer": {
"customAnalyzer": {
"type": "custom",
"tokenizer": "uax_url_email"
}
}
}
}
}
I clean my elastic DB and when i start my application i have exception:
MapperParsingException[analyzer [customAnalyzer] not found for field [name]]
What's wrong with my code?
Help me, please!
EDIT
Val, I thought #Setting is like an addition for #Document, but looks like they are interchangeably.
In my case i also have another model, with:
#Document(indexName = "indexName", type = "anotherTypeName")
So, first i create index with name "indexName" for anotherModel, next when elastic preparing Model, it see, that index with name "indexName" already created, and he does not use #Setting.
Now i have another quesion.
How to add custom analyzer to already created index in java code, for example in InitializingBean. Something like - is my analyzer created? no - create. yes - do not create.

Modify your elastic-setting.json file like this:
{
"index": {
"number_of_shards": "1",
"number_of_replicas": "0"
},
"analysis": {
"analyzer": {
"customAnalyzer": {
"type": "custom",
"tokenizer": "uax_url_email"
}
}
}
}
}
Note that you need to delete your index first and recreate it.
UPDATE
You can certainly add a custom analyzer via Java code, however, you won't be able to change your existing mapping in order to use that analyzer, so you're really better off wiping your index and recreating it from scratch with a proper elastic-setting.json JSON file.

For Val:
Yeah, i use something like this.
Previously, i had added #Setting in one of my entity class, but when i started app, index with same name was already created, before Spring Data had analysed entity with #Setting, and index was not modified, because index with same name was already created.
Now I add annotation #Setting(path = "elastic-setting.json") on abstract baseModel, and high level hierarchy class was scanned firstly and analyzer was created as well.

Related

How to update data type of a field in elasticsearch

I am publishing a data to elasticsearch using fluentd. It has a field Data.CPU which is currently set to string. Index name is health_gateway
I have made some changes in python code which is generating the data so now this field Data.CPU has now become integer. But still elasticsearch is showing it as string. How can I update it data type.
I tried running below commands in kibana dev tools:
PUT health_gateway/doc/_mapping
{
"doc" : {
"properties" : {
"Data.CPU" : {"type" : "integer"}
}
}
}
But it gave me below error:
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true."
}
],
"type" : "illegal_argument_exception",
"reason" : "Types cannot be provided in put mapping requests, unless the include_type_name parameter is set to true."
},
"status" : 400
}
There is also this document which says using mutate we can convert the data type but I am not able to understand it properly.
I do not want to delete the index and recreate as I have created a visualization based on this index and after deleting it will also be deleted. Can anyone please help in this.
The short answer is that you can't change the mapping of a field that already exists in a given index, as explained in the official docs.
The specific error you got is because you included /doc/ in your request path (you probably wanted /<index>/_mapping), but fixing this alone won't be sufficient.
Finally, I'm not sure you really have a dot in the field name there. Last I heard it wasn't possible to use dots in field names.
Nevertheless, there are several ways forward in your situation... here are a couple of them:
Use a scripted field
You can add a scripted field to the Kibana index-pattern. It's quick to implement, but has major performance implications. You can read more about them on the Elastic blog here (especially under the heading "Match a number and return that match").
Add a new multi-field
You could add a new multifield. The example below assumes that CPU is a nested field under Data, rather than really being called Data.CPU with a literal .:
PUT health_gateway/_mapping
{
"doc": {
"properties": {
"Data": {
"properties": {
"CPU": {
"type": "keyword",
"fields": {
"int": {
"type": "short"
}
}
}
}
}
}
}
}
Reindex your data within ES
Use the Reindex API. Be sure to set the correct mapping on the target index.
Delete and reindex everything from source
If you are able to regenerate the data from source in a timely manner, without disrupting users, you can simply delete the index and reingest all your data with an updated mapping.
You can update the mapping, by indexing the same field in multiple ways i.e by using multi fields.
Using the below mapping, Data.CPU.raw will be of integer type
{
"mappings": {
"properties": {
"Data": {
"properties": {
"CPU": {
"type": "string",
"fields": {
"raw": {
"type": "integer"
}
}
}
}
}
}
}
}
OR you can create a new index with correct index mapping, and reindex the data in it using the reindex API

Spring Data Elasticsearch setting annotation did not take effect

I'm trying with spring data elastic search and have a class defined like this:
#Data
#NoArgsConstructor
#AllArgsConstructor
#Document(indexName = "master", type = "master", shards = 1, replicas = 0)
#Setting(settingPath = "/settings/setting.json")
public class Master {
#Id
private String id;
#MultiField(mainField = #Field(type = FieldType.String, store = true),
otherFields = {
#InnerField(suffix = "autocomplete", type = FieldType.String, indexAnalyzer = "autocomplete", searchAnalyzer = "standard")
}
)
private String firstName;
private String lastName;
}
The setting file is under /src/main/settings/setting.json, which looks like this
{
"index": {
"analysis": {
"filter": {
"autocomplete_filter": {
"type": "edge_ngram",
"min_gram": 1,
"max_gram": 20
}
},
"analyzer": {
"autocomplete": {
"type": "custom",
"tokenizer": "standard",
"filter": [
"lowercase",
"autocomplete_filter"
]
}
}
}
}
}
I ran my test class by first deleting the index, and recreate the index like this
elasticsearchTemplate.deleteIndex(Master.class);
elasticsearchTemplate.createIndex(Master.class);
elasticsearchTemplate.putMapping(Master.class);
elasticsearchTemplate.refresh(Master.class);
But when I try to save something into the index there is this error message for MapperParsingException:
2017-10-04 18:56:31.806 ERROR 2942 --- [ main] .d.e.r.s.AbstractElasticsearchRepository : failed to load elasticsearch nodes : org.elasticsearch.index.mapper.MapperParsingException: analyzer [autocomplete] not found for field [autocomplete]
Spent 4 hours trying to figure this out, looked at the Debug mode log, nothing.
I tried to break the JSON format by deleting a comma, it broke so the JSON was being interpreted.
I used the RestAPI to query the master index but the settings doesn't seem to contain the autocomplete analyzer or any analyzer.
Weird thing is that my document can be saved and queried even with this error. But I do want this analyzer.
BTW, this is a parent class in a parent-child relationship, if that's relevant.
Finally got it figured out!
I have to put the same setting across all domains using the same index (both parent and child), then delete the index, restart the server, and it worked!

Is it possible to update an existing field in an index through mapping in Elasticsearch?

I've already created an index, and it contains data from my MySQL database. I've got few fields which are string in my table, where I need them as different types (integer & double) in Elasticsearch.
So I'm aware that I could do it through mapping as follows:
{
"mappings": {
"my_type": {
"properties": {
"userid": {
"type": "text",
"fielddata": true
},
"responsecode": {
"type": "integer"
},
"chargeamount": {
"type": "double"
}
}
}
}
}
But I've tried this when I'm creating the index as a new one. What I wanted to know is how can I update an existing field (ie: chargeamount in this scenario) using mapping as a PUT?
Is this possible? Any help could be appreciated.
Once a mapping type has been created, you're very constrained on what you can update. According to the official documentation, the only changes you can make to an existing mapping after it's been created are the following, but changing a field's type is not one of them:
In general, the mapping for existing fields cannot be updated. There
are some exceptions to this rule. For instance:
new properties can be added to Object datatype fields.
new multi-fields can be added to existing fields.
doc_values can be disabled, but not enabled.
the ignore_above parameter can be updated.

Updating ElasticSearch mapping

I have an index with more than 30.000.000 docs.
I need to add a new field for new upcoming documents.
But I need to specify the correct mapping of this new field.
Will this command leads to any reindexing operation for already existing documents?
PUT /my_index/_mappings/my_type
{
"my_type": {
"properties": {
"A": {
"properties": {
"new_field": {
"type": "string",
"analyzer": "analyzer_keyword"
}
}
}
}
}
}
No, above query will not lead to any kind of re-indexing.This will just add a new field in your mapping.New field will get added if you update existing document or create new document.
You can refer here for more details.

How to define dynamically / override the _default_ mapping in elasticsearch?

Use case :
The index/indices will be built dynamically from a template, so that
it will pick some custom settings (number of shards/replicas/ etc).
Generate dynamically the types and their mappings, enabling for them all the timestamp
& ttl fields. Also define the same parent type for all types, except
the parent type itself (I know its name).
{
"template": "*",
"settings": {
...................
},
"mappings": {
"_default_": {
"_parent": {
"type": "ParentType"
},
"_timestamp": {
"enabled": true,
"store": true
},
"_ttl": {
"enabled": true
}
}
}
}
How could I disable the _parent field for the ParentType type itself ?
After a little big of digging on the docs, I found out that you can override __default__ mappings using this work around.
{
"mappings": {
"_default_": {
"_parent": {
"type": "my_parent_type" # 1
}
},
"my_parent_type": {
"_parent": {
"type": "_dummy" # 2
}
}
}
}
1) This mapping configuration will be inherited by all your types including my_parent_typetype. Step 2 explains how to override your parent type _parent meta-field.
2) I found here https://www.elastic.co/guide/en/elasticsearch/reference/current/default-mapping.html that you can override any of the mappings from __default__ when you provide a specific mapping for the type in question.
In this case we are assigning the _parent mapping of my_parent_type to a _dummy parent type that does not have to exists in your index.
Note: ElasticSearch 2.x documentation states:
The _parent.type setting can only point to a type that doesn’t exist yet. This means that a type cannot become a parent type after it is has been created.
For this reason, you have to make sure you index first, a document that is not my_parent_type type, otherwise you'll get an at indexing time.
I hope this helps.

Resources