Failed to find geo_point field location Springframework Data Elasticsearch - elasticsearch

Failed to find geo_point field location. The location field should have geo_point in mapping but getting lat, lon.
Model Class:
import org.springframework.data.elasticsearch.annotations.GeoPointField;
import org.springframework.data.elasticsearch.core.geo.GeoPoint;
class EsMapping extends TestIndex{
#GeoPointField
private GeoPoint location;
#Field(type = FieldType.Double)
private double latitude;
#Field(type = FieldType.Double)
private double longitude;
.
.
}
TestIndex.java
class TestIndex{
#Field(type = FieldType.Text)
private String name;
.
.
}
Test Controller
#Autowired
private ElasticsearchOperations elasticsearchOperations;
IndexOperations esMappingIndex =
elasticsearchOperations.indexOps(EsMapping.class);
esMappingIndex.delete();
esMappingIndex.create();
esMappingIndex.putMapping(esMappingIndex.createMapping());
esMappingIndex.refresh();
Mapping: (not expected)
http://localhost:9200/testindex/_mapping
"location": {
"properties": {
"lat": {
"type": "float"
},
"lon": {
"type": "float"
}
}
},
"latitude": {
"type": "float"
},
"longitude": {
"type": "float"
},
Error:
{"error":{"root_cause":[{"type":"query_shard_exception","reason":"failed to find geo_point field [location] ","index_uuid":"QLCjshecRNqDMXtkrtbF9g","index":"testindex"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"query","grouped":true,"failed_shards":[{"shard":0,"index":"testindex","node":"jCKzz88BQXeL3wzyX6c8lQ","reason":{"type":"query_shard_exception","reason":"failed to find geo_point field [location]","index_uuid":"QLCjshecRNqDMXtkrtbF9g","index":"testindex"}}]},"status":400}
Expected Mapping
"location": {
"type": "geo_point"
},
"latitude": {
"type": "double"
},
"longitude": {
"type": "double"
},

When I define entity properties like above - with an index name geopoint-test - and do execute the mapping creating code (which is executed as well on application startup), I get the following mapping:
{
"geopoint-test": {
"mappings": {
"properties": {
"location": {
"type": "geo_point"
}
}
}
}
}
There are two differences to the mapping you show:
the location propert is mapped correctly
the latitude and longitude properties are not written to the mapping, as they are not annotated with #Field and so are left for automatic mapping.
The mapping you show is the one that will be automatically created.
Which version of Spring Data Elasticsearch are you using?
Edit 12.12.2020:
Using Spring Data Elasticsearch 4.0.2 and adding the two #Field annotations to the latitudeand longitude properties, both the initial setup of an index and the code to explicitly write the mapping produce this mapping:
{
"properties": {
"location": {
"type": "geo_point"
},
"latitude": {
"type": "double"
},
"longitude": {
"type": "double"
}
}
}

Related

Keyword field created automatically without any mapping in Entity class

My ElasticSearch version is 7.6.2 and my spring-boot-starter-data-elasticsearch is version 2.2.0.
Due to some dependency i am not upgrading ES to lastest version.
Problem i am facing is ES index is sometimes created with .keyword fields and sometimes it is just normal text field.
Below is my entity class. i am not able to find why this is happening. I read that all text field will have keyword field also. but why it is not created always.
My Entity class
#Setter
#Getter
#Document(indexName="myindex", createIndex=true, shards = 4)
public class MyIndex {
#Field(type = FieldType.Keyword)
private String place;
#Field(type = FieldType.Text)
private String name;
#Id
private String dynamicId = UUID.randomUUID().toString();
public MyIndex()
{}
Mapping in ES:
{
"mappings": {
"myindex": {
"properties": {
"place": {
"type": "text",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"name": {
"type": "text",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
},
"dynamicId": {
"type": "text",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
}
}
}
}
}
Sometimes it is created as below for the same entity class
{
"mappings": {
"myindex": {
"properties": {
"place": {
"type": "keyword"
},
"name": {
"type": "text"
},
"dynamicId": {
"type": "text",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
}
}
}
}
}
With the entity definition shown, when Spring Data Elasticsearch creates the index and writes the mapping, you will get the mapping shown in your second example with these value for the properties:
{
"properties": {
"place": {
"type": "keyword"
},
"name": {
"type": "text"
}
}
}
If you want to have a nested keyword property in Spring Data Elasticsearch you have to define it on the entity with the corresponding annotation.
Please notice: the #Id property is not mapped explicitly but will be dynamically mapped on first indexing of a document.
The mapping in the first case and the part in the second where a String is mapped as
"type": "text",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
}
is the default value that Elasticsearch uses when a document is indexed with a text field that was not mapped before - see the docs about dynamic mapping.
So your second example shows the mapping of an index that was created by Spring Data Elasticsearch and where some documents have been indexed.
The first one would be created by Elasticsearch if some other application creates the index and writes data into the index. It could also be that the index was created outside your application, and on application startup no mapping would then be written, because the index already exists. So you should review the way your indices are created.

How to rename a field in Elasticsearch?

I have an index in Elasticsearch with the following field mapping:
{
"version_data": {
"properties": {
"title": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
}
},
"updated_at": {
"type": "date"
},
"updated_by": {
"type": "keyword"
}
}
}
}
I have already created some documents in it and now want to rename version_data field with _version_data.
Is there any way in the Elasticsearch to rename a field within the mapping and in documents?
The closest thing is the alias data type.
In your mapping you could link it from the old to the new name like this:
PUT test/_mapping
{
"properties": {
"_version_data": {
"type": "alias",
"path": "version_data"
}
}
}
BTW I would generally avoid leading underscored since those normally used for internal fields like _id.

Elasticsearch - how to extract date from date time field and add it to each document as a new field

I am learning elasticssearch and in one of the demo databases given to me, i have a date time field saved with the name time_stamp. The date data is saved as text:
"time_stamp":"13-06-2019 04:44:23"
I want to create a new data field titled "date" and extract only the date from each document and store it within the same document. The current index mapping is as follows:
{
"vp1": {
"mappings": {
"dynamic": "false",
"properties": {
"client_id": {
"type": "text"
},
"encod": {
"type": "float"
},
"imagename": {
"type": "text"
},
"indx": {
"type": "text"
},
"machid": {
"type": "text"
},
"matchid": {
"type": "float"
},
"sequence_id": {
"type": "integer"
},
"time_stamp": {
"type": "text"
}
}
}
}
}
I am using python3 to interact with the index.
You first need to update your mapping in order to add the new field since the dynamic setting is set to false the new field cannot be created automatically:
PUT vp1/_mapping
{
"properties": {
"date": {
"type": "date"
}
}
}
Then, an easy way to achieve what you want is to do it like this:
POST vp1/_update_by_query
{
"script": {
"source": "ctx._source.date = /\\s/.split(ctx._source.time_stamp)[0]"
}
}

Use geo_point data type on field in _reindex api

I have index that contains two fields: longitude and latitude that are stored as float. I want to create new index and copy data from the first one but with different mappings. I use reindex api with elastic processors which can rename fields and give them different data types. When i try to create field with type "geo_point" it fails.
"type": "parse_exception",
"reason": "[type] type [geo_point] not supported, cannot convert field.",
but when i create new index i am able to create field with "geo_point" type.
i tried different workarounds but documentation says that with geo queries you can only use "geo_point" type.
is there any solution?
{
"description": "test pipe",
"processors": [
{
"convert": {
"field": "location",
"type": "geo_point"
}
}
]
}
added pipe definition.
OK, let's say that your current index mapping looks like this:
PUT oldindex
{
"mappings": {
"doc": {
"properties": {
"latitude": {
"type": "float"
},
"longitude": {
"type": "float"
}
}
}
}
}
You need to create a new index with the proper mapping, as follows
PUT newindex
{
"mappings": {
"doc": {
"properties": {
"location": {
"type": "geo_point"
}
}
}
}
}
And then, you can simply leverage the reindex API to copy the old index into the new one with some additional scripting to create the location field:
POST _reindex
{
"source": {
"index": "oldindex",
},
"dest": {
"index": "newindex"
},
"script": {
"source": "ctx._source.location = ['lat': ctx._source.latitude, 'lon': ctx._source.longitude]; ctx._source.remove('latitude'); ctx._source.remove('longitude'); "
}
}
And you're good to go with the location field in your new shiny index!

adding null_value through spring-data-elasticsearch annotation

I want to create user index like below using spring-data-elasticsearch-2.1.0. annotation.
I am not able to find any annotation to add "null_value": "NULL". This is required because our sorting order is failing.
"user": {
"properties": {
"firstName": {
"type": "string"
},
"lastName": {
"type": "string"
},
"displayName": {
"type": "string",
"analyzer": "word_analyzer",
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed",
"null_value": "NULL"
}
}
}
}
}
Domain class
private String firstName;
private String lastName;
#MultiField(
mainField = #Field(type = FieldType.String, analyzer = "word_analyzer"),
otherFields = {
#InnerField(suffix = "raw", type = FieldType.String, index = FieldIndex.not_analyzed)
}
)
private String displayName;
How to add "null_value": "NULL" through spring-data-elasticsearch annotation in InnerField? I do not want to creating index mapping externally.
At now it's only possible through #Mapping annotation. Create JSON file with mapping definition:
{
"type": "string",
"index": "analyzed",
"analyzer": "word_analyzer",
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed",
"null_value": "NULL"
}
}
}
And save it in your resources folder. In this example I save it in resources/elastic/document_display_name_mapping.json.
Annotate field with #Mapping annotation
#Mapping(mappingPath = "elastic/document_display_name_mapping.json")
private String displayName;
Referring to https://jira.spring.io/browse/DATAES-312
This is an open issue(Fixed but not merged).
This can be handled by adding "missing" : "_last"/"_first" and "unmapped_type" in sorting options..
Ref:- https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html#_ignoring_unmapped_fields
These options("missing", "unmapped_type", "mode") are not available.
Putting "null_value": "NULL" will not have correct sorting order in string field.
"null_value": "0" can satisfy sorting order for integer field.
is it possible to do something in settings itself to achive one usecase
usecase -- if sort direction is ASC then "missing" : "_first" and if sort direction is DESC then "missing" : "_last" .. which can be applied on raw field.

Resources