Spring boot Elasticsearch: Geo Point not creating correct mapping. Fails on searching - spring-boot

Question

I'm using Elasticsearch for Geo searches.
I have written code to generate random points (lat-lng) within a limit from a centre point. This is inserted into an index, on which the search happens later on.
Versions:
Spring Boot: 2.2.1.RELEASE
Spring Data Elastic Search: (spring-boot-starter-data-elasticsearch) 2.2.1.RELEASE
ElasticSearch: org.elasticsearch : 6.8.4
pom.xml
<parent>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-parent</artifactId>
<version>2.2.1.RELEASE</version>
<relativePath/>
</parent>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-data-elasticsearch</artifactId>
</dependency>
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>elasticsearch</artifactId>
<version>6.8.4</version>
</dependency>
Elastic Configuration:
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;
import lombok.extern.slf4j.Slf4j;
import org.elasticsearch.client.RestHighLevelClient;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.data.elasticsearch.client.ClientConfiguration;
import org.springframework.data.elasticsearch.client.RestClients;
import org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate;
#ConfigurationProperties(prefix = "elastic")
#Configuration
#Setter
#Getter
#NoArgsConstructor
#Slf4j
public class EsBeanConfig {
private String host;
private String port;
#Bean
public RestHighLevelClient client() {
ClientConfiguration clientConfiguration = ClientConfiguration.builder()
.connectedTo(host + ":" + port)
.build();
return RestClients.create(clientConfiguration).rest();
}
#Bean(name = "elasticsearchRestTemplate")
public ElasticsearchRestTemplate elasticsearchRestTemplate() {
return new ElasticsearchRestTemplate(client());
}
}
Elastic entity:
import lombok.Getter;
import lombok.NoArgsConstructor;
import lombok.Setter;
import lombok.ToString;
import org.springframework.data.annotation.Id;
import org.springframework.data.elasticsearch.annotations.Document;
import org.springframework.data.elasticsearch.annotations.Field;
import org.springframework.data.elasticsearch.annotations.FieldType;
import org.springframework.data.elasticsearch.annotations.GeoPointField;
import org.springframework.data.elasticsearch.core.geo.GeoPoint;
#Setter
#Getter
#NoArgsConstructor
#ToString
#Document(indexName = "geo_location_index", type = "geo-class-point-type", replicas = 0, shards = 1)
public class EsGeoPointEntity {
#Id
#Field(type = FieldType.Text)
private String id;
#Field(type = FieldType.Text)
private String storeName;
#GeoPointField
private GeoPoint geoPoint;
}
Index Creation: This creates the index, which is causing issue as it does not create geo_point type.
#PostConstruct
public void init() {
AnnotatedTypeScanner scanner = new AnnotatedTypeScanner(false, Document.class);
for (Class clazz : scanner.findTypes("com.geohash.entity.elastic")) {
Document doc = AnnotationUtils.findAnnotation(clazz, Document.class);
assert doc != null;
if (!elasticsearchRestTemplate.indexExists(doc.indexName())) {
log.info("Index - {} does not exists. Creating index now!!", doc.indexName());
elasticsearchRestTemplate.createIndex(clazz);
// elasticsearchRestTemplate.createIndex(doc.indexName()); --> Not using
}
elasticsearchRestTemplate.refresh(doc.indexName());
}
}
Elasticsearch's index mapping looks like this: [location is not of geo_point type]
{
"mapping": {
"geo-class-point-type": {
"properties": {
"id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"location": {
"properties": {
"lat": {
"type": "float"
},
"lon": {
"type": "float"
}
}
},
"storeName": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
Code to add a random location: I am generating random lat and lng values, and adding to index using GeoPoint.
private EsGeoPointEntity location(String namePrefix, LatLong latLong,
double radius) {
EsGeoPointEntity esGeoPointEntity = new EsGeoPointEntity();
String id = UUIDs.base64UUID();
esGeoPointEntity.setId(id);
esGeoPointEntity.setStoreName(namePrefix + id);
esGeoPointEntity.setLocation(
new GeoPoint(latLong.getLatitude().doubleValue(), latLong.getLongitude().doubleValue()));
return esGeoPointEntity;
}
Code to Insert data:
#Override
public void bulkIndex(List<?> objects) {
try {
List<IndexQuery> queries = new ArrayList<>();
Class<?> clazz = objects.get(0).getClass();
Document doc = AnnotationUtils.findAnnotation(clazz, Document.class);
String idMethodName = ApplicationUtils.getIdMethodName(clazz);
assert doc != null;
log.debug("index Name {} and index type {}", doc.indexName(), doc.type());
for (Object obj : objects) {
IndexQuery indexQuery = new IndexQuery();
// Getting the ID
indexQuery.setId(String.valueOf(ApplicationUtils.getProperty(obj, idMethodName)));
indexQuery.setObject(obj);
indexQuery.setIndexName(doc.indexName());
indexQuery.setType(doc.type());
queries.add(indexQuery);
}
if (!queries.isEmpty()) {
elasticsearchRestTemplate.bulkIndex(queries);
elasticsearchRestTemplate.refresh(doc.indexName());
}
} catch (Exception e) {
log.error("Error while bulk Indexing.", e);
throw e;
}
}
To find the number of location within a specified radius and a Geo Point, I'm using the below code:
#Override
public List<EsGeoPointEntity> findLocationWithinDistance(double latitude, double longitude,
double distance) {
try {
CriteriaQuery geoLocationCriteriaQuery = new CriteriaQuery(
new Criteria("geoPoint").within(
new GeoPoint(latitude, longitude), distance + "km"));
return elasticsearchRestTemplate
.queryForList(geoLocationCriteriaQuery, EsGeoPointEntity.class);
} catch (Exception e) {
log.error("Error while fetching nearby locations.", e);
throw e;
}
}
Error while fetching nearby locations:
2019-12-04 20:45:05.911 ERROR 51726 [http-nio-7676-exec-3] --- j.c.r.g.s.e.impl.EsIndexerServiceImpl Line: (92): Error while fetching nearby locations.
org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=search_phase_execution_exception, reason=all shards failed]
at org.elasticsearch.rest.BytesRestResponse.errorFromXContent(BytesRestResponse.java:177)
at org.elasticsearch.client.RestHighLevelClient.parseEntity(RestHighLevelClient.java:2053)
at org.elasticsearch.client.RestHighLevelClient.parseResponseException(RestHighLevelClient.java:2030)
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1777)
at org.elasticsearch.client.RestHighLevelClient.performRequest(RestHighLevelClient.java:1734)
at org.elasticsearch.client.RestHighLevelClient.performRequestAndParseEntity(RestHighLevelClient.java:1696)
at org.elasticsearch.client.RestHighLevelClient.search(RestHighLevelClient.java:1092)
at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.queryForPage(ElasticsearchRestTemplate.java:511)
at org.springframework.data.elasticsearch.core.ElasticsearchRestTemplate.queryForList(ElasticsearchRestTemplate.java:457)
at com.geohash.service.elastic.impl.EsIndexerServiceImpl.findLocationWithinDistance(EsIndexerServiceImpl.java:90)
at com.service.elastic.impl.EsLocationGenerationServiceImpl.findLocationWithinDistance(EsLocationGenerationServiceImpl.java:51)
at com.geohash.controller.EsLocationController.findLocationWithinDistance(EsLocationController.java:64)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:567)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:190)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:106)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:888)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:793)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1040)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:943)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:634)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:741)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:209)
at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:178)
at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:358)
at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:271)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:202)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:526)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:139)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:92)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:74)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:408)
at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:861)
at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1579)
at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.base/java.lang.Thread.run(Thread.java:835)
Suppressed: org.elasticsearch.client.ResponseException: method [POST], host [http://localhost:9200], URI [/geo_location_index/geo-class-point-type/_search?rest_total_hits_as_int=true&typed_keys=true&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=dfs_query_then_fetch&batched_reduce_size=512], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"query_shard_exception","reason":"failed to find geo_point field [geoPoint]","index_uuid":"xlQjtUmqTMiGsyGcZj9HGw","index":"geo_location_index"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"dfs","grouped":true,"failed_shards":[{"shard":0,"index":"geo_location_index","node":"hWqt86GoQJqJGovxnpqQRg","reason":{"type":"query_shard_exception","reason":"failed to find geo_point field [geoPoint]","index_uuid":"xlQjtUmqTMiGsyGcZj9HGw","index":"geo_location_index"}}]},"status":400}
at org.elasticsearch.client.RestClient$SyncResponseListener.get(RestClient.java:936)
at org.elasticsearch.client.RestClient.performRequest(RestClient.java:233)
at org.elasticsearch.client.RestHighLevelClient.internalPerformRequest(RestHighLevelClient.java:1764)
... 64 common frames omitted
Caused by: org.elasticsearch.client.ResponseException: method [POST], host [http://localhost:9200], URI [/geo_location_index/geo-class-point-type/_search?rest_total_hits_as_int=true&typed_keys=true&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&search_type=dfs_query_then_fetch&batched_reduce_size=512], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"query_shard_exception","reason":"failed to find geo_point field [geoPoint]","index_uuid":"xlQjtUmqTMiGsyGcZj9HGw","index":"geo_location_index"}],"type":"search_phase_execution_exception","reason":"all shards failed","phase":"dfs","grouped":true,"failed_shards":[{"shard":0,"index":"geo_location_index","node":"hWqt86GoQJqJGovxnpqQRg","reason":{"type":"query_shard_exception","reason":"failed to find geo_point field [geoPoint]","index_uuid":"xlQjtUmqTMiGsyGcZj9HGw","index":"geo_location_index"}}]},"status":400}
at org.elasticsearch.client.RestClient$1.completed(RestClient.java:552)
at org.elasticsearch.client.RestClient$1.completed(RestClient.java:537)
at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:122)
at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:181)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:448)
at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:338)
at org.apache.http.impl.nio.client.InternalRequestExecutor.inputReady(InternalRequestExecutor.java:83)
at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:591)
... 1 common frames omitted
PS: I have also tried this with, but same issue:
Elasticsearch: 6.4.3
Transport client: 6.4.3
Spring Data Elasticsearch: 3.1.10.RELEASE

Answer 1

Ah! I did miss a crucial information and by adding one small line of code, it worked.
Here goes: For Geo types, dynamic mapping will not work. So, enforce a mapping through code before any data gets inserted or create a mapping manually in ES.
I added the code in my #PostConstruct:
#PostConstruct
public void init() {
AnnotatedTypeScanner scanner = new AnnotatedTypeScanner(false, Document.class);
for (Class clazz : scanner.findTypes("com.geohash.entity.elastic")) {
Document doc = AnnotationUtils.findAnnotation(clazz, Document.class);
assert doc != null;
if (!elasticsearchRestTemplate.indexExists(doc.indexName())) {
log.info("Index - {} does not exists. Creating index now!!", doc.indexName());
elasticsearchRestTemplate.createIndex(clazz);
}
elasticsearchRestTemplate.refresh(doc.indexName());
elasticsearchRestTemplate.putMapping(clazz); --> Saviour
}
}
The mapping, thus created, is correct and has geo_point.
NOTE: Did not find this in the documentation.