How to create mappings for parent-child index and search for children filtered by parent - spring

Question

Goal:
I want to create a parent-child index with 2 entities. A profile and a comment. A profile (for simplicity) has a custom id (UUID converted to a string), age, and location (GeoPoint). A comment (for simplicity) has a custom id (UUID converted to a string). With this information, I want to be able to search for all comments given some filtering data against a profile. For example, I want to find all comments by profiles between the ages of 26 and 36 and is located within 100km of lat: 3.0, long 5.0.
Classes:
// Profile.kt
import org.elasticsearch.common.geo.GeoPoint
import org.springframework.data.annotation.Id
import org.springframework.data.elasticsearch.annotations.Document
import org.springframework.data.elasticsearch.annotations.Field
import org.springframework.data.elasticsearch.annotations.FieldType
import org.springframework.data.elasticsearch.annotations.GeoPointField
#Document(indexName = "message_board", createIndex = false, type = "profile")
data class Profile(
#Id
val profileId: String,
#Field(type = FieldType.Short, store = true)
val age: Short,
#GeoPointField
val location: GeoPoint
)
// Comment.kt
import org.springframework.data.annotation.Id
import org.springframework.data.elasticsearch.annotations.Document
import org.springframework.data.elasticsearch.annotations.Field
import org.springframework.data.elasticsearch.annotations.FieldType
import org.springframework.data.elasticsearch.annotations.Parent
#Document(indexName = "message_board", createIndex = false, type = "comment")
data class Comment(
#Id
val commentId: String,
#Field(type = FieldType.Text, store = true)
#Parent(type = "profile")
val parentId: String
)
// RestClientConfig.kt
import org.elasticsearch.client.RestHighLevelClient
import org.springframework.context.annotation.Configuration
import org.springframework.data.elasticsearch.client.ClientConfiguration
import org.springframework.data.elasticsearch.client.RestClients
import org.springframework.data.elasticsearch.config.AbstractElasticsearchConfiguration
#Configuration
class RestClientConfig(
private val elasticSearchConfig: ElasticSearchConfig
) : AbstractElasticsearchConfiguration() {
override fun elasticsearchClient(): RestHighLevelClient {
val clientConfiguration: ClientConfiguration = ClientConfiguration.builder()
.connectedTo("${elasticSearchConfig.endpoint}:${elasticSearchConfig.port}")
.build()
return RestClients.create(clientConfiguration).rest()
}
}
// Controller.kt
import org.springframework.web.bind.annotation.RestController
import org.springframework.data.elasticsearch.core.ElasticsearchOperations
import org.springframework.data.elasticsearch.core.mapping.IndexCoordinates
import org.springframework.http.MediaType
import org.springframework.web.bind.annotation.GetMapping
import org.springframework.web.bind.annotation.RequestMapping
#RestController
#RequestMapping("/", produces = [MediaType.APPLICATION_JSON_VALUE])
class Controller constructor(
private val elasticsearchOperations: ElasticsearchOperations
) {
init {
elasticsearchOperations.indexOps(IndexCoordinates.of("message_board")).let { indexOp ->
if (!indexOp.exists() && indexOp.create()) {
val profileMapping = indexOp.createMapping(Profile::class.java)
println("Profile Mapping: $profileMapping")
indexOp.putMapping(profileMapping)
val commentMapping = indexOp.createMapping(Comment::class.java)
println("Comment Mapping: $commentMapping")
indexOp.putMapping(commentMapping)
indexOp.refresh()
}
}
}
#GetMapping("comments")
fun getComments(): List<Comment> {
val searchQuery = NativeSearchQueryBuilder()
.withFilter(
HasParentQueryBuilder(
"profile",
QueryBuilders
.boolQuery()
.must(
QueryBuilders
.geoDistanceQuery("location")
.distance(100, DistanceUnit.KILOMETERS)
.point(3.0, 5.0)
)
.must(
QueryBuilders
.rangeQuery("age")
.gte(26)
.lte(36)
),
false
)
)
.build()
return elasticsearchOperations.search(searchQuery, Comment::class.java, IndexCoordinates.of("message_board")).toList().map(SearchHit<Comment>::getContent)
}
}
My Setup:
I have elasticsearch running in docker via:
docker run --name es -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" -d -v es_data:/usr/share/elasticsearch/data docker.elastic.co/elasticsearch/elasticsearch:7.4.2
Spring Boot: "2.3.2.RELEASE"
Spring Data Elasticsearch: "4.0.2.RELEASE"
Issues:
I'm failing to get past the init block of my controller with the following exception:
Profile Mapping: MapDocument#?#? {"properties":{"age":{"store":true,"type":"short"},"location":{"type":"geo_point"}}}
Comment Mapping: MapDocument#?#? {"_parent":{"type":"profile"},"properties":{"parentId":{"store":true,"type":"text"}}}
Suppressed: org.elasticsearch.client.ResponseException: method [PUT], host [http://localhost:9200], URI [/message_board/_mapping?master_timeout=30s&timeout=30s], status line [HTTP/1.1 400 Bad Request]
Caused by: org.elasticsearch.ElasticsearchStatusException: Elasticsearch exception [type=mapper_parsing_exception, reason=Root mapping definition has unsupported parameters: [_parent : {type=profile}]]
I need a solution that doesn't involve making a direct POST request to ES. Ideally, this is solved with the Elasticsearch client API. It feels like there's something missing with my annotations on the data classes however I couldn't find any documentation about this.

Answer 1

This is no problem of using the REST API. These calls are created by the Elasticsearch RestHighlevelClient.
Having multiple types in one index is not supported anymore by Elasticsearch since version 7.0.0. So you cannot model your data in this way.
Elasticsearch supports the join data type for this. We are currently working on a PR that will add the support for this for the next version of Spring Data Elasticsearch (4.1).

Answer 2

I was able to find a short-term solution with the following changes.
// Comment.kt
#Document(indexName = "message_board")
data class Comment(
#Id
val commentId: String,
val relationField: Map<String, String>
)
// Profile.kt
#Document(indexName = "message_board")
data class Profile(
#Id
val profileId: String,
#Field(type = FieldType.Short)
val age: Short,
#GeoPointField
val location: GeoPoint,
#Field(type = FieldType.Text)
val relationField: String = "profile"
)
// Controller.kt
class Controller constructor(
private val elasticsearchOperations: ElasticsearchOperations
) {
init {
elasticsearchOperations.indexOps(IndexCoordinates.of("message_board")).let { indexOp ->
if (!indexOp.exists() && indexOp.create()) {
val relationMap = Document.from(
mapOf(
"properties" to mapOf(
"relationField" to mapOf(
"type" to "join",
"relations" to mapOf(
"profile" to "comment"
)
),
"location" to mapOf(
"type" to "geo_point"
)
)
)
)
indexOp.putMapping(relationMap)
indexOp.refresh()
}
}
}
}
Note the relationField added to both data classes as well as the manually generated mapping document. Now, ES has a proper mapping on initialization:
{
"message_board" : {
"mappings" : {
"properties" : {
"location" : {
"type" : "geo_point"
},
"relationField" : {
"type" : "join",
"eager_global_ordinals" : true,
"relations" : {
"profile" : "comment"
}
}
}
}
}
}
Now, creating a profile is straightforward:
val profile = Profile(
profileId = UUID.randomUUID().toString(),
age = 27,
location = GeoPoint(3.0, 5.0)
)
val indexQuery = IndexQueryBuilder()
.withId(profile.profileId)
.withObject(profile)
.build()
elasticsearchOperations.index(indexQuery, IndexCoordinates.of("message_board"))
However, creating a comment is a little tricker because a routing ID is required:
val comment = Comment(
commentId = UUID.randomUUID().toString(),
relationField = mapOf(
"name" to "comment",
"parent" to profileId
)
)
val bulkOptions = BulkOptions.builder()
.withRoutingId(profileId)
.build()
val indexQuery = IndexQueryBuilder()
.withId(comment.commentId)
.withObject(comment)
.withParentId(profileId)
.build()
elasticsearchOperations.bulkIndex(listOf(indexQuery), bulkOptions, IndexCoordinates.of("message_board"))
This is how I was able to get a parent-child relation with the new JOIN relation type.