ElasticsearchRepository skip null values - spring

Question

I have the Repo to interact with ES index:
#Repository
public interface RegDocumentRepo extends ElasticsearchRepository<RegDocument, String> {
}
RegDocument class is a POJO of reg-document index:
#Document(indexName = "reg-document")
#Data
#AllArgsConstructor
#NoArgsConstructor
public class RegDocument {
#Id
String id;
#Field(type = FieldType.Nested, includeInParent = true)
private List<Map<String, Object>> attachments;
private String author;
#Field(type = FieldType.Nested, includeInParent = true)
private List<Map<String, Object>> classification;
private String content;
private String intent;
#Field(type = FieldType.Nested, includeInParent = true)
private List<Map<String, Object>> links;
private String name;
#Field(name = "publication_date")
private String publicationDate;
private Integer raiting;
private Long status;
private String title;
private String type;
private String version;
}
To hide my business-logic I use service:
#RequiredArgsConstructor
#Service
public class SearchServiceImpl {
#Autowired
RegDocumentRepo regDocumentRepo;
public RegDocument updateRating(String uuid, Integer rating) throws IOException {
final RegDocument regDocument = regDocumentRepo
.findById(uuid)
.orElseThrow(() -> new IOException(String.format("No document with %s id", uuid)));
Integer ratingFromDB = regDocument.getRaiting();
ratingFromDB = ratingFromDB == null ? rating : ratingFromDB + rating;
regDocument.setRaiting(ratingFromDB);
final RegDocument save = regDocumentRepo.save(regDocument);
return save;
}
}
So I had the such document in my ES index:
{
"_index" : "reg-document",
"_type" : "_doc",
"_id" : "9wEgQnQBKzq7IqBZMDaO",
"_score" : 1.0,
"_source" : {
"raiting" : null,
"attachments" : null,
"author" : null,
"type" : "answer",
"classification" : [
{
"code" : null,
"level" : null,
"name" : null,
"description" : null,
"id_parent" : null,
"topic_type" : null,
"uuid" : null
}
],
"intent" : null,
"version" : null,
"content" : "В 2019 году размер материнского капитала составляет 453026 рублей",
"name" : "Каков размер МСК в 2019 году?",
"publication_date" : "2020-08-26 06:49:10",
"rowkey" : null,
"links" : null,
"status" : 1
}
}
But after I update my ranking score, I have next structure:
{
"_index" : "reg-document",
"_type" : "_doc",
"_id" : "9wEgQnQBKzq7IqBZMDaO",
"_score" : 1.0,
"_source" : {
"raiting" : 4,
"type" : "answer",
"classification" : [
{
"code" : null,
"level" : null,
"name" : null,
"description" : null,
"id_parent" : null,
"topic_type" : null,
"uuid" : null
}
],
"content" : "В 2019 году размер материнского капитала составляет 453026 рублей",
"name" : "Каков размер МСК в 2019 году?",
"publication_date" : "2020-08-26 06:49:10",
"status" : 1
}
}
As you can see, Java service skip NULL values. But if the field is nested, null values were saved.
ElasticSearch version - 7.8.0
maven dependency for spring-data:
<dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-elasticsearch</artifactId>
<version>4.0.0.RELEASE</version>
</dependency>
So how can i SAVE null values, not skip them?
**
UDP
**
I have investigated spring-data-elasticsearch-4.0.0 dependency and find out, as Best Answer author said, that MappingElasticsearchConverter.java has following methods:
#Override
public void write(Object source, Document sink) {
Assert.notNull(source, "source to map must not be null");
if (source instanceof Map) {
// noinspection unchecked
sink.putAll((Map<String, Object>) source);
return;
}
Class<?> entityType = ClassUtils.getUserClass(source.getClass());
TypeInformation<?> type = ClassTypeInformation.from(entityType);
if (requiresTypeHint(type, source.getClass(), null)) {
typeMapper.writeType(source.getClass(), sink);
}
Optional<Class<?>> customTarget = conversions.getCustomWriteTarget(entityType, Map.class);
if (customTarget.isPresent()) {
sink.putAll(conversionService.convert(source, Map.class));
return;
}
ElasticsearchPersistentEntity<?> entity = type.getType().equals(entityType)
? mappingContext.getRequiredPersistentEntity(type)
: mappingContext.getRequiredPersistentEntity(entityType);
writeEntity(entity, source, sink, null);
}
This methods explain why nested data was saved as null and wasn't skip. It just put Map inside.
So the next method use reflection in such way. So if it is a null value, it's just skip it:
protected void writeProperties(ElasticsearchPersistentEntity<?> entity, PersistentPropertyAccessor<?> accessor,
MapValueAccessor sink) {
for (ElasticsearchPersistentProperty property : entity) {
if (!property.isWritable()) {
continue;
}
Object value = accessor.getProperty(property);
if (value == null) {
continue;
}
if (property.hasPropertyConverter()) {
ElasticsearchPersistentPropertyConverter propertyConverter = property.getPropertyConverter();
value = propertyConverter.write(value);
}
if (!isSimpleType(value)) {
writeProperty(property, value, sink);
} else {
Object writeSimpleValue = getWriteSimpleValue(value);
if (writeSimpleValue != null) {
sink.set(property, writeSimpleValue);
}
}
}
}
There is no official solution. So i have created a Jira ticket

Answer 1

The null values of the inner objects are stored, because this happens when the Map with null values for keys is stored.
Entity properties with a null value are not persisted by Spring Data Elasticsearch are not persisted as this it would store information that is not needed for saving/retrieving the data.
If you need the null values to be written, this would mean, that we'd need to add some flag to the #Field annotation for this, can you add an issue in Jira (https://jira.spring.io/projects/DATAES/issues) for this?
Edit: Implemented in versions 4.0.4.RELEASE and 4.1.0.RC1