How to format date correctly using Spring Data Elasticsearch - elasticsearch

Question

I'm using SpringBoot 2.2.5 with Elasticsearch 6.8.6. I'm in progress of migrating from Spring Data Jest to using the Spring Data Elasticsearch REST transport mechanism with ElasticsearchEntityMapper.
I have a Date field with the following definition:
#JsonFormat(shape = JsonFormat.Shape.STRING, pattern = "yyyy-MM-dd'T'HH:mm:ss.SSSZ")
#Field(type = FieldType.Date, format = DateFormat.custom, pattern = "yyyy-MM-dd'T'HH:mm:ss.SSSZ")
private Date date;
I would like the date stored in Elasticsearch like this:
"date": "2020-04-02T14:49:05.672+0000"
When I start the application, the index is created but when I try to save the entity I get the following exception:
Caused by: org.elasticsearch.client.ResponseException: method [POST], host [http://localhost:9200], URI [/trends/estrend?timeout=1m], status line [HTTP/1.1 400 Bad Request]
{"error":{"root_cause":[{"type":"mapper_parsing_exception","reason":"failed to parse field [date] of type [date] in document with id 'rS5UP3EB9eKtCTMXW_Ky'"}],"type":"mapper_parsing_exception","reason":"failed to parse field [date] of type [date] in document with id 'rS5UP3EB9eKtCTMXW_Ky'","caused_by":{"type":"illegal_argument_exception","reason":"Invalid format: \"1585905425266\" is malformed at \"5266\""}},"status":400}
Any pointers on what I'm doing wrong and what I should do to fix it?
Configuration and entity definitions below:
#Configuration
public class ElasticsearchConfig extends AbstractElasticsearchConfiguration {
#Value("${spring.data.elasticsearch.host}")
private String elasticSearchHost;
#Value("${spring.data.elasticsearch.port}")
private String elasticSearchPort;
#Bean
public RestHighLevelClient elasticsearchClient() {
final ClientConfiguration clientConfiguration = ClientConfiguration.builder()
.connectedTo(elasticSearchHost + ":" + elasticSearchPort)
.usingSsl()
.build();
return RestClients.create(clientConfiguration).rest();
}
#Bean
public EntityMapper entityMapper() {
ElasticsearchEntityMapper entityMapper = new ElasticsearchEntityMapper(elasticsearchMappingContext(), new DefaultConversionService());
entityMapper.setConversions(elasticsearchCustomConversions());
return entityMapper;
}
}
package com.es.test;
import java.util.Date;
import java.util.UUID;
import com.fasterxml.jackson.annotation.JsonFormat;
import org.springframework.data.annotation.Id;
import org.springframework.data.elasticsearch.annotations.DateFormat;
import org.springframework.data.elasticsearch.annotations.Document;
import org.springframework.data.elasticsearch.annotations.Field;
import org.springframework.data.elasticsearch.annotations.FieldType;
#Document(indexName = "trends")
public class EsTrend {
#Id
private UUID id;
#JsonFormat(shape = JsonFormat.Shape.STRING, pattern = "yyyy-MM-dd'T'HH:mm:ss.SSSZ")
#Field(type = FieldType.Date, format = DateFormat.custom, pattern = "yyyy-MM-dd'T'HH:mm:ss.SSSZ")
private Date date;
private String entityOrRelationshipId;
// getter and setters
}
Update:
If I disable the ElasticsearchEntityMapper bean, I don't get the exception and the date is written in the correct format to Elasticsearch. Is there anything else I need to configure for the ElasticsearchEntityMapper?

Answer 1

First, please don't use the Jackson based default mapper. It is removed in the next major version of Spring Data Elasticsearch (4.0). Then there will be no choice available, and internally the ElasticsearchEntityMapperis used.
As to your problem: The ElasticsearchEntityMapperin version 3.2, which is used by Spring Boot currently, does not use the date relevant information from the #Field attribute to convert the entity, it is only used for the index mappings creation. This was a missing feature or bug and is fixed in the next major version, the whole mapping process was overhauled there.
What you can do in your current situation: You need to add custom converters. You can do this in your configuration class like this:
#Configuration
public class ElasticsearchConfig extends AbstractElasticsearchConfiguration {
private static SimpleDateFormat formatter = new SimpleDateFormat("yyyy-MM-dd'T'HH:mm:ss.SSSZ");
#Value("${spring.data.elasticsearch.host}")
private String elasticSearchHost;
#Value("${spring.data.elasticsearch.port}")
private String elasticSearchPort;
#Bean
public RestHighLevelClient elasticsearchClient() {
final ClientConfiguration clientConfiguration = ClientConfiguration.builder()
.connectedTo(elasticSearchHost + ":" + elasticSearchPort)
.usingSsl()
.build();
return RestClients.create(clientConfiguration).rest();
}
#Bean
public EntityMapper entityMapper() {
ElasticsearchEntityMapper entityMapper = new ElasticsearchEntityMapper(elasticsearchMappingContext(), new DefaultConversionService());
entityMapper.setConversions(elasticsearchCustomConversions());
return entityMapper;
}
#Override
public ElasticsearchCustomConversions elasticsearchCustomConversions() {
return new ElasticsearchCustomConversions(Arrays.asList(DateToStringConverter.INSTANCE, StringToDateConverter.INSTANCE));
}
#WritingConverter
enum DateToStringConverter implements Converter<Date, String> {
INSTANCE;
#Override
public String convert(Date date) {
return formatter.format(date);
}
}
#ReadingConverter
enum StringToDateConverter implements Converter<String, Date> {
INSTANCE;
#Override
public Date convert(String s) {
try {
return formatter.parse(s);
} catch (ParseException e) {
return null;
}
}
}
}
You still need to have the dateformat in the #Field anotation though, because it is needed to create the correct index mappings.
And you should change your code to use the Java 8 introduced time classes like LocalDate or LocalDateTime, Spring Data Elasticsearch supports these out of the box, whereas java.util.Date would need custom converters.
Edit 09.04.2020: added the necessary #WritingConverter and #ReadingConverter annotations.
Edit 19.04.2020: Spring Data Elasticsearch 4.0 will support the java.util.Date class out of the box with the #Field annotation as well.

Answer 2

As I am a new joiner,I can't comment under #P.J.Meisch's anwser by the stack rules.
I also faced the problem, and solved it with #P.J.Meisch's anwser.
But just a little change with the #ReadingConverter.
Infact, the raw type read from ES, is Long, and the result type in java we need is LocalDateTime. Thus, the read converter shoud be Long to LocalDateTime.
Code follows below:
#Configuration
public class ElasticsearchClientConfig extends AbstractElasticsearchConfiguration {
public final static int TIME_OUT_MILLIS = 50000;
#Autowired
private ElasticsearchProperties elasticsearchProperties;
#Override
#Bean
public RestHighLevelClient elasticsearchClient() {
final ClientConfiguration clientConfiguration = ClientConfiguration.builder()
.connectedTo(elasticsearchProperties.getHost() + ":" + elasticsearchProperties.getPort())
.withBasicAuth(elasticsearchProperties.getName(), elasticsearchProperties.getPassword())
.withSocketTimeout(TIME_OUT_MILLIS)
.withConnectTimeout(TIME_OUT_MILLIS)
.build();
return RestClients.create(clientConfiguration).rest();
}
/**
* Java LocalDateTime to ElasticSearch Date mapping
*
* #return EntityMapper
*/
#Override
#Bean
public EntityMapper entityMapper() {
ElasticsearchEntityMapper entityMapper = new ElasticsearchEntityMapper(elasticsearchMappingContext(), new DefaultConversionService());
entityMapper.setConversions(elasticsearchCustomConversions());
return entityMapper;
}
#Override
public ElasticsearchCustomConversions elasticsearchCustomConversions() {
return new ElasticsearchCustomConversions(Arrays.asList(DateToStringConverter.INSTANCE, LongToLocalDateTimeConverter.INSTANCE));
}
#WritingConverter
enum DateToStringConverter implements Converter<Date, String> {
/**
* instance
*/
INSTANCE;
#Override
public String convert(#NonNull Date date) {
return DateUtil.format(date, DateConstant.TIME_PATTERN);
}
}
**#ReadingConverter
enum LongToLocalDateTimeConverter implements Converter<Long, LocalDateTime> {
/**
* instance
*/
INSTANCE;
#Override
public LocalDateTime convert(#NonNull Long s) {
return LocalDateTime.ofInstant(Instant.ofEpochMilli(s), ZoneId.systemDefault());
}
}**
}
and the DateUtil file:
public class DateUtil {
/**
* lock obj
*/
private static final Object LOCK_OBJ = new Object();
/**
* sdf Map for different pattern
*/
private static final Map<String, ThreadLocal<SimpleDateFormat>> LOCAL_MAP = new HashMap<>();
/**
* thread safe
*
* #param pattern pattern
* #return SimpleDateFormat
*/
private static SimpleDateFormat getSdf(final String pattern) {
ThreadLocal<SimpleDateFormat> tl = LOCAL_MAP.get(pattern);
if (tl == null) {
synchronized (LOCK_OBJ) {
tl = LOCAL_MAP.get(pattern);
if (tl == null) {
System.out.println("put new sdf of pattern " + pattern + " to map");
tl = ThreadLocal.withInitial(() -> {
System.out.println("thread: " + Thread.currentThread() + " init pattern: " + pattern);
return new SimpleDateFormat(pattern);
});
LOCAL_MAP.put(pattern, tl);
}
}
}
return tl.get();
}
/**
* format
*
* #param date date
* #param pattern pattern
* #return String
*/
public static String format(Date date, String pattern) {
return getSdf(pattern).format(date);
}
}
at last,
pls vote for #P.J.Meisch, not me.