Hi I have a question on how to create an elastic search query for a nested composite object with date ranges and additional field parameters like so
[{
"name": "A",
"availability": [
{
"partial": true,
"dates": {
"gte": "2020-12-01",
"lte": "2020-12-02"
}
}
]
},
{
"name": "B",
"availability": [
{
"partial": true,
"dates": {
"gte": "2020-12-05",
"lte": "2020-12-06"
}
},
{
"partial": false,
"dates": {
"gte": "2020-12-08",
"lte": "2020-12-11"
}
}
]
}]
This is my entity data
#Document(indexName = "workers")
public class Worker {
#Id
private String id;
#Field(type = FieldType.Text)
private String name;
#Field(type = FieldType.Nested)
private List<Availability> availability;
}
public class Availability {
#Field(type = FieldType.Boolean)
private boolean partial;
#Field(type = FieldType.Date_Range, format = DateFormat.custom, pattern = "uuuu-MM-dd")
private Map<String, LocalDate> dates;
}
This is the search query that I have currently written, but the results come as empty
final BoolQueryBuilder queryBuilder = QueryBuilders.boolQuery();
queryBuilder.must(QueryBuilders.termQuery("availability.partial", query.isPartial()));
RangeQueryBuilder availability = QueryBuilders.rangeQuery("availability.dates")
.gte(query.getStartDate())
.lte(query.getEndDate());
queryBuilder.must(availability);
Pageable pageable = PageRequest.of(pageNumber, pageSize);
// #formatter:off
return new NativeSearchQueryBuilder()
.withPageable(pageable)
.withQuery(queryBuilder)
.build();
This is my query dto
public class WorkerQuery {
private boolean partial;
private LocalDate startDate;
private LocalDate endDate;
}
// Request data
{
"partial": true,
"startDate": "2020-12-01",
"endDate": "2020-12-02"
}
Great start!! You're just missing a nested query since availability is nested. The Java query needs to be like this:
final BoolQueryBuilder queryBuilder = QueryBuilders.boolQuery();
queryBuilder.must(QueryBuilders.termQuery("availability.partial", query.isPartial()));
RangeQueryBuilder availability = QueryBuilders.rangeQuery("availability.dates")
.gte(query.getStartDate())
.lte(query.getEndDate())
.relation("within");
queryBuilder.must(availability);
final NestedQueryBuilder nested = QueryBuilders.nestedQuery("availability", queryBuilder);
Pageable pageable = PageRequest.of(pageNumber, pageSize);
// #formatter:off
return new NativeSearchQueryBuilder()
.withPageable(pageable)
.withQuery(nested)
.build();
Related
The target is to query specific fields from an index via a spring boot app.
Questions in the end.
The data in elasticsearch are created from Elastic Stack with Beats and Logstash etc. There is some inconsistency, eg some fields may be missing on some hits.
The spring app does not add the data and has no control on the fields and indexes
The query I need, with _source brings
GET index-2022.07.27/_search
{
"from": 0,
"size": 100,
"_source": ["#timestamp","message", "agent.id"],
"query": {
"match_all": {}
}
}
brings the hits as
{
"_index": "index-2022.07.27",
"_id": "C1zzPoIBgxar5OgxR-cs",
"_score": 1,
"_ignored": [
"event.original.keyword"
],
"_source": {
"agent": {
"id": "ddece977-9fbb-4f63-896c-d3cf5708f846"
},
"#timestamp": "2022-07-27T09:18:27.465Z",
"message": """a message"""
}
},
and with fields instead of _source is
{
"_index": "index-2022.07.27",
"_id": "C1zzPoIBgxar5OgxR-cs",
"_score": 1,
"_ignored": [
"event.original.keyword"
],
"fields": {
"#timestamp": [
"2022-07-27T09:18:27.465Z"
],
"agent.id": [
"ddece977-9fbb-4f63-896c-d3cf5708f846"
],
"message": [
"""a message"""
]
}
},
How can I get this query with Spring Boot ?
I lean on StringQuery with the RestHighLevelClient as below but cant get it to work
Query searchQuery = new StringQuery("{\"_source\":[\"#timestamp\",\"message\",\"agent.id\"],\"query\":{\"match_all\":{}}}");
SearchHits<Items> productHits = elasticsearchOperations.search(
searchQuery,
Items.class,
IndexCoordinates.of(CURRENT_INDEX));
What form must Items.class have? What fields?
I just need timestamp, message, agent.id. The later is optional, it may not exist.
How will the mapping work?
versions:
Elastic: 8.3.2
Spring boot: 2.6.6
elastic (mvn): 7.15.2
spring-data-elasticsearch (mvn): 4.3.3
official documentation states that with RestHighLevelClient the versions should be supported
Support for upcoming versions of Elasticsearch is being tracked and
general compatibility should be given assuming the usage of the
high-level REST client.
You can define an entity class for the data you want to read (note I have a nested class for the agent):
#Document(indexName = "index-so", createIndex = false)
public class SO {
#Id
private String id;
#Field(name = "#timestamp", type = FieldType.Date, format = DateFormat.date_time)
private Instant timestamp;
#Field(type = FieldType.Object)
private Agent agent;
#Field(type = FieldType.Text)
private String message;
public String getId() {
return id;
}
public void setId(String id) {
this.id = id;
}
public Instant getTimestamp() {
return timestamp;
}
public void setTimestamp(Instant timestamp) {
this.timestamp = timestamp;
}
public Agent getAgent() {
return agent;
}
public void setAgent(Agent agent) {
this.agent = agent;
}
public String getMessage() {
return message;
}
public void setMessage(String message) {
this.message = message;
}
class Agent {
#Field(name = "id", type = FieldType.Keyword)
private String id;
public String getId() {
return id;
}
public void setId(String id) {
this.id = id;
}
}
}
The query then would be:
var query = new NativeSearchQueryBuilder()
.withQuery(matchAllQuery())
.withSourceFilter(new FetchSourceFilter(
new String[]{"#timestamp", "message", "agent.id"},
new String[]{}))
.build();
var searchHits = operations.search(query, SO.class);
I'm trying to get data from multiple tables and put it in Array List of class, and then convert it to JSON Object.
But when i'm trying to parse it to json using Jackson Object Mapper all the lists are converted as below
Using ObjectMapper().writeValueAsString for deserialization from class objects to json
```{
"College": [
{
"institution": [
{
"instId": "T34",
"Country": "India",
"Code": "T33"
},
{
"instId": "T22",
"Country": "India",
"Code": "T22"
}
],
"Rating": [
{
"star": "4"
"comments": "good"
},
{
"star": "2"
"comments": "ok"
},
}
]
}```
But i want the result as below
{
"College": [
{
"institution": [
{
"instId": "T34",
"Country": "India",
"Code": "T33"
}
],
"Rating": [
{
"star": "4"
"comments": "good"
}
]
},
{
"institution": [
{
"instId": "T22",
"Country": "India",
"Code": "T22"
}
],
"Rating": [
{
"star": "2"
"comments": "ok"
}
]
}
]
}
The above is just an example.
Please help in getting the desired output.
Below are the class files used.
public class AllCollege{
List<College> college = new ArrayList<>();
public List<College> getCollege() {
return college;
}
public void setCollege(List<College> college) {
this.college = college;
}
}
public class College{
private List<Institution> institution = new ArrayList<>();
private List<Rating> rating = new ArrayList<>();
public List<Institution> getInstitution() {
return institution;
}
public void setInstitution(List<Institution> institution) {
this.institution = institution;
}
public List<Rating> getRating() {
return rating;
}
public void setRating(List<Rating> rating) {
this.rating = rating;
}
}
public class Institution {
private String instId;
private String country;
private String code;
public String getInstId() {
return instId;
}
public void setInstId(String instId) {
this.instId = instId;
}
public String getCountry() {
return country;
}
public void setCountry(String country) {
this.country = country;
}
public String getCode() {
return code;
}
public void setCode(String code) {
this.code = code;
}
}
public class Rating {
private String star;
private String comments;
public String getStar() {
return star;
}
public void setStar(String star) {
this.star = star;
}
public String getComments() {
return comments;
}
public void setComments(String comments) {
this.comments = comments;
}
}
Below is where the data from tables is set into ArrayList and then converted to json string.
session = sessionFactory.openSession();
String sql = "from institution";
Query<InstDto> query = session.createQuery(sql);
List<Institution> configdtoList =query.list();
College alc = new College();
alc.setInstitution(configdtoList);
.
.
.
similarly Rating table.
List<College> clist = new new ArrayList<>();
clist.add(alc);
AllCollege ac = new AllCollege();
ac.setCollege(clist);
String responseJson = new ObjectMapper().writeValueAsString(ac)
class structure as below it will help you to parse:
public class Sample {
#JsonProperty("College")
private List<College> college;
}
public class College {
private List<Institution> institution;
#JsonProperty("Rating")
private List<Rating> rating;
}
public class Rating {
private String comments;
private String star;
}
public class Institution {
#JsonProperty("Code")
private String code;
#JsonProperty("Country")
private String country;
private String instId;
}
I have created an HashMap contains the List<AllCollege> as value and then used json parser which worked as expected.
i reindex data from es 2.4 to 5.6.
data in es 2.4 have 2 types,and the 2 type is parent-child relation.
when reindex it to es 5.6,the index only contains single type,the parent-child relation by using join type to resolving.
the data above works ok.
the mapping example like this, it contains a join type:
"mappings": {
"doc": {
"properties": {
"my_join_field": {
"eager_global_ordinals": true,
"type": "join",
"relations": {
"question": "answer"
}
},
"name": {
"type": "text",
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
}
}
}
}
}
how to mapping join type by using spring data elasticSearch:
in old version code es 2.4,i can mapping it like this:
#Document(indexName = ParentEntity.INDEX, type = ParentEntity.PARENT_TYPE, shards = 1, replicas = 0, refreshInterval = "-1")
public class ParentEntity {
public static final String INDEX = "parent-child";
public static final String PARENT_TYPE = "parent-entity";
public static final String CHILD_TYPE = "child-entity";
#Id
private String id;
#Field(type = FieldType.Text, store = true)
private String name;
public ParentEntity() {
}
public ParentEntity(String id, String name) {
this.id = id;
this.name = name;
}
public String getId() {
return id;
}
public String getName() {
return name;
}
#Override
public String toString() {
return new ToStringCreator(this).append("id", id).append("name", name).toString();
}
#Document(indexName = INDEX, type = CHILD_TYPE, shards = 1, replicas = 0, refreshInterval = "-1")
public static class ChildEntity {
#Id
private String id;
#Field(type = FieldType.Text, store = true)
#Parent(type = PARENT_TYPE)
private String parentId;
#Field(type = FieldType.Text, store = true)
private String name;
public ChildEntity() {
}
public ChildEntity(String id, String parentId, String name) {
this.id = id;
this.parentId = parentId;
this.name = name;
}
public String getId() {
return id;
}
public String getParentId() {
return parentId;
}
public String getName() {
return name;
}
#Override
public String toString() {
return new ToStringCreator(this).append("id", id).append("parentId", parentId).append("name", name).toString();
}
}
}
how can i Mapping join type by using spring data elasticSearch v3.0.10?
Today, i tried the entity below to working at spring data elasticSearch 3.0.10:
#Document(indexName = "join_index", type = "join_mapping")
#Data
public class JoinEntity {
#Id
private String id;
#Mapping(mappingPath = "/mappings/join_type.json")
private Map<String,String> relationType;
#Field(type = FieldType.Keyword)
private String name;
//#Parent(type = "question")
#Field(type = FieldType.Keyword)
private String parentId;
}
join_type.json below:
{
"type": "join",
"relations": {
"question": "answer"
}
}
it create index and put mapping work ok:
#RunWith(SpringJUnit4ClassRunner.class)
#ContextConfiguration("classpath:elasticsearch-template-test.xml")
public class ElasticsearchTemplateJoinTests {
#Autowired
private ElasticsearchTemplate elasticsearchTemplate;
#Before
public void before() {
clean();
elasticsearchTemplate.deleteIndex(JoinEntity.class);
elasticsearchTemplate.createIndex(JoinEntity.class);
elasticsearchTemplate.putMapping(JoinEntity.class);
elasticsearchTemplate.refresh(JoinEntity.class);
}
#Test
public void shouldCreateIndexAndMappingSuccess(){
Map mapping = elasticsearchTemplate.getMapping(JoinEntity.class);
assertThat(mapping, is(notNullValue()));
Map properties = (Map) mapping.get("properties");
assertThat(properties, is(notNullValue()));
assertThat(properties.containsKey("name"), is(true));
Map file = (Map) properties.get("relationType");
assertThat(file, is(notNullValue()));
assertThat(((String) file.get("type")), is("join"));
}
}
when index parent work ok too,but index child it throws exception:
#Test
public void shouldIndexParentAndChildSuccess(){
JoinEntity parenEntity = new JoinEntity();
parenEntity.setName("parent_name");
parenEntity.setRelationType(Collections.singletonMap("name","question"));
IndexQuery parentQuery = new IndexQueryBuilder().withId("11").withObject(parenEntity).build();
final String id = elasticsearchTemplate.index(parentQuery);
assertThat("11",is(id));
JoinEntity childEntity = new JoinEntity();
childEntity.setName("child_name");
Map<String,String> joinRelation = new HashMap<>(2);
joinRelation.put("name","answer");
joinRelation.put("parent", "11");
childEntity.setRelationType(joinRelation);
childEntity.setParentId("11");
IndexQuery childQuery = new IndexQueryBuilder().withId("22").withObject(childEntity).build();
elasticsearchTemplate.index(childQuery);
}
exception:
MapperParsingException[failed to parse
]; nested: IllegalArgumentException[[routing] is missing for join field [relationType]];
at org.elasticsearch.index.mapper.DocumentParser.wrapInMapperParsingException(DocumentParser.java:171)
how can i resolve this problem or Mapping the new version Parent-child relation correctly?thks!!
Elasticsearch needs the parent document routing parameter when you index child document check this
This is because both parent and child documents must be indexed in same shard to join to work.
However I couldn't figure out a way to solve this using Spring data elasticsearch. The only way that worked was using RestHighLevelClient
The recent version of Spring Data ElasticSearch had added support for this doc
Your child indexing would be something like,
IndexRequest indexRequest = new IndexRequest();
indexRequest.source(objectMapper.writeValueAsString(childEntity),XContentType.JSON);
indexRequest.id("22"); //child doc id
indexRequest.index(INDEX_NAME);
indexRequest.type(INDEX_TYPE);
indexRequest.routing("11"); //parent doc id
restHighLevelClient.index(indexRequest);
Finally, i gived up the parent-child relation, i split them into two separate indexs. some advance feature should be used less if not neccessary.
I need help to query nested documents. Using Spring Boot with MongoDB.
Structure:
public class Holiday {
#Id
private String id;
private Integer year;
private Map<String, List<HolidayElement>> holidays = new HashMap<>();
}
public class HolidayElement {
private String name;
#JsonFormat(pattern="yyyy-MM-dd")
private Date date;
private String note;
}
After saving everything the Json looks like:
[
{
"id": "5a153331b3cb1f0001e1edeb",
"year": 2017,
"holidays": {
"BB": [
{
"name": "Neujahrstag",
"date": "2017-01-01",
"note": ""
},
...
],
"HH": [
{ ... }
]
}
]
Now how can I get for instance: List of "HolidayElement" where the State is "BB"?
Assuming you have a repository like HolidayRepository, you need to create a custom implementation since you want to use MongoTemplate. So your HolidayRepository will look like
#Repository
public interface HolidayRepository extends MongoRepository<Holiday, String>, HolidayRepositoryCustom {
}
And declare two new files HolidayRepositoryCustom and HolidayRepositoryImpl in the same directory(very important) as HolidayRepository
public interface HolidayRepositoryCustom {
List<HolidayElement> findByMapId(final String mapId);
}
And the Impl class will look like this
public class HolidayRepositoryImpl implements HolidayRepositoryCustom {
private final MongoTemplate mongoTemplate;
#Autowired
public HolidayRepositoryImpl(final MongoTemplate mongoTemplate) {
this.mongoTemplate = mongoTemplate;
}
#Override
public List<HolidayElement> findByMapId(String mapId) {
final QueryBuilder queryBuilder = QueryBuilder.start();
queryBuilder
.and("holidays."+mapId).exists(true);
final DBObject projection = new BasicDBObject();
projection.put("holidays."+mapId, 1);
String collectionName = "Holiday";//Change to your collection name
try( final DBCursor dbCursor = mongoTemplate.getCollection(collectionName).find(queryBuilder.get(), projection)){
if(dbCursor.hasNext()){
DBObject next = dbCursor.next();
Map<String, List<HolidayElement>> holidayElements =
(Map<String, List<HolidayElement>>) next.get("holidays");
return holidayElements.get(mapId);
}
}
return Lists.newArrayList();
}
}
I need to generate a result with the number of alerts of each level for each user.
A structure similar to the following:
{
"identitity": "59e3b9dc5a3254691f327b67",
"alerts": [
{
"level": "INFO",
"count": "3"
},
{
"level": "ERROR",
"count": "10"
}
]
}
The alert entitity has the following structure:
#Document(collection = AlertEntity.COLLECTION_NAME)
public class AlertEntity {
public final static String COLLECTION_NAME = "alerts";
#Id
private ObjectId id;
#Field
private AlertLevelEnum level = AlertLevelEnum.INFO;
#Field("title")
private String title;
#Field("payload")
private String payload;
#Field("create_at")
private Date createAt = new Date();
#Field("delivered_at")
private Date deliveredAt;
#Field("delivery_mode")
private AlertDeliveryModeEnum deliveryMode =
AlertDeliveryModeEnum.PUSH_NOTIFICATION;
#Field("parent")
#DBRef
private ParentEntity parent;
#Field("son")
#DBRef
private SonEntity son;
private Boolean delivered = Boolean.FALSE;
}
I have implemented the following method tried to project the result in a nested way. But the "Identity" field is always null and the "alerts" field is a empty collection.
#Override
public List<AlertsBySonDTO> getAlertsBySon(List<String> sonIds) {
TypedAggregation<AlertEntity> alertsAggregation =
Aggregation.newAggregation(AlertEntity.class,
Aggregation.group("son.id", "level").count().as("count"),
Aggregation.project().and("son.id").as("id")
.and("alerts").nested(
bind("level", "level").and("count")));
// Aggregation.match(Criteria.where("_id").in(sonIds)
AggregationResults<AlertsBySonDTO> results = mongoTemplate.
aggregate(alertsAggregation, AlertsBySonDTO.class);
List<AlertsBySonDTO> alertsBySonResultsList = results.getMappedResults();
return alertsBySonResultsList;
}
The result I get is the following:
{
"response_code_name": "ALERTS_BY_SON",
"response_status": "SUCCESS",
"response_http_status": "OK",
"response_info_url": "http://yourAppUrlToDocumentedApiCodes.com/api/support/710",
"response_data": [
{
"identity": null,
"alerts": []
},
{
"identity": null,
"alerts": []
}
],
"response_code": 710
}
The result DTO is as follows:
public final class AlertsBySonDTO implements Serializable {
private static final long serialVersionUID = 1L;
#JsonProperty("identity")
private String id;
#JsonProperty("alerts")
private ArrayList<Map<String, String>> alerts;
public AlertsBySonDTO() {
super();
}
public AlertsBySonDTO(String id, ArrayList<Map<String, String>> alerts) {
super();
this.id = id;
this.alerts = alerts;
}
public String getId() {
return id;
}
public void setId(String id) {
this.id = id;
}
public ArrayList<Map<String, String>> getAlerts() {
return alerts;
}
public void setAlerts(ArrayList<Map<String, String>> alerts) {
this.alerts = alerts;
}
}
What needs to be done to project the result in a nested way?
Thanks in advance
In aggregation framework there is an $unwind operator which will basically transform your one element collection with nested array of two elements to two separate documents with one element from this array. So you'll get:
{
"identitity": "59e3b9dc5a3254691f327b67",
"alerts": {
"level": "INFO",
"count": "3"
}
}
{
"identitity": "59e3b9dc5a3254691f327b67",
"alerts": {
"level": "ERROR",
"count": "10"
}
}
And this is where you can start your group by with count. Should be working fine.