Best way to combined hibernate search queries on different indexes - elasticsearch

Question

We have the following situation
Given the following two entities
#Indexed
#Spatial(spatialMode = SpatialMode.HASH)
#Entity
#Table(name = "address")
Address{
#Field
#Basic
#Column(name = "state")
private String state;
#Field
#Basic
#Column(name = "town_city")
private String townCity;
#Field
#Longitude
#Basic
#Column(name = "x_coord")
private Double xCoord;
#Field
#Latitude
#Basic
#Column(name = "y_coord")
private Double yCoord;
}
And
#Indexed
#Entity
#Table(name = "person")
Person{
#Field
#Column(name = "weight")
private Double weight;
#Column(name = "age")
private Integer age;
#org.hibernate.annotations.Cache(usage =
org.hibernate.annotations.CacheConcurrencyStrategy.READ_WRITE)
#ManyToMany
#Cascade({org.hibernate.annotations.CascadeType.SAVE_UPDATE})
#JoinTable(name = "person_address",
joinColumns = {#JoinColumn(name = "person_id")},
inverseJoinColumns = {#JoinColumn(name = "address_id")})
private Set<Address> addressSet = new HashSet<>();
}
Getters and Setters rest of the fields omitted
We want to return in our search results as an example people within a 5KM radius of a given position who are also within an age range.
So
FullTextSession fullTextSession = Search.getFullTextSession(entityManagerFactory.unwrap(SessionFactory.class).openSession());
this.queryBuilder = fullTextSession.getSearchFactory()
.buildQueryBuilder().forEntity(Person.class)
.overridesForField("identifiers.identifier_edge", "identifier_query_analyzer")
.get();
this.bool = queryBuilder.bool();
LocalDateTime lowerLocalDateTime = localDateTime.withYear(localDateTime.getYear() - upperAge);
lowerDate = Date.from(lowerLocalDateTime.atZone(ZoneId.systemDefault()).toInstant());
LocalDateTime upperLocalDateTime = localDateTime.withYear(localDateTime.getYear() - lowerAge);
upperDate = Date.from(upperLocalDateTime.atZone(ZoneId.systemDefault()).toInstant());
bool.must(getQueryBuilder().range().onField("datesOfBirth.dateOfBirth").from(lowerDate).to(upperDate).createQuery());
which will give us people within the relevant age range
We have a separte query to get the address id's within a radius around a given point
public Set<Integer> getSpatialAddressResults(SpatialSearchCommand spatialSearchCommand) {
FullTextSession fullTextSession = Search.getFullTextSession(entityManagerFactory.unwrap(SessionFactory.class).openSession());
this.userSearchPreference = userSearchPreference;
this.queryBuilder = fullTextSession.getSearchFactory()
.buildQueryBuilder().forEntity(Address.class)
.get();
this.bool = queryBuilder.bool();
Set<Integer> addressIdSet = new HashSet<>();
bool.must(getQueryBuilder().spatial()
.within(spatialSearchCommand.getRadius(), Unit.KM).ofLatitude
(spatialSearchCommand.getLat()).andLongitude(spatialSearchCommand.getLng()).createQuery());
FullTextQuery fullTextQuery =
fullTextSession.createFullTextQuery(bool.createQuery(), Address.class)
.setProjection("addressId")
.initializeObjectsWith(ObjectLookupMethod.SECOND_LEVEL_CACHE,
DatabaseRetrievalMethod.QUERY);
List results = fullTextQuery.list();
for (Object result : results) {
Object[] arrayResult = (Object[]) result;
addressIdSet.add(((Integer) arrayResult[0]));
}
if (addressIdSet.size() == 0) {
addressIdSet.add(-1);
}
return addressIdSet;
}
Which we use like below (in reality these are done in separate classes but for simplicity I have just shown the relevant code
Set<Integer> localAddressIds = getSpatialAddressResults(new SpatialSearchCommand(userSearchPreference.getRadius(), userSearchPreference.getLat(), userSearchPreference.getLng()));
if(localAddressIds.size() > 0){
BooleanJunction<BooleanJunction> localSquQueryBool = getQueryBuilder().bool();
for (Integer localAddressId : localAddressIds) {
localSquQueryBool.should(getQueryBuilder().keyword().onField("currentLocation.address.indexId").matching(localAddressId).createQuery());
if(!personSearchCommand.getCurrentOnly()){
localSquQueryBool.should(getQueryBuilder().keyword().onField("locations.address.indexId").matching(localAddressId).createQuery());
}
}
bool.must(localSquQueryBool.createQuery());
}
The problem is there can be a huge amount of addresses returned which results in a BooleanQueryTooManyClauses: maxClauseCount is set to 1024
The question really is what is the best way to combined queries on two different indexed entities to avoid problems like above.

Answer 1

Essentially, you are trying to implement a join operation. As you can see, there are technical challenges to joins which are not easily solved on the client side.
Generally, the recommended approach in Elasticsearch and Lucene is to avoid joins when you can. Instead, you will de-normalize your schema: within the document representing each person, embed a copy of every address. Then you will be able to express all your constraints in a single query targeting the person index.
This is done by annotating the addresses property in Person with #IndexedEmbedded.
Now, as you can imagine, this de-normalization comes at a cost: whenever an address is changed, Hibernate Search has to update the relevant person.
To that end, you will need to add a List<Person> property to your Address class and annotate it with #ContainedIn in order for Hibernate Search to be able to fetch the persons to reindex whenever an address is modified.
In short, change your model to this:
//#Indexed // No longer needed
#Spatial(spatialMode = SpatialMode.HASH, name = "location") // Give a name to the spatial field
#Entity
#Table(name = "address")
Address {
// Add this
#ManyToMany(mappedBy = "addressSet")
#ContainedIn
private Set<Person> personSet = new HashSet<>();
#Field
#Basic
#Column(name = "state")
private String state;
#Field
#Basic
#Column(name = "town_city")
private String townCity;
//#Field// This is not necessary
#Longitude
#Basic
#Column(name = "x_coord")
private Double xCoord;
//#Field// This is not necessary
#Latitude
#Basic
#Column(name = "y_coord")
private Double yCoord;
}
#Indexed
#Entity
#Table(name = "person")
Person{
#Field
#Column(name = "weight")
private Double weight;
#Column(name = "age")
private Integer age;
#org.hibernate.annotations.Cache(usage =
org.hibernate.annotations.CacheConcurrencyStrategy.READ_WRITE)
#ManyToMany
#Cascade({org.hibernate.annotations.CascadeType.SAVE_UPDATE})
#JoinTable(name = "person_address",
joinColumns = {#JoinColumn(name = "person_id")},
inverseJoinColumns = {#JoinColumn(name = "address_id")})
#IndexedEmbedded // Add this
private Set<Address> addressSet = new HashSet<>();
#Transient
#IndexedEmbedded // Also add this
public Address getCurrentAddress() {
// This was missing in your schema, I suppose it's a getter that picks the current address from addressSet?
}
}
Then reindex. Your Person documents will now have two new fields: addressSet.location and currentAddress.location.
Then write your query like this:
FullTextSession fullTextSession = Search.getFullTextSession(entityManagerFactory.unwrap(SessionFactory.class).openSession());
this.queryBuilder = fullTextSession.getSearchFactory()
.buildQueryBuilder().forEntity(Person.class)
.overridesForField("identifiers.identifier_edge", "identifier_query_analyzer")
.get();
this.bool = queryBuilder.bool();
LocalDateTime lowerLocalDateTime = localDateTime.withYear(localDateTime.getYear() - upperAge);
lowerDate = Date.from(lowerLocalDateTime.atZone(ZoneId.systemDefault()).toInstant());
LocalDateTime upperLocalDateTime = localDateTime.withYear(localDateTime.getYear() - lowerAge);
upperDate = Date.from(upperLocalDateTime.atZone(ZoneId.systemDefault()).toInstant());
bool.must(getQueryBuilder().range().onField("datesOfBirth.dateOfBirth").from(lowerDate).to(upperDate).createQuery());
SpatialSearchCommand spatialSearchCommand = new SpatialSearchCommand(userSearchPreference.getRadius(), userSearchPreference.getLat(), userSearchPreference.getLng());
// The magic happens below
BooleanJunction<BooleanJunction> localSquQueryBool = getQueryBuilder().bool();
localSquQueryBool.should(getQueryBuilder().spatial()
.onField("currentAddress.location")
.within(spatialSearchCommand.getRadius(), Unit.KM)
.ofLatitude(spatialSearchCommand.getLat())
.andLongitude(spatialSearchCommand.getLng())
.createQuery());
if(!personSearchCommand.getCurrentOnly()) {
localSquQueryBool.should(getQueryBuilder().spatial()
.onField("addressSet.location")
.within(spatialSearchCommand.getRadius(), Unit.KM)
.ofLatitude(spatialSearchCommand.getLat())
.andLongitude(spatialSearchCommand.getLng())
.createQuery());
}
bool.must(localSquQueryBool.createQuery());