How do I implement tenant-based routing for elasticsearch in JHipster? - elasticsearch

Question

I´m currently trying to implement multi-tenancy into my JHipster microservices. However, I can't find a way to implement tenant-based routing for elasticsearch.
So far I have managed to implement datasource routing for the PostgreSQL DBs similar to the following article: https://websparrow.org/spring/spring-boot-dynamic-datasource-routing-using-abstractroutingdatasource
When I started looking for ways to implement multi tenancy in elasticsearch, I found the following article: https://techblog.bozho.net/elasticsearch-multitenancy-with-routing/
There I read about tenant-based routing. First I tried looking it up on the internet, but anything I found was either over 5 years old or not related to java, much less to Spring/Jhipster. Then I tried looking into the methods of ElasticSearchTemplate, the annotation variables of #Document and #Settings and the configuration options in the .yml file, but didn't find anything useful.
I'm currently using Jhipster version 7.9.3, which uses the Spring-Boot version 2.7.3. All the microservices were created with JDL and on half of them I put elasticsearch into the configuration. The other half does not matter.
Edit: I want to add that multi-tenancy in my database is archived by database separation(Tenant1 uses DB1, Tenant2 uses DB2 etc.). The tenant variable is an enum and not included in my entities.
Edit2: I implemented my own solution. I use the tenants as indexes and use my ContextHolder from DataSource Routing to route to the correct tenant index. For that I had to do some changes the elasticsearchTemplate in the generated classes of the package "<my.package.name>.repository.search".
It might not be the most efficient way to reach multi tenancy with elasticsearch, but it doesn't need much configuration.
Here is the code:
public interface ProductSearchRepository extends ElasticsearchRepository<Product, Long>, ProductSearchRepositoryInternal {}
interface ProductSearchRepositoryInternal {
Stream<Product> search(String query);
Stream<Product> search(Query query);
void index(Product entity);
}
class ProductSearchRepositoryInternalImpl implements ProductSearchRepositoryInternal {
private final ElasticsearchRestTemplate elasticsearchTemplate;
private final ProductRepository repository;
ProductSearchRepositoryInternalImpl(ElasticsearchRestTemplate elasticsearchTemplate, ProductRepository repository) {
this.elasticsearchTemplate = elasticsearchTemplate;
this.repository = repository;
}
#Override
public Stream<Product> search(String query) {
NativeSearchQuery nativeSearchQuery = new NativeSearchQuery(queryStringQuery(query));
return search(nativeSearchQuery);
}
#Override
public Stream<Product> search(Query query) {
return elasticsearchTemplate.search(query, Product.class, IndexCoordinates.of(TenantContextHolder.getTenantContext().getTenant())).map(SearchHit::getContent).stream();
}
#Override
public void index(Product entity) {
repository.findById(entity.getId()).ifPresent(t -> elasticsearchTemplate.save(t, IndexCoordinates.of(TenantContextHolder.getTenantContext().getTenant())));
}
}
Edit3: Since people might not know where ".getTenant()" comes from, I'll show my tenant enumeration:
public enum Tenant {
TENANTA("tenant_a"),
TENANTB("tenant_b");
String tenant;
Tenant(String name) {
this.tenant=name;
}
public String getTenant() {
return this.tenant;
}
}
Edit4: My solution is not working as planned. I will give an update once I found a better and more robust solution.
Edit5: I have found out how to implement tenant-based routing. First you have to add the following Annotation to your entities:
#org.springframework.data.elasticsearch.annotations.Routing(value = "tenant")
In my case I had to include the enum "Tenant" into my entities along with the getter and setter:
#Transient
private Tenant tenant;
public Tenant getTenant() {
return tenant;
}
public void setTenant(Tenant tenant) {
this.tenant = tenant;
}
Then I have to set the tenant during the processing of a REST request before it gets indexed by elasticsearchtemplate:
entity.setTenant(TenantContextHolder.getTenantContext());
As for the search function, I had to add a term query as a filter to enable routing:
#Override
public Stream<Product> search(String query) {
NativeSearchQuery nativeSearchQuery = new NativeSearchQuery(queryStringQuery(query)
, QueryBuilders.termQuery("_routing", TenantContextHolder.getTenantContext()));
return search(nativeSearchQuery);
}
The method "setRoute(String route)" of "nativeSearchQuery" either does not work in my case or I didn't understand how it works.
I have successfully tested this implementation with GET and POST requests. Currently I have a problem with elasticsearch overwriting data if the id of the entity from one tenant I want to save is the same id as another entity with a different tenant.

Answer 1

After some trial and error, I found a solution to the overwriting problem and successfully completed and tested my implementation of tenant-based routing. Here is the code:
Product.java
import java.io.Serializable;
import javax.persistence.*;
import org.hibernate.annotations.Cache;
import org.hibernate.annotations.CacheConcurrencyStrategy;
import org.springframework.data.elasticsearch.annotations.Field;
#Entity
#Table(name = "product")
#Cache(usage = CacheConcurrencyStrategy.READ_WRITE)
#org.springframework.data.elasticsearch.annotations.Document(indexName = "product")
#SuppressWarnings("common-java:DuplicatedBlocks")
#org.springframework.data.elasticsearch.annotations.Routing(value = "tenant")
public class Product implements Serializable {
private static final long serialVersionUID = 1L;
#Transient
private Tenant tenant;
#Transient
#Field(name = "elastic_id")
#org.springframework.data.annotation.Id
private String elasticsearchId;
#Id
#GeneratedValue(strategy = GenerationType.SEQUENCE, generator =
"sequenceGenerator")
#SequenceGenerator(name = "sequenceGenerator")
#Column(name = "id")
#Field("postgres_id")
private Long id;
//Getters, Setters and other variables
}
ProductSearchRepository
public interface ProductSearchRepository extends ElasticsearchRepository<Product, Long>, ProductSearchRepositoryInternal {}
interface ProductSearchRepositoryInternal {
Stream<Product> search(String query);
Stream<Product> search(Query query);
void index(Product entity);
Product save(Product entity);
void deleteById(Long id);
}
#Transactional
class ProductSearchRepositoryInternalImpl implements ProductSearchRepositoryInternal {
private final ElasticsearchRestTemplate elasticsearchTemplate;
private final ProductRepository repository;
ProductSearchRepositoryInternalImpl(ElasticsearchRestTemplate elasticsearchTemplate, ProductRepository repository) {
this.elasticsearchTemplate = elasticsearchTemplate;
this.repository = repository;
}
#Override
public Stream<Product> search(String query) {
NativeSearchQuery nativeSearchQuery = new NativeSearchQuery(queryStringQuery(query)
, QueryBuilders.termQuery("_routing", TenantContextHolder.getTenantContext()));
nativeSearchQuery.setMaxResults(30);
return search(nativeSearchQuery);
}
#Override
public Stream<Product> search(Query query) {
return elasticsearchTemplate.search(query, Product.class).map(SearchHit::getContent).stream();
}
#Override
public void index(Product entity) {
entity.setTenant(TenantContextHolder.getTenantContext());
repository.findById(Long.valueOf(entity.getId())).ifPresent(t -> {
entity.setElasticsearchId(entity.getTenant()+String.valueOf(entity.getId()));
elasticsearchTemplate.save(t);
});
}
#Override
public Product save(Product entity) {
entity.setTenant(TenantContextHolder.getTenantContext());
entity.setElasticsearchId(entity.getTenant()+String.valueOf(entity.getId()));
return elasticsearchTemplate.save(entity);
}
#Override
public void deleteById(Long id) {
elasticsearchTemplate.delete(TenantContextHolder.getTenantContext() + String.valueOf(id), Product.class);
}
}