Spring data elastic search with sort not working - elasticsearch

I am using elastic search db and spring data.
Following is my document in which I am searching and that search result should return sorted and pageable list.
#Data
#EqualsAndHashCode(exclude = { "id" })
#Document(indexName = "job", type = "job")
public class JobDocument implements Serializable {
private static final long serialVersionUID = 1L;
#Id
private String id;
#Field(type = FieldType.String, index = FieldIndex.analyzed, store = true)
private String name;
#Field(type = FieldType.String, index = FieldIndex.analyzed, store = true)
private String desc;
#Field(type = FieldType.Date, store = true)
private LocalDateTime dateTime; // java.Time
}
I am searching text in desc field and want to sort it by dateTime field. My search service does following,
BoolQueryBuilder queryBuilder = boolQuery();
queryBuilder.must(QueryBuilders.queryStringQuery("*" + desc + "*").lenient(true).field("desc"));
NativeSearchQueryBuilder searchQuery = new NativeSearchQueryBuilder();
searchQuery.withPageable(pageable);
searchQuery.withSort(SortBuilders.fieldSort("dateTime")
.order(SortOrder.DESC))
searchQuery.withQuery(queryBuilder);
Page<JobDocument> jobs = jobRepo.search(searchQuery.build());
Following is my repo,
public interface JobDAO extends ElasticsearchRepository<JobDocument, String>
{}
Search and pageable is working but Sorting is not working.
Am I missing something?

Their is a sort option available in pageable object.Instead of using the sort and pagination separately, you can add them into a single object PageRequest.
BoolQueryBuilder queryBuilder = boolQuery();
queryBuilder.must(QueryBuilders.queryStringQuery("*" + desc + "*").lenient(true).field("desc"));
NativeSearchQueryBuilder searchQuery = new NativeSearchQueryBuilder();
searchQuery.withPageable(new PageRequest(0, 10, Sort.Direction.DESC, "dateTime"));
searchQuery.withQuery(queryBuilder);

Related

How to search float fields as text in elastic using QueryBuilder

I ve document named plan that correspond plan entity
#Entity
#Table(name = "plan")
#Cache(usage = CacheConcurrencyStrategy.READ_WRITE)
#org.springframework.data.elasticsearch.annotations.Document(indexName = "plan")
public class Plan extends AbstractAuditingEntity implements Serializable {
#Id
#GeneratedValue
#Column(name = "id")
#Field(type = FieldType.Text , fielddata = true)
private UUID id;
#NotNull
#Column(name = "name", nullable = false, unique = true)
private String name;
#Column(name = "description")
private String description;
#Column(name = "current_price")
#Field(type = FieldType.Text , fielddata = true )
private Float currentPrice;
}
Here my method search implementation
public Page<Plan> search(String query, Pageable pageable) {
NativeSearchQuery nativeSearchQuery = new NativeSearchQuery(queryStringQuery("*"+query+"*").defaultOperator(Operator.AND));
nativeSearchQuery.setPageable(pageable);
List<Plan> hits = elasticsearchTemplate
.search(nativeSearchQuery, Plan.class)
.map(SearchHit::getContent)
.stream()
.collect(Collectors.toList());
return new PageImpl<>(hits, pageable, hits.size());
}
Name and Description are searchable but float field isn't .
Marking as FieldType.Float doesn't give expected result .

How to query elasticsearch from spring with LocalDate

Having this SearchQuery:
final SearchQuery searchQuery = new NativeSearchQueryBuilder().withQuery(QueryBuilders.rangeQuery("updateTime").gte(LocalDate.now())).build();
final List<ActivityValue> listOf = elasticsearchTemplate.queryForList(searchQuery, ActivityValue.class);
With Entity ActivityValue:
#Document(indexName = "tracking1", type = "activity")
public class ActivityValue {
#Id
private String id;
#Field(type = FieldType.Date, index = false, store = true, format = DateFormat.custom, pattern = "yyyy-MM-dd")
#JsonFormat(shape = JsonFormat.Shape.STRING, pattern = "yyyy-MM-dd")
private LocalDate updateTime;
#Field(type = FieldType.Object, includeInParent = true)
private Vendor vendor;
#Field(type = FieldType.Object, includeInParent = true)
private QCriteria quality;
public ActivityValue() {
}
//setter and getter
}
If i run the query and try to receive the list i get following exception:
caused by: java.io.IOException: can not write type [class java.time.LocalDate]
The entity is stored before with the actual date as LocalDate.
I'm uncertain what is the best/easiest way to query elasticsearch and to resolve this error. Can anybody help?
final SearchQuery searchQuery = new NativeSearchQueryBuilder().withQuery(QueryBuilders.rangeQuery("updateTime").gte(LocalDate.now().toString())).build();
final List<ActivityValue> listOf = elasticsearchTemplate.queryForList(searchQuery, ActivityValue.class);
I solved this problem by using LocalDate.now().toString() instead of LocalDate.now()

Elastic search case insensitive

I have the following annotation based elastic search configuration, I've set the index not to be analyzed because I don't want these fields to be tokenized:
#Document(indexName = "abc", type = "efg")
public class ResourceElasticSearch {
#Id
private String id;
#Field(type = FieldType.String, index = FieldIndex.not_analyzed)
private String name;
#Field(type = FieldType.String, store = true)
private List<String> tags = new ArrayList<>();
#Field(type = FieldType.String)
private String clientId;
#Field(type = FieldType.String, index = FieldIndex.not_analyzed)
private String virtualPath;
#Field(type = FieldType.Date)
private Date lastModifiedTime;
#Field(type = FieldType.Date)
private Date lastQueryTime;
#Field(type = FieldType.String)
private String modificationId;
#Field(type = FieldType.String)
private String realPath;
#Field(type = FieldType.String)
private String extension;
#Field(type = FieldType.String)
private ResourceType type;
Is it possible by using annotations to make the searches on the name, virtualPath and tags to be case-insensitive?
The search looks like this, search by wildcard is required:
private QueryBuilder getQueryBuilderForSearch(SearchCriteria criteria) {
String virtualPath = criteria.getPath();
return boolQuery()
.must(wildcardQuery("virtualPath", virtualPath))
.must(wildcardQuery("name", criteria.getName()));
}
Not really possible what you want to do and it's not about Spring Data configuration, it's about Elasticsearch itself: you indexed data as not_analyzed and it will stay that way.
Also, if you wanted case insensitive data I suggest indexing with keyword analyzer combined with a lowercase token filter.
I've found something based on Andrei Stefan's suggestion which has a similar result to using the annotations:
#Bean
public Client client() throws IOException {
TransportClient client = new TransportClient();
TransportAddress address = new InetSocketTransportAddress(env.getProperty("elasticsearch.host"), Integer.parseInt(env.getProperty("elasticsearch.port")));
client.addTransportAddress(address);
XContentBuilder settingsBuilder = XContentFactory.jsonBuilder()
.startObject()
.startObject("analysis")
.startObject("analyzer")
.startObject("keyword")
.field("tokenizer", "keyword")
.array("filter", "lowercase")
.endObject()
.endObject()
.endObject()
.endObject();
if (!client.admin().indices().prepareExists("abc").execute().actionGet().isExists()) {
client.admin().indices().prepareCreate("abc").setSettings(settingsBuilder).get();
}
return client;
}
You can add #Setting, which consumes file path, after #Document, settings file should contain json like this:
{"analysis":{"analyzer":{"case_insensitive":{"type":"custom","tokenizer":"whitespace","char_filter":["html_strip"],"filter":["lowercase","asciifolding"]}}}}
and field annotation with analyzer #Field(type = FieldType.Keyword, analyzer = "case_insensitive")

Hibernate Search not returning results

I am building an application with Hibernate Search 4.5.1 and Spring 4.0.5.RELEASE. I am trying to index the following class:
#Entity
#Indexed
#Analyzer(impl= org.apache.lucene.analysis.standard.StandardAnalyzer.class)
#Table(name="SONG")
#XmlRootElement(name="song")
public class Song
{
#Id
#GeneratedValue(strategy = GenerationType.AUTO)
#Column(name = "ID", updatable = false, nullable = false)
private Long id;
#Field(store = Store.YES)
#Column(name="NAME", length=255)
private String name;
#Field(store = Store.YES)
#Column(name="ALBUM", length=255)
private String album;
#Field(store = Store.YES)
#Column(name="ARTIST", length=255)
private String artist;
#NotNull
#Column(name="PATH", length=255)
private String path;
#NotNull
#Column(name="PATH_COVER", length=255)
private String cover;
#NotNull
#Column(name="LAST_VOTE")
private Date date;
#Field(store = Store.YES)
#NotNull
#Column(name="N_VOTES")
private int nvotes;
#NotNull
#Column(name="ACTIVE", nullable=false, columnDefinition="TINYINT(1) default 0")
private boolean active;
#OneToOne(fetch=FetchType.LAZY)
#JoinColumn(name="IMAGE_ID",insertable=true,updatable=true,nullable=false,unique=false)
private Image image;
#IndexedEmbedded
#ManyToOne(fetch = FetchType.LAZY)
#JoinColumn(name = "PLAYLIST_ID", nullable = false)
private PlayList playList;
#OneToMany(mappedBy = "song")
private Set<UserVotes> userServices = new HashSet<UserVotes>();
I am building a junit test case which looks like this:
#RunWith(SpringJUnit4ClassRunner.class)
#ContextConfiguration(locations = {"classpath:jukebox-servlet-test.xml"})
#Transactional
public class SongDaoTest {
#Autowired
public I_PlaceDao placeDao;
#Autowired
public I_PlayListDao playListDao;
#Autowired
public I_SongDao songDao;
#Before
public void prepare() throws Exception
{
Operation operation = sequenceOf(CommonOperations.DISABLE_CONTRAINTS, CommonOperations.DELETE_ALL,CommonOperations.INSERT_SONG_DATA, CommonOperations.ENABLE_CONTRAINTS);
DbSetup dbSetup = new DbSetup(new DriverManagerDestination("jdbc:mysql://localhost:3306/jukebox", "root", "mpsbart"), operation);
dbSetup.launch();
FullTextSession fullTextSession = Search.getFullTextSession(placeDao.getSession());
fullTextSession.createIndexer().startAndWait();
}
#Test
#Rollback(false)
public void searchTest()
{
PlayList playList = playListDao.read(1l);
List<Song> songs = songDao.search(playList, "offspring", 1, 10);
assertEquals(10, songs.size());
}
The search method implementation is:
#SuppressWarnings("unchecked")
public List<Song> search(PlayList playlist, String searchTerm,int page,int limit)
{
FullTextSession fullTextSession = Search.getFullTextSession(getSession());
QueryBuilder queryBuilder = fullTextSession.getSearchFactory().buildQueryBuilder().forEntity(Song.class).get();
BooleanQuery luceneQuery = new BooleanQuery();
luceneQuery.add(queryBuilder.keyword().onFields("name","album","artist").matching("*"+searchTerm+"*").createQuery(), BooleanClause.Occur.MUST);
luceneQuery.add(queryBuilder.phrase().onField("playList.place.id").sentence("\""+playlist.getPlace().getId()+"\"").createQuery(), BooleanClause.Occur.MUST);
luceneQuery.add(queryBuilder.phrase().onField("playList.id").sentence("\""+playlist.getId()+"\"").createQuery(), BooleanClause.Occur.MUST);
// wrap Lucene query in a javax.persistence.Query
FullTextQuery query = fullTextSession.createFullTextQuery(luceneQuery, Song.class);
org.apache.lucene.search.Sort sort = new Sort(new SortField("n_votes",SortField.INT));
query.setSort(sort);
List<Song> songs = query.setFirstResult(page*limit).setMaxResults(limit).list();
return songs;
}
The test result fails, it does not find any matching object. When using luke lucene I can see that there are results, if I try the query generated by hibernate on luke it does return elements. The query generated by hibernate is: +(name:metallica album:metallica artist:metallica) +playList.place.id:"1" +playList.id:"1"
I have also noticed on luke lucene that some index terms have a length up to six characters, for an instance, one song's artist it's "The Offspring" and the terms stored in the index are "the" and "offspr". The first one it's ok, but shouldn't the second term be "offspring". Why is it truncating the name?
In case it helps anybody, I was able to fix it by changing the query to this:
FullTextSession fullTextSession = org.hibernate.search.Search.getFullTextSession(getSession());
QueryBuilder qb = fullTextSession.getSearchFactory().buildQueryBuilder().forEntity(Song.class).get();
if(searchTerm==null || searchTerm.equals(""))
searchTerm="*";
else
searchTerm="*"+searchTerm+"*";
Query luceneQuery1 = qb.bool()
.should(qb.keyword().wildcard().onField("name").matching(searchTerm).createQuery())
.should(qb.keyword().wildcard().onField("album").matching(searchTerm).createQuery())
.should(qb.keyword().wildcard().onField("artist").matching(searchTerm).createQuery()).createQuery();
Query luceneQuery2 = qb.bool()
.must(qb.keyword().wildcard().onField("playList.place.id").matching(playlist.getPlace().getId()).createQuery())
.must(qb.keyword().wildcard().onField("playList.id").matching(playlist.getId()).createQuery())
.createQuery();
BooleanQuery finalLuceneQuery=new BooleanQuery();
finalLuceneQuery.add(luceneQuery1, BooleanClause.Occur.MUST);
finalLuceneQuery.add(luceneQuery2, BooleanClause.Occur.MUST);
FullTextQuery query = fullTextSession.createFullTextQuery(finalLuceneQuery, Song.class);
org.apache.lucene.search.Sort sort = new Sort(new SortField("nvotes",SortField.INT,true));
query.setSort(sort);
List<Song> songs = query.setFirstResult(page*limit).setMaxResults(limit).list();
in case of you have check that field value is null or not null then you must add following line on field where indexing field in class
#Field(index=Index.YES,analyze=Analyze.NO,store=Store.YES,indexNullAs=Field.DEFAULT_NULL_TOKEN)
Search on field
if you want null value then
booleanQuery.must(qb.keyword().onField("callReminder").matching("null").createQuery());
if you don't want null value
booleanQuery.must(qb.keyword().onField("callReminder").matching("null").createQuery()).not();
refrence document:http://docs.jboss.org/hibernate/search/4.1/reference/en-US/html/search-mapping.html#search-mapping-entity

Parent/Child relationships in spring-data-elastic-search

I'm using Spring-Data-Elastic-Search for searching/caching purposes.
I need to execute a query which uses child(TermCache) and parent(ConceptCache) properties
and return instances of child objects(this means i can't use nested objects).
i have the following structure:
#Document(indexName = "termweb" , type = "term")
public class TermCache {
#Id
private String id;
private String name;
private LanguageDTO language;
private String status;
private String definition;
#Field(type = FieldType.String, store = true)
#Parent(type = "concept")
private Long conceptId;
private String displayId;
private Map<Long, String> fields = new HashMap<>();
//todo think about storing it as a collection of nested objects
}
#Document( indexName = "termweb" , type = "concept")
public class ConceptCache implements ConceptDTO{
#Id
private String id;
private String displayId;
private Long dictionaryId;
private String dictionaryName;
private Map<Long, String> fields = new HashMap<>();
}
I need a hint on how to handle this type of tasks; should i use two separate queries or should i somehow fetch properties of a parent or maybe something else?
Agreed, We are lacking on documentation which we will be improving with upcoming release.
If you have any question about spring data elasticsearch stackoverflow probably is not best way to get answer(as we wont be notified for new thread), we have separate google group for question/queries https://groups.google.com/forum/#!forum/spring-data-elasticsearch-devs
Without having any idea about what exactly you are trying to achieve with above entities, i can give you an example of sample parent child entities as below
#Document(indexName = "parent-child", type = "parent-entity")
public class ParentEntity {
#Id
private String id;
#Field(type = FieldType.String, index = FieldIndex.analyzed, store = true)
private String name;
// setter/getter
public ParentEntity() {
}
public ParentEntity(String id, String name) {
this.id = id;
this.name = name;
}
}
#Document(indexName = "parent-child", type = "child-entity")
public class ChildEntity {
#Id
private String id;
#Field(type = FieldType.String, store = true)
#Parent(type = "parent-entity")
private String parentId;
#Field(type = FieldType.String, index = FieldIndex.analyzed, store = true)
private String name;
public ChildEntity() {
}
public ChildEntity(String id, String parentId, String name) {
this.id = id;
this.parentId = parentId;
this.name = name;
}
}
// indexing parent (you can use many other ways to index that includes using repositories)
ParentEntity parent1 = new ParentEntity("parent1", "First Parent");
IndexQuery parentIndex1 = new IndexQuery();
parentIndex1.setId(parent1.getId());
parentIndex1.setObject(parent1);
elasticsearchTemplate.index(parentIndex1);
ParentEntity parent2 = new ParentEntity("parent2", "Second Parent");
IndexQuery parentIndex2 = new IndexQuery();
parentIndex2.setId(parent2.getId());
parentIndex2.setObject(parent2);
elasticsearchTemplate.index(parentIndex2);
// indexing child
ChildEntity child1 = new ChildEntity("child1", parent1.getId(), "First");
IndexQuery childIndex1 = new IndexQuery();
childIndex1.setId(child1.getId());
childIndex1.setObject(child1);
childIndex1.setParentId(child1.getParentId());
elasticsearchTemplate.index(childIndex1);
ChildEntity child2 = new ChildEntity("child2", parent1.getId(), "Second");
IndexQuery childIndex2 = new IndexQuery();
childIndex2.setId(child2.getId());
childIndex2.setObject(child2);
childIndex2.setParentId(child2.getParentId());
elasticsearchTemplate.index(childIndex2);
// searching
there are several available option while searching on Parent/Child entities, that includes has children, has parent and top children queries.
QueryBuilder query = topChildrenQuery("child-entity", QueryBuilders.termQuery("name", child1name.toLowerCase()));
SearchQuery searchQuery = new NativeSearchQueryBuilder().withQuery(query).build();
List<ParentEntity> parents = elasticsearchTemplate.queryForList(searchQuery, ParentEntity.class);
Hope this small example will give you basic understanding how to use parent child. have a look at ParentChildTests for more.
If you still have more question please feel free to contact us.
You should simply use hasparent query of filter : http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl-has-parent-filter.html#query-dsl-has-parent-filter
This will make a request on parent field and result in children documents of the matching parents documents. You can then use a filter on the returned child document :)

Resources