I'm using the newer 8.1 Java API for Elastic in Kotlin, and getting behavior that isn't what I would expect, nor what I get when using the manual REST API. Here's my code:
val boolQuery = BoolQuery.of { boolBuilder -> boolBuilder
.must { mustBuilder ->
query.forEachIndexed { index, char ->
val prefix = if (index > 0) {
"* "
} else { "" }
val queryString = QueryStringQuery.of { queryBuilder -> queryBuilder
.query("${prefix}${char}*")
}
mustBuilder.queryString(queryString)
}
mustBuilder
}
}
In practice this only appears to hit on the last string query. With an input of query="rp" I would expect the following request to be made:
{
"query": {
"bool" : {
"must" : [
{ "query_string" : { "query" : "r*" } },
{ "query_string" : { "query" : "* p*" } }
]
}
}
}
When running that exact request, it does behave as I expect/intend. I can't tell how to pull out the request the Java API is sending over without monitoring the traffic, but if I understand the pattern then I would think these are isomorphic. I've also confirmed that the Kotlin code is calling the Java API as intended based on my input.
What am I doing wrong here?
Apparently asking the question was enough to figure it out – each subquery needs its own builder. Here's the modified code now working as expected:
val boolQuery = BoolQuery.of { boolBuilder ->
query.forEachIndexed { index, char ->
val prefix = if (index > 0) {
"* "
} else { "" }
boolBuilder.must { mustBuilder ->
val queryString = QueryStringQuery.of { queryBuilder -> queryBuilder
.query("${prefix}${char}*")
}
mustBuilder.queryString(queryString)
}
}
boolBuilder
}
Related
I am using org.springframework.data:spring-data-elasticsearch:4.1.0 with elasticsearch 7.8.1.
I have a requirement where I need to lookup partial search for multiple attributes. I have implemented wildcard bool queries which is working fine except it is not able to do lookup for empty spaces.
Here is my actual query:
GET /maintenance_logs/_search
{
"query": {
"bool": {
"must": [
{
"term": {
"vinNumber.keyword": "DH34ASD7SDFF84742"
}
},
{
"term": {
"organizationId": 1
}
}
],
"minimum_should_match": 1,
"should": [
{
"wildcard": {
"dtcCode": {
"value": "*Cabin*"
}
}
},
{
"wildcard": {
"subSystem": {
"value": "*Cabin*"
}
}
},
{
"wildcard": {
"maintenanceActivity": {
"value": "*Cabin*"
}
}
},
{
"wildcard": {
"description": {
"value": "*Cabin*"
}
}
}
]
}
}
}
Here is my SearchRequest:
public static SearchRequest buildSearchRequest(final String indexName,
final SearchRequestDTO dto,
final String vinNumber,
final Integer organizationId, Pageable pageable) {
try {
final int page = pageable.getPageNumber();
final int size = pageable.getPageSize();
final int from = page <= 0 ? 0 : pageable.getPageSize();
SearchRequest searchRequest = new SearchRequest(indexName);
SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();
BoolQueryBuilder boolQueryBuilder = QueryBuilders.boolQuery();
final QueryBuilder vinQuery = QueryBuilders.termQuery("vinNumber.keyword", vinNumber);
final QueryBuilder orgIdQuery = QueryBuilders.termQuery("organizationId", organizationId);
boolQueryBuilder.must(vinQuery);
boolQueryBuilder.must(orgIdQuery);
boolQueryBuilder.minimumShouldMatch(1);
boolQueryBuilder.should(QueryBuilders.wildcardQuery("dtcCode", "*" + dto.getSearchTerm() + "*"));
boolQueryBuilder.should(QueryBuilders.wildcardQuery("subSystem", "*" + dto.getSearchTerm() + "*"));
boolQueryBuilder.should(QueryBuilders.wildcardQuery("maintenanceActivity", "*" + dto.getSearchTerm() + "*"));
boolQueryBuilder.should(QueryBuilders.wildcardQuery("description", "*" + dto.getSearchTerm() + "*"));
searchSourceBuilder.query(boolQueryBuilder);
searchSourceBuilder
.from(from)
.size(size)
.sort(SortBuilders.fieldSort("statsDate")
.order(SortOrder.DESC));
searchRequest.source(searchSourceBuilder);
return searchRequest;
} catch (final Exception e) {
e.printStackTrace();
return null;
}
}
This works fine except I am unable to search for strings like "Cabin pressure".
If you want to be able to search a multi-token value like "Cabin pressure" to be findable with a wildcard query using "Cabin pressure" you need to define it in the mapping as being of type keyword.
The wildcard search searches for single terms that match the wildcard expression, and "Cabin pressure" by default is split into two terms, "Cabin" and "pressure".
In Spring Data Elasticsearch the way to do this is to use #Field(type = FieldType.Keyword, but you'd either need to delete and recreate the index to have the new mapping applied, or you need to create a new index and reindex the existing one into the new. That's because index mappings cannot be updated and in your existing index the type is by default defined as text.
And, if you store "Cabin pressure" as one term - type keyword - don't forget that this will be a different thing than "cabin pressure". Keywords are not normalized, so that upper and lower case differences matter.
Assume that there is a query such as:
query test1 {
students {
id
name
address_pin
address_city
yearAdmitted
laptopOS
phoneOS
}
}
I can access this entire query text at runtime in graphql-java.
Is there anyway to get the query-name 'students'?
Tried this and it works for basic queries, not sure if this is the correct way to do so.
private String getQueryName(String input) {
Document doc = new Parser().parseDocument(input);
List<OperationDefinition> definitionList = doc.getDefinitionsOfType(OperationDefinition.class);
for (OperationDefinition definition : definitionList) {
if (definition != null && definition.getOperation().toString().equalsIgnoreCase("QUERY")) {
Field field = definition.getSelectionSet().getSelectionsOfType(Field.class).stream().findFirst().orElse(null);
if (field != null) {
return field.getName();
}
}
}
return null;
}
I have an index with a field chunks with keyword type which is just an a list of keyword. When I search through I do something like
{
"query": {
"bool": {
"filter": [
{
"term": {
"chunks": "chunk1"
}
},
{
"term": {
"chunks": "chunk2"
}
}
]
}
}
}
So I can retrieve all documents where there are both "chunk1" and "chunk2" inside the chunks field. However what if I care about the order? My solution is too have a script like
String[] chunks = doc['chunks'];
int c = 0;
String chunk = params.chunks[0];
for (int i = 0; i < chunks.length; ++i) {
if (chunk == chunks[i]) {
if (++c == params.chunks.length) {
return true;
}
chunk = params.chunks[c];
}
}
return false;
where params.chunks is something like ["chunk1", "chunk2"]. The problem is the doc['chunks'] is unordered, while params._source isn't allowed from _search context.
It should be possible somehow, because Elastic itself has similar functionality for multitext search, so I can emulate the same field structure.
I want to use Elasticsearch's aggregation to do OLAP data analysis.
What I want to do is nested a scriptedMetric aggregation into a term aggregation,as below (it's correct)
{
"from": 0,
"size": 0,
"query":{
"bool":{
"must":[
{
"match":{
"poi_id":1
}
}
]
}
},
"aggregations": {
"poi_id": {
"terms": {
"script": {
"inline": "doc['poi_id'].value + 1"
}
},
"aggregations": {
"price": {
"sum": {
"field": "price"
}
}
}
}
}
}
But I didn't find how to do this in Elasticsearch's java api.
I've tried it this way:
SearchResponse response = client.prepareSearch("poi")
.setSearchType(SearchType.DFS_QUERY_THEN_FETCH)
.setFetchSource(new String[]{"poi_id","poi_name"}, null)
.setQuery(QueryBuilders.termQuery("poi_id", 1))
.addAggregation(AggregationBuilders.terms("poi_id").subAggregation((AggregationBuilders.scriptedMetric("poi_id").mapScript(new Script("doc['poi_id'].value + 1")))))
.execute()
.actionGet();
But got an error
Caused by: NotSerializableExceptionWrapper[: value source config is invalid; must have either a field context or a script or marked as unwrapped]; nested: IllegalStateException[value source config is invalid; must have either a field context or a script or marked as unwrapped];
I've searched a lot, but can't find a demo.
Any help would be appreciated.
Thanks!
#Override
public Map<String, Object> sumPriceAggregation(String field, int page, int size) {
if (StringUtils.isEmpty(field)) {
field = "brandName";
}
NativeSearchQueryBuilder queryBuilder = new NativeSearchQueryBuilder();
PageRequest pageRequest = PageRequest.of(page, size);
queryBuilder.withPageable(pageRequest);
queryBuilder.withSourceFilter(new FetchSourceFilter(new String[] {""}, null));
String termStr = field.toUpperCase();
TermsAggregationBuilder termsAggregationBuilder = AggregationBuilders.terms(termStr)
.field(field)
.subAggregation(AggregationBuilders.sum("totalPrice").field("price")); //be aware this is subAggregation
NativeSearchQueryBuilder nativeSearchQueryBuilder = new NativeSearchQueryBuilder();
nativeSearchQueryBuilder.addAggregation(termsAggregationBuilder);
AggregatedPage<GamingLaptop> aggregatedPage = elasticsearchRestTemplate.queryForPage(
nativeSearchQueryBuilder.build(), GamingLaptop.class);
Aggregations aggregations = aggregatedPage.getAggregations();
ParsedStringTerms stringTerms = aggregations.get(termStr);
List<? extends Terms.Bucket> buckets = stringTerms.getBuckets();
HashMap<String, Object> map = new HashMap<>();
buckets.parallelStream().forEach(bucket -> {
String key = bucket.getKeyAsString();
long docCount = bucket.getDocCount();
map.put(key, docCount);
ParsedSum sum = (ParsedSum) bucket.getAggregations().asMap().get("totalPrice"); //make sure you get the aggregation here
map.putIfAbsent("sum", sum.getValue());
});
return map;
}
'value source config is invalid; must have either a field context or a script or marked as unwrapped' i encountered this error as well, please read the comments in the codes, which is my solution. Either ParsedStringTerms or TermsAggregationBuilder need to be retrieved.
I have tested the elastic search highlight field function and it was working fine . I used elastic search 2.4.4 and spring-data-elasticsearch-2.0.0.RELEASE
The sample code is in the below post
How to provide highlighting with Spring data elasticsearch
I have recently upgraded to elastic search to 5.5.0 and spring-data-elasticsearch-3.0.0.M4
when I test the same code , highlight does not happen
Below is the sample code
SearchQuery searchQuery = new NativeSearchQueryBuilder().withIndices("occindex")
.withPageable(new PageRequest(0, mySpecification.getNoOfRecords()))
.withQuery(QueryBuilders.multiMatchQuery(
searchText.toLowerCase()).field("transformedTitle", 10.0f).
minimumShouldMatch("50%").fuzziness(Fuzziness.ONE).prefixLength(3)
.field("transformedDesription").type(MultiMatchQueryBuilder.Type.BEST_FIELDS))
.withHighlightFields(
new HighlightBuilder.Field("transformedTitle").preTags("<span style='background-color: #FFFF00'>")
.postTags("</span>"),
new HighlightBuilder.Field("transformedDesription").fragmentSize(250).numOfFragments(3)
.preTags("<span style='background-color: #FFFF00'>").postTags("</span>"))
.build();
Page<MyResultRecord> sampleEntities = elasticsearchTemplate.queryForPage(searchQuery,
MyResultRecord.class, new SearchResultMapper() {
#Override
public <T> AggregatedPage<T> mapResults(SearchResponse response, Class<T> clazz, Pageable pageable) {
List<MyResultRecord> chunk = new ArrayList<MyResultRecord>();
for (SearchHit searchHit : response.getHits()) {
if (response.getHits().getHits().length <= 0) {
return null;
}
MyResultRecord myResultRecord = new MyResultRecord();
myResultRecord.setRecordId(searchHit.getId());
Map<String, Object> source = searchHit.getSource();
myResultRecord.setRisk((String) source.get("actualRisk"));
String highlightedTitle = null;
System.out.println( " Check the highlighted fileds " + searchHit.getHighlightFields());
System.out.println( " Is this null ?? " + searchHit.getHighlightFields().get("transformedTitle"));
if (searchHit.getHighlightFields().get("transformedTitle") != null)
highlightedTitle = searchHit.getHighlightFields().get("transformedTitle").fragments()[0]
.toString();
else
highlightedTitle = (String) source.get("transformedTitle");
myResultRecord.setHighlightedTitle(highlightedTitle);
myResultRecord.setScore(searchHit.getScore());
chunk.add(myResultRecord);
}
if (chunk.size() > 0) {
return new AggregatedPageImpl(chunk);
}
return null;
}
});
Is there any code change needed in order to highlight fields in elastic search 5.5.0 ?
When I print the query in the elastic search log , I find that only one highlight field is passed to Elastic Search
{
"from":0,
"size":2,
"query":{
"multi_match":{
"query":" My Query String",
"fields":[
"transformedDesription^1.0",
"transformedTitle^1.0"
],
"type":"best_fields",
"operator":"OR",
"slop":0,
"prefix_length":0,
"max_expansions":50,
"lenient":false,
"zero_terms_query":"NONE",
"boost":1.0
}
},
"highlight":{
"fields":{
"transformedDesription":{
"pre_tags":[
"<bold>"
],
"post_tags":[
"</bold>"
]
}
}
}
}
When I debugged , I observed that NativeSearchQuery has two highlighted fields , but the final query sent to Elastic Search has only request for one highlight field.
Made it work by changing the below code in org.springframework.data.elasticsearch.core.ElasticsearchTemplate.doSearch
Existing Code
if (searchQuery.getHighlightFields() != null) {
for (HighlightBuilder.Field highlightField : searchQuery.getHighlightFields()) {
searchRequest.highlighter(new HighlightBuilder().field(highlightField));
}
}
Modified Code
if (searchQuery.getHighlightFields() != null) {
HighlightBuilder myBuilder = new HighlightBuilder();
for (HighlightBuilder.Field highlightField : searchQuery.getHighlightFields()) {
myBuilder.field(highlightField);
}
searchRequest.highlighter(myBuilder);
}