Elasticsearch script parameter containsAll with value from document - elasticsearch

I'm trying to use an elasticsearch script query using the Java SDK to determine if a collection of IDs passed as a parameter contains all of the values in the indexed document.
Here is a simple example, that does not work, i'm expecting this to match the a document, but it doesn't.
The document attributes field is an array with a single item [2]
BoolQueryBuilder qb = boolQuery();
Map<String, Object> params = new HashMap<String, Object>();
params.put( "list", Arrays.asList( 1L, 2L, 3L ) );
String expr = "params.list.containsAll(doc['attributes'])";
Script script = new Script( ScriptType.INLINE, "painless", expr, Collections.emptyMap(), params );
qb.should( scriptQuery( script ) );
However, i've tried these various hard coded alternatives that do work
Hard coded .contains(2) works as expected
BoolQueryBuilder qb = boolQuery();
Map<String, Object> params = new HashMap<String, Object>();
params.put( "list", Arrays.asList( 2L ) );
String expr = "params.list.contains(2)";
Script script = new Script( ScriptType.INLINE, "painless", expr, Collections.emptyMap(), params );
qb.should( scriptQuery( script ) );
Hard coded .containsAll([2]) works as expected
BoolQueryBuilder qb = boolQuery();
Map<String, Object> params = new HashMap<String, Object>();
params.put( "list", Arrays.asList( 2L ) );
String expr = "params.list.containsAll([2])";
Script script = new Script( ScriptType.INLINE, "painless", expr, Collections.emptyMap(), params );
qb.should( scriptQuery( script ) );

Gotta cast the params.list elements before calling containsAll:
{
"query": {
"bool": {
"must": [
{
"script": {
"script": {
"inline": """
params.list.stream()
.map(num -> (long) num)
.collect(Collectors.toList())
.containsAll(
doc['attributes'].stream()
.collect(Collectors.toList()))""",
"params": {
"list": [
2
]
}
}
}
}
]
}
}
}
assuming your mapping truly defined the atributes as such:
{
"mappings": {
"properties": {
"attributes": {
"type": "long"
}
}
}
}

Related

Elasticsearch Java - use Search Template to query

I have below code working fine in my java service.
Query searchQuery = new StringQuery(
"{\"bool\":{\"must\":[{\"match\":{\"id\":\"" + id + "\"}}]}}");
SearchHits<Instance> instanceSearchHits = elasticsearchOperations.search(searchQuery, Instance.class, IndexCoordinates.of("test"));
log.info("hits :: " + instanceSearchHits.getSearchHits().size());
Now, I want to save this query as a template in elastic and just pass params and search template from java service to elastic to execute the query.
Search Template added in Elastic
PUT _scripts/search-template-1
{
"script": {
"lang": "mustache",
"source": {
"query": {
"bool": {
"must": [
{
"term": {
"id": "{{id}}"
}
}
]
}
}
},
"params": {
"id": "id to search"
}
}
}
Call this template
GET test/_search/template
{
"id": "search-template-1",
"params": {
"id": "f52c2c62-e921-4410-847f-25ea0f3eeb40"
}
}
But unfortunately not able to find API reference for the same to call this search template from JAVA (spring-data-elasticsearch)
As mentioned by val this can be used to call the search template query from java
SearchTemplateRequest request = new SearchTemplateRequest();
request.setRequest(new SearchRequest("posts"));
request.setScriptType(ScriptType.STORED);
request.setScript("title_search");
Map<String, Object> params = new HashMap<>();
params.put("field", "title");
params.put("value", "elasticsearch");
params.put("size", 5);
request.setScriptParams(params);
SearchTemplateResponse response = client.searchTemplate(request, RequestOptions.DEFAULT);
SearchResponse searchResponse = response.getResponse();
SearchHits searchHits = searchResponse.getHits();
log.info("hits :: " + searchHits.getMaxScore());
searchHits.forEach(searchHit -> {
log.info("this is the response, " + searchHit.getSourceAsString());
});
This is currently not yet possible. There is an issue for this in Spring Data Elasticsearch.

Is an nGram fuzzy search possible?

I'm trying to get an nGram filter to work with a fuzzy search, but it won't. Specifically, I'm trying to get "rugh" to match on "rough".
I don't know whether it's just not possible, or it is possible but I've defined the mapping wrong, or the mapping is fine but my search isn't defined correctly.
Mapping:
{
settings = new
{
index = new
{
number_of_shards = 1,
number_of_replicas = 1,
analysis = new
{
filter = new
{
edge_ngram_filter = new
{
type = "nGram",
min_gram = 3,
max_gram = 8
}
}, // filter
analyzer = new
{
analyzer_ngram = new
{
type = "custom",
tokenizer = "standard",
filter = new string[]
{
"lowercase",
"edge_ngram_filter"
}
}
} // analyzer
} // analysis
} // index
}, // settings
mappings = new
{
j_cv = new
{
properties = new
{
Text = new
{
type = "text",
include_in_all = false,
analyzer = "analyzer_ngram",
search_analyzer = "standard"
}
}
} // j_cv
} // mappings
}
Document:
{
Id = Guid.NewGuid(),
Name = "Jimmy Riddle",
Keyword = new List<string>(new string[] { "Hunting", "High", "Hotel", "California" }),
Text = "Rough Justice was a program on BBC some years ago. It was quite interesting. Will this match?"
}
Search:
{
query = new
{
query_string = new
{
fields = new string[] { "Text" },
fuzziness = "3",
query = "rugh"
}
}
}
Incidentally, "ugh" does match which is what you'd expect.
Thanks for any help you can give,
Adam.
The same analyzer should usually be applied at index and search time, so search_analyzer=standard is wrong, it should be working if you remove it.
https://www.elastic.co/guide/en/elasticsearch/reference/current/search-analyzer.html
Edit:
You forgot the fuzzy operator "~" in your query, if you add it to "rugh" it will work!

Conditional query with spring mongotemplate

I want to use conditional query.
Here is my query
db.projects.aggregate([
{
"$group": {
"_id": "$iecode",
"treatmentArms": { "$first": "$evaluationDTOList" }
}
},
{ "$unwind": "$treatmentArms" },
{
"$group": {
"_id": null,
"Package": {
"$sum": {
"$cond": [
{ "$eq": [ "$treatmentArms.mechanismOrPkg", "Package" ] },
1, 0
]
}
},
"Constraint-relaxing mechanisms": {
"$sum": {
"$cond": [
{
"$and": [
{ "$eq": [ "$treatmentArms.mechanismOrPkg", "Mechanism" ] },
{ "$eq": [ "$treatmentArms.mechanismTested1", "Constraint-relaxing mechanisms" ] }
]
},
1,
0 ]
}
},
"Delivery mechanisms": {
"$sum": {
"$cond": [
{
"$and": [
{ "$eq": [ "$treatmentArms.mechanismOrPkg", "Mechanism" ] },
{ "$eq": [ "$treatmentArms.mechanismTested1", "Delivery mechanisms" ] }
]
},
1,
0 ]
}
},
"Other": {
"$sum": {
"$cond": [
{
"$and": [
{ "$eq": [ "$treatmentArms.mechanismOrPkg", "Mechanism" ] },
{ "$eq": [ "$treatmentArms.mechanismTested1", "Other" ] }
]
},
1,
0 ]
}
}
}
}
])
Here is my java code
DBObject groupByIECode = new BasicDBObject("$group",
new BasicDBObject("_id", new BasicDBObject("iecode","$iecode")).append("treatmentArms",new BasicDBObject("$first","$evaluationDTOList")));
System.out.println("groupByIECode: "+groupByIECode.toString());
DBObject unwind = new BasicDBObject("$unwind","$treatmentArms");
System.out.println("unwind: "+unwind.toString());
DBObject finalCalculation = new BasicDBObject("$group",new BasicDBObject("_id",null))
.append(
"Package", new BasicDBObject(
"$sum", new BasicDBObject(
"$cond", new Object[]{
new BasicDBObject(
"$eq", new Object[]{ "$treatmentArms.mechanismOrPkg", "Package"}
),
1,
0
}
)
)
);
System.out.println("finalCalculation: "+finalCalculation);
final AggregationOutput output = projects.aggregate(match,groupByIECode,unwind,finalCalculation);
It gives me MongoException$DuplicateKey
Later I found out that $cond operator is not supported in spring mongotemplate. So how do I implememt this conditional query with spring mongotemplate.
This link has some explanation but it do not shows full implementation
From the documentation, a canonical example for using the Spring Data MongoDB support for the MongoDB Aggregation Framework looks as follows:
import static org.springframework.data.mongodb.core.aggregation.Aggregation.*;
Aggregation agg = newAggregation(
pipelineOP1(),
pipelineOP2(),
pipelineOPn()
);
AggregationResults<OutputType> results = mongoTemplate.aggregate(agg,
"INPUT_COLLECTION_NAME", OutputType.class);
List<OutputType> mappedResult = results.getMappedResults();
Note that if you provide an input class as the first parameter to the
newAggregation method the MongoTemplate will derive the name of the
input collection from this class. Otherwise if you don’t specify
an input class you must provide the name of the input collection
explicitly. If an input-class and an input-collection is provided the
latter takes precedence.
For your query, create a workaround that implements the AggregationOperation interface to take in a DBObject that represents a single group operation in an aggregation pipeline with the $cond operator:
public class GroupAggregationOperation implements AggregationOperation {
private DBObject operation;
public GroupAggregationOperation (DBObject operation) {
this.operation = operation;
}
#Override
public DBObject toDBObject(AggregationOperationContext context) {
return context.getMappedObject(operation);
}
}
Then implement the $group operation as a DBObject in the aggregation pipeline that is the same as the one you have:
DBObject operation = (DBObject) new BasicDBObject("$group", new BasicDBObject("_id", null))
.append(
"Package", new BasicDBObject(
"$sum", new BasicDBObject(
"$cond", new Object[]{
new BasicDBObject(
"$eq", new Object[]{ "$treatmentArms.mechanismOrPkg", "Package"}
),
1,
0
}
)
)
);
which you can then use as:
import static org.springframework.data.mongodb.core.aggregation.Aggregation.*;
GroupAggregationOperation groupOp = new GroupAggregationOperation(operation);
Aggregation agg = newAggregation(
group("iecode").first("treatmentArms").as("treatmentArms"),
unwind("treatmentArms"),
groupOp
);
AggregationResults<Entity> results = mongoTemplate.aggregate(agg, Entity.class);
List<Entity> entities = results.getMappedResults();

How to provide highlighting with Spring data elasticsearch

it seems that SpringData ES don't provide classes to fetch highlights returned by ES. Spring Data can return Lists of Objects but the highlights sections in the Json returned by ES is in a separated part that is not handled by the "ElasticSearchTemplate" class.
Code example :-
QueryBuilder query = QueryBuilders.matchQuery("name","tom");
SearchQuery searchQuery =new NativeSearchQueryBuilder().withQuery(query).
with HighlightFields(new Field("name")).build();
List<ESDocument> publications = elasticsearchTemplate.queryForList
(searchQuery, ESDocument.class);
I might be wrong, but I can't figure out to do only with SpringDataES. Someone can post an example of how we can get highlights with Spring Data ES ?
Thanks in advance !
From the test cases in spring data elasticsearch I've found solution to this :
This can be helpful.
#Test
public void shouldReturnHighlightedFieldsForGivenQueryAndFields() {
//given
String documentId = randomNumeric(5);
String actualMessage = "some test message";
String highlightedMessage = "some <em>test</em> message";
SampleEntity sampleEntity = SampleEntity.builder().id(documentId)
.message(actualMessage)
.version(System.currentTimeMillis()).build();
IndexQuery indexQuery = getIndexQuery(sampleEntity);
elasticsearchTemplate.index(indexQuery);
elasticsearchTemplate.refresh(SampleEntity.class);
SearchQuery searchQuery = new NativeSearchQueryBuilder()
.withQuery(termQuery("message", "test"))
.withHighlightFields(new HighlightBuilder.Field("message"))
.build();
Page<SampleEntity> sampleEntities = elasticsearchTemplate.queryForPage(searchQuery, SampleEntity.class, new SearchResultMapper() {
#Override
public <T> Page<T> mapResults(SearchResponse response, Class<T> clazz, Pageable pageable) {
List<SampleEntity> chunk = new ArrayList<SampleEntity>();
for (SearchHit searchHit : response.getHits()) {
if (response.getHits().getHits().length <= 0) {
return null;
}
SampleEntity user = new SampleEntity();
user.setId(searchHit.getId());
user.setMessage((String) searchHit.getSource().get("message"));
user.setHighlightedMessage(searchHit.getHighlightFields().get("message").fragments()[0].toString());
chunk.add(user);
}
if (chunk.size() > 0) {
return new PageImpl<T>((List<T>) chunk);
}
return null;
}
});
assertThat(sampleEntities.getContent().get(0).getHighlightedMessage(), is(highlightedMessage));
}
Spring Data Elasticsearch 4.0 now has the SearchPage result type, which makes things a little easier if we need to return highlighted results:
This is a working sample:
String query = "(id:123 OR id:456) AND (database:UCLF) AND (services:(sealer?), services:electronic*)"
NativeSearchQuery searchQuery = new NativeSearchQueryBuilder()
.withPageable(pageable)
.withQuery(queryStringQuery(query))
.withSourceFilter(sourceFilter)
.withHighlightFields(new HighlightBuilder.Field("goodsAndServices"))
.build();
SearchHits<Trademark> searchHits = template.search(searchQuery, Trademark.class, IndexCoordinates.of("trademark"));
SearchPage<Trademark> page = SearchHitSupport.searchPageFor(searchHits, searchQuery.getPageable());
return (Page<Trademark>) SearchHitSupport.unwrapSearchHits(page);
And this would be the response from Page object in json:
{
"content": [
{
"id": "123",
"score": 12.10748,
"sortValues": [],
"content": {
"_id": "1P0XzXIBdRyrchmFplEA",
"trademarkIdentifier": "abc234",
"goodsAndServices": null,
"language": "EN",
"niceClass": "2",
"sequence": null,
"database": "UCLF",
"taggedResult": null
},
"highlightFields": {
"goodsAndServices": [
"VARNISHES, <em>SEALERS</em>, AND NATURAL WOOD FINISHES"
]
}
}
],
"pageable": {
"sort": {
"unsorted": true,
"sorted": false,
"empty": true
},
"offset": 0,
"pageNumber": 0,
"pageSize": 20,
"unpaged": false,
"paged": true
},
"searchHits": {
"totalHits": 1,
"totalHitsRelation": "EQUAL_TO",
"maxScore": 12.10748,
"scrollId": null,
"searchHits": [
{
"id": "123",
"score": 12.10748,
"sortValues": [],
"content": {
"_id": "1P0XzXIBdRyrchmFplEA",
"trademarkIdentifier": "abc234",
"goodsAndServices": null,
"language": "EN",
"niceClass": "2",
"sequence": null,
"database": "UCLF",
"taggedResult": null
},
"highlightFields": {
"goodsAndServices": [
"VARNISHES, <em>SEALERS</em>, AND NATURAL WOOD FINISHES"
]
}
}
],
"aggregations": null,
"empty": false
},
"totalPages": 1,
"totalElements": 1,
"size": 20,
"number": 0,
"numberOfElements": 1,
"last": true,
"first": true,
"sort": {
"unsorted": true,
"sorted": false,
"empty": true
},
"empty": false
}
Actually, you could do the following, with a custom ResultExtractor:
QueryBuilder query = QueryBuilders.matchQuery("name", "tom");
SearchQuery searchQuery = new NativeSearchQueryBuilder()
.withQuery(query)
.withHighlightFields(new Field("name")).build();
return elasticsearchTemplate.query(searchQuery.build(), new CustomResultExtractor());
And then
public class CustomResultExtractor implements ResultsExtractor<List<MyClass>> {
private final DefaultEntityMapper defaultEntityMapper;
public CustomResultExtractor() {
defaultEntityMapper = new DefaultEntityMapper();
}
#Override
public List<MyClass> extract(SearchResponse response) {
return StreamSupport.stream(response.getHits().spliterator(), false)
.map(this::searchHitToMyClass)
.collect(Collectors.toList());
}
private MyClass searchHitToMyClass(SearchHit searchHit) {
MyElasticSearchObject myObject;
try {
myObject = defaultEntityMapper.mapToObject(searchHit.getSourceAsString(), MyElasticSearchObject.class);
} catch (IOException e) {
throw new ElasticsearchException("failed to map source [ " + searchHit.getSourceAsString() + "] to class " + MyElasticSearchObject.class.getSimpleName(), e);
}
List<String> highlights = searchHit.getHighlightFields().values()
.stream()
.flatMap(highlightField -> Arrays.stream(highlightField.fragments()))
.map(Text::string)
.collect(Collectors.toList());
// Or whatever you want to do with the highlights
return new MyClass(myObject, highlights);
}}
Note that I used a list but you could use any other iterable data structure. Also, you could do something else with the highlights. Here I'm simply listing them.
https://stackoverflow.com/a/37163711/6643675
The first answer does works,but I found some pageable problems with its returned result,which display with the wrong total elements and toalpages.Arter I checkout the DefaultResultMapper implementation, the returned statement shoud be return new AggregatedPageImpl((List<T>) chunk, pageable, totalHits, response.getAggregations(), response.getScrollId(), maxScore);,and then it works with paging.wish i could help you guys~

ElasticSearch NEST library to retrieve certain fields

Trying to achieve following,
Retrieve articles that match given id and genre
Retrieve selected fields for matching records
I have tried this with Sense(chrome plugin),
POST /d3acampaign/article/_search
{
"fields": ["title","genre"] ,
"query": {
"filtered": {
"filter": {
"bool": {
"must": [{
"term": {
"id": "6"
}
},
{
"term": {
"genre": "metal"
}
}]
}
}
}
}
}
For C# code i am trying to build the query using following construct,
FilterContainer fc = null;
TermFilter title = new TermFilter()
{
Field = "id",
Value = "6",
};
TermFilter genre = new TermFilter()
{
Field = "genre",
Value = "metal",
};
fc = title & genre;
QueryContainer qc = new FilteredQuery() { Filter = fc };
var searchRequest = new SearchRequest
{
SearchType = Elasticsearch.Net.SearchType.QueryAndFetch,
Query = qc,
Indices = new IndexNameMarker[] {"journal"},
Types = new TypeNameMarker[] { "article" },
};
var r = client.SearchAsync<Article>(searchRequest);
var l = (List<Article>) r.Result.Documents;
I am able to run this query and get matching records, but i amnt sure how to specify selected fields to retrieve. Let me know what can be changed in C# code to specify necessary fields.
Thanks in advance.
Based on this answer you can modify your request object as follow:
var searchRequest = new SearchRequest
{
...
Fields = new List<PropertyPathMarker>
{
Property.Path<Article>(p => p.YourField)
}
};
or if you will decide to use source filtering:
var searchRequest = new SearchRequest
{
...
Source = new SourceFilter
{
Include = new []
{
Property.Path<Article>(p => p.YourField)
}
}
};
Hope it helps.

Resources