NEST / Elastic Search Auto-Completion 'Field [suggest] is not a completion suggest field' - elasticsearch

In between projects, started to play around with Elastic Search and NEST in preparation for some auto-completion features coming up.
I followed examples here and in the various docs ... and it seems like what I have should work. There must be some subtle difference with what i am doing.
This is the object that represents my document.
[ElasticType(IdProperty="KeyInstn")]
public class Instn
{
[ElasticProperty(Name = "keyinstn", Index = FieldIndexOption.Analyzed, Type = FieldType.Integer)]
public int KeyInstn { get; set; }
public string Type { get; set; }
[ElasticProperty(Name = "name", Index = FieldIndexOption.Analyzed, Type = FieldType.String)]
public string Name { get; set; }
[ElasticProperty(Name = "ticker", Index = FieldIndexOption.Analyzed, Type = FieldType.String)]
public string Ticker { get; set; }
[ElasticProperty(Name = "street", Index = FieldIndexOption.Analyzed, Type = FieldType.String)]
public string Street { get; set; }
[ElasticProperty(Name = "city", Index = FieldIndexOption.Analyzed, Type = FieldType.String)]
public string City { get; set; }
[ElasticProperty(Name = "state", Index = FieldIndexOption.Analyzed, Type = FieldType.String)]
public string State { get; set; }
[ElasticProperty(Name = "zip", Index = FieldIndexOption.Analyzed, Type = FieldType.String)]
public string Zip { get; set; }
[ElasticProperty(Name = "country", Index = FieldIndexOption.Analyzed, Type = FieldType.String)]
public string Country { get; set; }
[ElasticProperty(Name = "suggest", Type = FieldType.Completion)]
public CompletionField Suggest { get; set; }
}
This is the code that I use to create my index:
var index = client.CreateIndex("snl", c => c
.NumberOfReplicas(1)
.NumberOfShards(5)
.Settings(s => s
.Add("merge.policy.merge_factor", "10")
.Add("search.slowlog.threshold.fetch.warn", "1s")
)
.AddMapping<Instn>(m => m.MapFromAttributes()
.Properties(props => props
.Completion(s => s
.Name(p => p.Suggest)
.IndexAnalyzer("simple")
.SearchAnalyzer("simple")
.MaxInputLength(20)
.Payloads()
.PreservePositionIncrements()
.PreserveSeparators()
)
)
));
I am populating my documents using IndexMany in batches of 1000 documents.
I then perform my query:
var suggestResponse = this._client.Suggest<Instn>(s => s
.Index("snl")
.Completion("suggest", c => c
.Text(this.AutoCompleteText.Text)
.OnField("suggest")));
The error I receive is:
Field [suggest] is not a completion suggest field
Using SENSE, I pulls the mappings and notice that "suggest" doesn't have a type. The examples I followed indicate that the type should be 'completion.:'
Elasticsearch - Autocomplete with NEST
{
"snl": {
"mappings": {
"instn": {
"properties": {
"city": {
"type": "string"
},
"country": {
"type": "string"
},
"keyinstn": {
"type": "long"
},
"name": {
"type": "string"
},
"state": {
"type": "string"
},
"suggest": {
"properties": {
"input": {
"type": "string"
},
"output": {
"type": "string"
},
"payload": {
"type": "long"
},
"weight": {
"type": "double"
}
}
},
"type": {
"type": "string"
},
"zip": {
"type": "string"
}
}
}
}
}
}
In place of the CompletionObject I created myself, I also tried the intrinsic object, "SuggestField."
If anyone could point me in the right direction of what I am doing wrong, I'd appreciate it. FWIW "normal" searches work fine and return appropriate results.
Thanks.
UPDATE 1/26/2016
So, managed to get an index created that looks "better" ... I was able to get around the new error "analyzer on completion field must be set when search_analyzer is set" by omitting the search analyzer during creation and now my index looks like this:
{
"snl": {
"aliases": {},
"mappings": {
"instn": {
"properties": {
"city": {
"type": "string"
},
"country": {
"type": "string"
},
"keyinstn": {
"type": "long"
},
"name": {
"type": "string"
},
"state": {
"type": "string"
},
"suggest": {
"properties": {
"completion": {
"properties": {
"field": {
"type": "string"
}
}
},
"input": {
"type": "string"
},
"output": {
"type": "string"
},
"payload": {
"type": "long"
},
"text": {
"type": "string"
},
"weight": {
"type": "double"
}
}
},
"type": {
"type": "string"
},
"zip": {
"type": "string"
}
}
}
},
"settings": {
"index": {
"creation_date": "1453835225862",
"number_of_shards": "5",
"number_of_replicas": "1",
"uuid": "90QKK1OyRMKtwcyjAwPElA",
"version": {
"created": "2010199"
}
}
},
"warmers": {}
}
}
However, no completion searches return any results. Is it because the search analyzer omission? If so, how do I get around the "analyzer on completion field must be set when search_analyzer is set" error?

Related

elasticsearch 7 nest aggregation text keyword error

I Have an index with the following mappings:
{
"winnings": {
"mappings": {
"properties": {
"handId": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"playerId": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"value": {
"type": "float"
}
}
}
}
}
generated from the class:
public class ElasticWinnings
{
public Guid Id { get; set; }
public Guid HandId { get; set; }
public Guid PlayerId { get; set; }
public decimal Value { get; set; }
}
I created that in nest with the ConnectionSettings:
.DefaultMappingFor<ElasticWinnings>(u =>
u.IndexName("winnings")
.IdProperty(x => x.Id)
);
when I try and run the following query:
var result = _client.Search<ElasticWinnings>(s =>
s.Aggregations(a =>
a.Terms("term_Agg", t =>
t.Field(f => f.PlayerId)
.Aggregations(aa =>
aa.Sum("sum", sum =>
sum.Field(f => f.Value))
)
))
);
I get a 400 back, with the error:
type: illegal_argument_exception Reason: "Fielddata is disabled on text fields by default
It creates this query:
{
"aggs":{
"term_Agg":{
"aggs":{
"sum":{
"sum":{
"field":"value"
}
}
},
"terms":{
"field":"playerId"
}
}
}
}
If I changed that query to:
{
"aggs":{
"term_Agg":{
"aggs":{
"sum":{
"sum":{
"field":"value"
}
}
},
"terms":{
"field":"playerId.keyword"
}
}
}
}
and used that in postman, it works.
I am not sure why it is not putting the .keyword into the query. Is it the way the nest client is configured, the indicies or the query?
You need to change your query a little bit to tell NEST to use keyword field instead of text, you can do this with .Suffix extension method. Link to docs.
var result = _client.Search<ElasticWinnings>(s =>
s.Aggregations(a =>
a.Terms("term_Agg", t =>
t.Field(f => f.PlayerId.Suffix("keyword"))
.Aggregations(aa =>
aa.Sum("sum", sum =>
sum.Field(f => f.Value))
)
))
);
Hope that helps.
The solution I found was to add [Keyword] to the PlayerId property in ElasticWinnings class.
I kept the .DefaultMappingFor<ElasticWinnings>(u => u.IndexName("winnings") in the creation of the ConnectionSettings class, but added this before the Elastic client is returned:
var client = new ElasticClient(settings);
client.Indices.Create("winnings", c =>
c.Map<ElasticWinnings>(m => m.AutoMap())
);
Without adding the section above, it did not apply the attributes. This changed my mappings (http://localhost:9200/winnings/_mappings) to
{
"winnings": {
"mappings": {
"properties": {
"handId": {
"type": "keyword"
},
"id": {
"type": "keyword"
},
"playerId": {
"type": "keyword"
},
"value": {
"type": "double"
}
}
}
}
}
This is the docs about setting up the mappings https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/fluent-mapping.html

Problem in nest query for sorting in aggregation

I need to get a query from Elasticsearch for last registered records in specified area in range of specified serial numbers.
for this reason I mapped my index in this shape:
{ "settings": {
"index": {
"number_of_shards": 5,
"number_of_replicas": 2
}
},
"mapping": {
"AssetStatus": {
"properties": {
"serialnumber": {
"type": "text",
"fielddata": true
},
"vehiclestate": {
"type": "text",
"fielddata": true
},
"vehiclegeopoint": {
"type": "geo-point",
"fielddata": true
},
"vehiclespeed": {
"type": "number",
"fielddata": true
},
"vehiclefuelpercent": {
"type": "text",
"fielddata": true
},
"devicebatterypercent": {
"type": "text",
"fielddata": true
},
"networklatency": {
"type": "text",
"fielddata": true
},
"satellitescount": {
"type": "number",
"fielddata": true
},
"createdate": {
"type": "date",
"fielddata": true
}
}
}
}
}
and this query works correctly
{
"query": {
"bool": {
"must": [
{
"term": {
"serialnumber.keyword": "2228187d-b1a5-4e18-82bb-4d12438e0ec0"
}
},
{
"range": {
"vehiclegeopoint.lat": {
"gt": "31.287958",
"lt": "31.295485"
}
}
},
{
"range": {
"vehiclegeopoint.lon": {
"gt": "48.639844",
"lt": "48.652032"
}
}
}
],
"must_not": [],
"should": []
}
},
"from": 0,
"size": 0,
"sort": [],
"aggs": {
"SerialNumberGroups": {
"terms": {
"field": "serialnumber.keyword"
},
"aggs": {
"tops": {
"top_hits": {
"sort": [
{
"createdate.keyword": {
"order": "desc"
}
}
],
"size": 1
}
}
}
}
}
}
whereas my nest query has this error
Invalid NEST response built from a unsuccessful low level call on
POST: /fms2/AssetStatus/_search?typed_keys=true
Audit trail of this API call:
[1] BadResponse: Node: http://localhost:9200/ Took: 00:00:00.1917118
OriginalException: Elasticsearch.Net.ElasticsearchClientException: Request failed to execute. Call: Status code 400 from: POST
/fms2/AssetStatus/_search?typed_keys=true. ServerError: Type:
parsing_exception Reason: "Unknown key for a VALUE_STRING in [field]."
Request:
force it to be set on the response.>
Response:
ConnectionSettings to force it to be set on the response.>
my nest query is this
var searchResponse =
Client.Search<AssetStatus>(x => x
.Index(settings.DefaultIndex)
.Type("AssetStatus")
.Query(fq => fq.GeoBoundingBox(c => c.Field(f => f.VehicleGeoPoint).BoundingBox(new GeoLocation(TopLeft.Lat, TopLeft.Lon), new GeoLocation(BottomRight.Lat, BottomRight.Lon))))
.Query(fq =>
fq.Bool(b => b.
Filter(
f => f.Match(m => m.Field(g => g.SerialNumber.Suffix("keyword").Equals(sn)))
)
))
.Aggregations(a => a
.Terms("group_by_SerialNumber", st => st
.Field(o => o.SerialNumber.Suffix("keyword"))
.Size(0)
.Aggregations(b=> b.TopHits("top_hits", lastRegistered => lastRegistered
.Field(bf=> bf.CreateDate.Suffix("keyword"))
.Size(1)))
))
);
This problem is for sorting in aggregation
Problem was in my POCO, as you can see in my nest query I use uppercase letters for naming my properties. I should use Nest library for using data annotation for elastic's POCO.
[ElasticsearchType(Name = "AssetStatus")]
public class AssetStatus
{
[Text]
[PropertyName("serialnumber")]
public string SerialNumber { get; set; }
[Text]
[PropertyName("vehiclestate")]
public string VehicleState { get; set; }
[GeoPoint]
[PropertyName("vehiclegeopoint")]
public GeoPoint VehicleGeoPoint { get; set; }
[Number]
[PropertyName("vehiclespeed")]
public int VehicleSpeed { get; set; }
[Text]
[PropertyName("vehiclefuelpercent")]
public string VehicleFuelPercent { get; set; }
[Text]
[PropertyName("devicebatterypercent")]
public string DeviceBatteryPercent { get; set; }
[Text]
[PropertyName("networklatency")]
public string NetworkLatency { get; set; }
[Number]
[PropertyName("satellitescount")]
public byte SatellitesCount { get; set; }
[Date]
[PropertyName("createdate")]
public string CreateDate { get; set; }
}

ElasticSearch Nest: AutoMap with DynamicTemplates

I'm trying use a dynamic template in ES such that all string fields are multifields. I also want to apply some specific mappings to certain fields.
Take the following example class:
[ElasticsearchType(Name = "sample1")]
public class Sample1
{
public string ID { get; set; }
[String(Index = FieldIndexOption.No)]
public string DoNotIndex { get; set; }
public string MultiField1 { get; set; }
public string MultiField2 { get; set; }
}
I want to then create the dynamic template and apply the mapping to DoNotIndex using the following command:
_client.Map<Sample1>(m => m
.AutoMap()
.DynamicTemplates(dt=> dt
.DynamicTemplate("all_strings_multifields", t => t
.MatchMappingType("string")
.Mapping(tm => tm
.String(mf => mf
.Index(FieldIndexOption.Analyzed)
.Fields(mff => mff
.String(s => s
.Name("raw")
.Index(FieldIndexOption.NotAnalyzed)
)
)
)
)
)
)
)
.VerifySuccessfulResponse();
The result is:
{
"test1": {
"mappings": {
"sample1": {
"dynamic_templates": [
{
"all_strings_multifields": {
"match_mapping_type": "string",
"mapping": {
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed"
}
},
"index": "analyzed",
"type": "string"
}
}
}
],
"properties": {
"doNotIndex": {
"type": "keyword",
"index": false
},
"iD": {
"type": "text"
},
"multiField1": {
"type": "text"
},
"multiField2": {
"type": "text"
}
}
}
}
}
}
Results
you'll see that the DoNotIndex property is indeed correct, but the multifield1 and multifield2 are not correct (they are not multi fields).
Workaround
I know I can "fix" this by NOT doing the AutoMap() and instead specifying each of the special indexes but there are a lot of fields and that isn't as clean of a solution.
Can I do AutoMap with DynamicTemplates?
Dynamic templates only apply to fields that are dynamically added to a mapping, so properties explicitly mapped with .AutoMap() will not be affected by dynamic mapping.
There is however a way to apply conventions to explicit mappings with NEST using the visitor pattern. It looks like you're using Elasticsearch 5.0, so you should use the text and keyword mappings.
First define a visitor
[ElasticsearchType(Name = "sample1")]
public class Sample1
{
public string ID { get; set; }
[Keyword(Index = false)]
public string DoNotIndex { get; set; }
public string MultiField1 { get; set; }
public string MultiField2 { get; set; }
}
public class AllStringsMultiFieldsVisitor : NoopPropertyVisitor
{
public override void Visit(ITextProperty type, PropertyInfo propertyInfo, ElasticsearchPropertyAttributeBase attribute)
{
// if a custom attribute has been applied, let it take precedence
if (propertyInfo.GetCustomAttribute<ElasticsearchPropertyAttributeBase>() == null)
{
type.Fields = new Properties
{
{
"raw", new KeywordProperty()
}
};
}
base.Visit(type, propertyInfo, attribute);
}
}
Then pass an instance of the visitor to .AutoMap()
client.Map<Sample1>(m => m
.AutoMap(new AllStringsMultiFieldsVisitor())
.DynamicTemplates(dt => dt
.DynamicTemplate("all_strings_multifields", t => t
.MatchMappingType("text")
.Mapping(tm => tm
.Text(mf => mf
.Index(true)
.Fields(mff => mff
.Keyword(s => s
.Name("raw")
)
)
)
)
)
)
);
produces
{
"dynamic_templates": [
{
"all_strings_multifields": {
"match_mapping_type": "text",
"mapping": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
},
"index": true
}
}
}
],
"properties": {
"iD": {
"fields": {
"raw": {
"type": "keyword"
}
},
"type": "text"
},
"doNotIndex": {
"type": "keyword",
"index": false
},
"multiField1": {
"fields": {
"raw": {
"type": "keyword"
}
},
"type": "text"
},
"multiField2": {
"fields": {
"raw": {
"type": "keyword"
}
},
"type": "text"
}
}
}
I should point out however that the default automapping for a C# string property in NEST 5.0 is to map it as a text field with a keyword sub field with ignore_above:256. NEST 5.0 was released to nuget earlier this week
client.Map<Sample1>(m => m
.AutoMap()
.DynamicTemplates(dt => dt
.DynamicTemplate("all_strings_multifields", t => t
.MatchMappingType("text")
.Mapping(tm => tm
.Text(mf => mf
.Index(true)
.Fields(mff => mff
.Keyword(s => s
.Name("raw")
)
)
)
)
)
)
);
produces
{
"dynamic_templates": [
{
"all_strings_multifields": {
"match_mapping_type": "text",
"mapping": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
},
"index": true
}
}
}
],
"properties": {
"iD": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
},
"doNotIndex": {
"type": "keyword",
"index": false
},
"multiField1": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
},
"multiField2": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
}
}
}

Combining Attribute / Fluent Mapping for Copy_To with Nest

I want to use the copy_to functionality with Nest. I've read that I need to use the fluent mapping (Elasticsearch Nest and CopyTo).
Is it possible to use the attribute based mapping then fluent mapping on top of that to add copy_to? If so, are there any examples? I'm having difficulty finding the answer.
The field I want to copy to does not exist in my model class. I just want to search on it in elasticsearch.
[ElasticType(IdProperty = "CustomerId", Name = "customer_search")]
public class CustomerSearchResult : BindableBase
{
[ElasticProperty(Name = "customer_id", Index = FieldIndexOption.NotAnalyzed, Type = FieldType.String)]
public int CustomerId { get; set; }
[ElasticProperty(Name = "account_type", Index = FieldIndexOption.NotAnalyzed, Type = FieldType.String)]
public string AccountType { get; set; }
[ElasticProperty(Name = "short_name", Index = FieldIndexOption.NotAnalyzed, Type = FieldType.String)]
public string ShortName { get; set; }
[ElasticProperty(Name = "legacy_name", Index = FieldIndexOption.NotAnalyzed, Type = FieldType.String)]
public string LegacyName { get; set; }
[ElasticProperty(Name = "legacy_contact_name", Index = FieldIndexOption.NotAnalyzed, Type = FieldType.String)]
public string LegacyContactName { get; set; }
[ElasticProperty(Name = "city", Index = FieldIndexOption.NotAnalyzed, Type = FieldType.String)]
public string City { get; set; }
[ElasticProperty(Name = "state_abbreviation", Index = FieldIndexOption.NotAnalyzed, Type = FieldType.String)]
public string StateAbbreviation { get; set; }
[ElasticProperty(Name = "country", Index = FieldIndexOption.NotAnalyzed, Type = FieldType.String)]
public string Country { get; set; }
[ElasticProperty(Name = "postal_code", Index = FieldIndexOption.NotAnalyzed, Type = FieldType.String)]
public string PostalCode { get; set; }
}
In the above class I want to use ShortName, LegacyName, and LegacyContactName and copy_to a field called "search" which will be an analyzed field.
Something like the following should do it
void Main()
{
var settings = new ConnectionSettings(new Uri("http://localhost:9200"));
var connection = new InMemoryConnection(settings);
var client = new ElasticClient(connection: connection);
var indexResponse = client.CreateIndex("customer_searches", c => c
.AddMapping<CustomerSearchResult>(m => m
.MapFromAttributes()
.Properties(p => p
.String(s => s.Name("short_name").CopyTo("search").Index(FieldIndexOption.NotAnalyzed))
.String(s => s.Name("legacy_name").CopyTo("search").Index(FieldIndexOption.NotAnalyzed))
.String(s => s.Name("legacy_contact_name").CopyTo("search").Index(FieldIndexOption.NotAnalyzed))
.String(s => s.Name("search").Index(FieldIndexOption.Analyzed))
)
)
);
Console.WriteLine(Encoding.UTF8.GetString(indexResponse.RequestInformation.Request));
}
Which outputs
{
"settings": {
"index": {}
},
"mappings": {
"customer_search": {
"properties": {
"customer_id": {
"index": "not_analyzed",
"type": "string"
},
"account_type": {
"index": "not_analyzed",
"type": "string"
},
"short_name": {
"index": "not_analyzed",
"copy_to": [
"search"
],
"type": "string"
},
"legacy_name": {
"index": "not_analyzed",
"copy_to": [
"search"
],
"type": "string"
},
"legacy_contact_name": {
"index": "not_analyzed",
"copy_to": [
"search"
],
"type": "string"
},
"city": {
"index": "not_analyzed",
"type": "string"
},
"state_abbreviation": {
"index": "not_analyzed",
"type": "string"
},
"country": {
"index": "not_analyzed",
"type": "string"
},
"postal_code": {
"index": "not_analyzed",
"type": "string"
},
"search": {
"index": "analyzed",
"type": "string"
}
}
}
}
}
The call to Properties() overrides the default conventions and attribute mappings so you need to specify that the fields are not_analyzed there as well.

How to exclude inherited object properties from mappings

I'm trying to setup a mapping for an object that looks like this:
class TestObject
{
public long TestID { get; set; }
[ElasticProperty(Type = FieldType.Object)]
public Dictionary<long, List<DateTime>> Items { get; set; }
}
I use the following mapping code (where Client is IElasticClient):
this.Client.Map<TestObject>(m => m.MapFromAttributes());
I get the following mapping result:
{
"mappings": {
"testobject": {
"properties": {
"items": {
"properties": {
"comparer": {
"type": "object"
},
"count": {
"type": "integer"
},
"item": {
"type": "date",
"format": "dateOptionalTime"
},
"keys": {
"properties": {
"count": {
"type": "integer"
}
}
},
"values": {
"properties": {
"count": {
"type": "integer"
}
}
}
}
},
"testID": {
"type": "long"
}
}
}
}
This becomes a problem when I want to do a search like this:
{
"query_string": {
"query": "[2015-06-03T00:00:00.000 TO 2015-06-05T23:59:59.999]",
"fields": [
"items.*"
]
}
}
This causes exceptions, that I guess are because of all the fields in the items object are not of the same type. What is the proper mapping to searches of this type?
I was able to fix this by using the following mapping:
this.Client.Map<TestObject>(m => m.MapFromAttributes())
.Properties(p => p
.Object<Dictionary<long, List<DateTime>>>(o => o.Name("items")));

Resources