I want to use the copy_to functionality with Nest. I've read that I need to use the fluent mapping (Elasticsearch Nest and CopyTo).
Is it possible to use the attribute based mapping then fluent mapping on top of that to add copy_to? If so, are there any examples? I'm having difficulty finding the answer.
The field I want to copy to does not exist in my model class. I just want to search on it in elasticsearch.
[ElasticType(IdProperty = "CustomerId", Name = "customer_search")]
public class CustomerSearchResult : BindableBase
{
[ElasticProperty(Name = "customer_id", Index = FieldIndexOption.NotAnalyzed, Type = FieldType.String)]
public int CustomerId { get; set; }
[ElasticProperty(Name = "account_type", Index = FieldIndexOption.NotAnalyzed, Type = FieldType.String)]
public string AccountType { get; set; }
[ElasticProperty(Name = "short_name", Index = FieldIndexOption.NotAnalyzed, Type = FieldType.String)]
public string ShortName { get; set; }
[ElasticProperty(Name = "legacy_name", Index = FieldIndexOption.NotAnalyzed, Type = FieldType.String)]
public string LegacyName { get; set; }
[ElasticProperty(Name = "legacy_contact_name", Index = FieldIndexOption.NotAnalyzed, Type = FieldType.String)]
public string LegacyContactName { get; set; }
[ElasticProperty(Name = "city", Index = FieldIndexOption.NotAnalyzed, Type = FieldType.String)]
public string City { get; set; }
[ElasticProperty(Name = "state_abbreviation", Index = FieldIndexOption.NotAnalyzed, Type = FieldType.String)]
public string StateAbbreviation { get; set; }
[ElasticProperty(Name = "country", Index = FieldIndexOption.NotAnalyzed, Type = FieldType.String)]
public string Country { get; set; }
[ElasticProperty(Name = "postal_code", Index = FieldIndexOption.NotAnalyzed, Type = FieldType.String)]
public string PostalCode { get; set; }
}
In the above class I want to use ShortName, LegacyName, and LegacyContactName and copy_to a field called "search" which will be an analyzed field.
Something like the following should do it
void Main()
{
var settings = new ConnectionSettings(new Uri("http://localhost:9200"));
var connection = new InMemoryConnection(settings);
var client = new ElasticClient(connection: connection);
var indexResponse = client.CreateIndex("customer_searches", c => c
.AddMapping<CustomerSearchResult>(m => m
.MapFromAttributes()
.Properties(p => p
.String(s => s.Name("short_name").CopyTo("search").Index(FieldIndexOption.NotAnalyzed))
.String(s => s.Name("legacy_name").CopyTo("search").Index(FieldIndexOption.NotAnalyzed))
.String(s => s.Name("legacy_contact_name").CopyTo("search").Index(FieldIndexOption.NotAnalyzed))
.String(s => s.Name("search").Index(FieldIndexOption.Analyzed))
)
)
);
Console.WriteLine(Encoding.UTF8.GetString(indexResponse.RequestInformation.Request));
}
Which outputs
{
"settings": {
"index": {}
},
"mappings": {
"customer_search": {
"properties": {
"customer_id": {
"index": "not_analyzed",
"type": "string"
},
"account_type": {
"index": "not_analyzed",
"type": "string"
},
"short_name": {
"index": "not_analyzed",
"copy_to": [
"search"
],
"type": "string"
},
"legacy_name": {
"index": "not_analyzed",
"copy_to": [
"search"
],
"type": "string"
},
"legacy_contact_name": {
"index": "not_analyzed",
"copy_to": [
"search"
],
"type": "string"
},
"city": {
"index": "not_analyzed",
"type": "string"
},
"state_abbreviation": {
"index": "not_analyzed",
"type": "string"
},
"country": {
"index": "not_analyzed",
"type": "string"
},
"postal_code": {
"index": "not_analyzed",
"type": "string"
},
"search": {
"index": "analyzed",
"type": "string"
}
}
}
}
}
The call to Properties() overrides the default conventions and attribute mappings so you need to specify that the fields are not_analyzed there as well.
Related
I'm in trouble with Nest. I have a brand new index on Elasticsearch 5.1.1 and I'm trying to define a type mapping by dotnet core.
My classes look like that:
public class Review
{
public Guid Id { get; set; }
public User User { get; set; }
public Movie Movie { get; set; }
public int Grade { get; set; }
public string Title { get; set; }
public string Comment { get; set; }
public bool HasSpoiler { get; set; }
public bool BoughtInIngresso { get; set; }
public ReviewStatus Status { get; set; }
public DateTime Date { get; set; }
}
public class User
{
public string Id { get; set; }
public string Name { get; set; }
}
public class Movie
{
public string Id { get; set; }
public string Name { get; set; }
}
In my application, I'm trying to define a short form of the type mapping (just for test) like that:
var pool = new StaticConnectionPool(nodes);
var settings = new ConnectionSettings(pool);
settings.DefaultIndex(elasticSettings.IndexName);
var client = new ElasticClient(settings);
client.Map<Review>(m =>
m.Index("my_index")
.Type("reviews")
.Properties(ps=>
ps.Keyword(k=>
k.Name("title"))
.Text(t=>
t.Name("comment"))
)
);
And the final result is this. Observe the reviews AND review mapping being created. I only want "reviews", not "review".
{
"my_index": {
"mappings": {
"reviews": {
"properties": {
"comment": {
"type": "text"
},
"title": {
"type": "keyword"
}
}
},
"review": {
"properties": {
"boughtInSite": {
"type": "boolean"
},
"comment": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"date": {
"type": "date"
},
"grade": {
"type": "long"
},
"hasSpoiler": {
"type": "boolean"
},
"id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"movie": {
"properties": {
"id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"status": {
"type": "long"
},
"title": {
"type": "keyword"
},
"user": {
"properties": {
"id": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"name": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
}
}
}
}
}
}
What I'm doing wrong? This happen if I use AutoMap() too. I don't want to map with attributes because I want to preserve my POCO classes, but if was the only way, I can do.
Some help?
You haven't shown how you are indexing Review types, but I suspect that you're not specifying the type as "reviews" when indexing; in this case, NEST will infer the type from the C# POCO type name.
NEST provides a way for the Elasticsearch type "reviews" to be associated with the C# POCO type Review using InferMappingFor<T>() on ConnectionSettings
public class Review
{
public Guid Id { get; set; }
public User User { get; set; }
public Movie Movie { get; set; }
public int Grade { get; set; }
public string Title { get; set; }
public string Comment { get; set; }
public bool HasSpoiler { get; set; }
public bool BoughtInIngresso { get; set; }
public ReviewStatus Status { get; set; }
public DateTime Date { get; set; }
}
public enum ReviewStatus
{
Good,
Bad,
NotTooShabby
}
public class User
{
public string Id { get; set; }
public string Name { get; set; }
}
public class Movie
{
public string Id { get; set; }
public string Name { get; set; }
}
void Main()
{
var pool = new SingleNodeConnectionPool(new Uri("http://localhost:9200"));
var defaultIndex = "default-index";
var connectionSettings = new ConnectionSettings(pool)
.InferMappingFor<Review>(m => m
.TypeName("reviews")
)
.DefaultIndex(defaultIndex);
var client = new ElasticClient(connectionSettings);
client.CreateIndex("my_index", c => c
.Mappings(m => m
.Map<Review>(mm => mm
.Properties(ps => ps
.Keyword(k => k
.Name(n => n.Title)
)
.Text(t => t
.Name(n => n.Comment)
)
)
)
)
);
}
With this approach, you don't need to specify the type on every request now, although you can override it per request if you need to.
With AutoMap(), you can let NEST infer the Elasticsearch field types from the C# POCO property types, then use .Properties() to override any mappings you'd like.
I am using elastic search with nest 5.0
I need to increase ignore_above from default 256 to 512.
Can I do this by the Attribute mappings?
Otherwise how can I do this using the fluent API?
You can use [Keyword(IgnoreAbove=512)] to change the value:
[ElasticsearchType(Name = "mytype")]
class mytype
{
// default: will be mapped to both keyword (keyword search)
// and text (full-text search)
public string defaultString { get; set; }
// by default ignore above is set to 256
[Keyword]
public string keywordType { get; set; }
// change ignore above
[Keyword(IgnoreAbove=512)]
public string longKeyword { get; set; }
// text type, full-text search
[Text]
public string textType { get; set; }
// store but not searched
[Text(Index = false)]
public string textTypeNotSearchable { get; set; }
}
This is the created index in ElasticSearch for the above type:
"defaultString": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"keywordType": {
"type": "keyword"
},
"longKeyword": {
"type": "keyword",
"ignore_above": 512
},
"textType": {
"type": "text"
},
"textTypeNotSearchable": {
"type": "text",
"index": false
}
I'm trying use a dynamic template in ES such that all string fields are multifields. I also want to apply some specific mappings to certain fields.
Take the following example class:
[ElasticsearchType(Name = "sample1")]
public class Sample1
{
public string ID { get; set; }
[String(Index = FieldIndexOption.No)]
public string DoNotIndex { get; set; }
public string MultiField1 { get; set; }
public string MultiField2 { get; set; }
}
I want to then create the dynamic template and apply the mapping to DoNotIndex using the following command:
_client.Map<Sample1>(m => m
.AutoMap()
.DynamicTemplates(dt=> dt
.DynamicTemplate("all_strings_multifields", t => t
.MatchMappingType("string")
.Mapping(tm => tm
.String(mf => mf
.Index(FieldIndexOption.Analyzed)
.Fields(mff => mff
.String(s => s
.Name("raw")
.Index(FieldIndexOption.NotAnalyzed)
)
)
)
)
)
)
)
.VerifySuccessfulResponse();
The result is:
{
"test1": {
"mappings": {
"sample1": {
"dynamic_templates": [
{
"all_strings_multifields": {
"match_mapping_type": "string",
"mapping": {
"fields": {
"raw": {
"type": "string",
"index": "not_analyzed"
}
},
"index": "analyzed",
"type": "string"
}
}
}
],
"properties": {
"doNotIndex": {
"type": "keyword",
"index": false
},
"iD": {
"type": "text"
},
"multiField1": {
"type": "text"
},
"multiField2": {
"type": "text"
}
}
}
}
}
}
Results
you'll see that the DoNotIndex property is indeed correct, but the multifield1 and multifield2 are not correct (they are not multi fields).
Workaround
I know I can "fix" this by NOT doing the AutoMap() and instead specifying each of the special indexes but there are a lot of fields and that isn't as clean of a solution.
Can I do AutoMap with DynamicTemplates?
Dynamic templates only apply to fields that are dynamically added to a mapping, so properties explicitly mapped with .AutoMap() will not be affected by dynamic mapping.
There is however a way to apply conventions to explicit mappings with NEST using the visitor pattern. It looks like you're using Elasticsearch 5.0, so you should use the text and keyword mappings.
First define a visitor
[ElasticsearchType(Name = "sample1")]
public class Sample1
{
public string ID { get; set; }
[Keyword(Index = false)]
public string DoNotIndex { get; set; }
public string MultiField1 { get; set; }
public string MultiField2 { get; set; }
}
public class AllStringsMultiFieldsVisitor : NoopPropertyVisitor
{
public override void Visit(ITextProperty type, PropertyInfo propertyInfo, ElasticsearchPropertyAttributeBase attribute)
{
// if a custom attribute has been applied, let it take precedence
if (propertyInfo.GetCustomAttribute<ElasticsearchPropertyAttributeBase>() == null)
{
type.Fields = new Properties
{
{
"raw", new KeywordProperty()
}
};
}
base.Visit(type, propertyInfo, attribute);
}
}
Then pass an instance of the visitor to .AutoMap()
client.Map<Sample1>(m => m
.AutoMap(new AllStringsMultiFieldsVisitor())
.DynamicTemplates(dt => dt
.DynamicTemplate("all_strings_multifields", t => t
.MatchMappingType("text")
.Mapping(tm => tm
.Text(mf => mf
.Index(true)
.Fields(mff => mff
.Keyword(s => s
.Name("raw")
)
)
)
)
)
)
);
produces
{
"dynamic_templates": [
{
"all_strings_multifields": {
"match_mapping_type": "text",
"mapping": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
},
"index": true
}
}
}
],
"properties": {
"iD": {
"fields": {
"raw": {
"type": "keyword"
}
},
"type": "text"
},
"doNotIndex": {
"type": "keyword",
"index": false
},
"multiField1": {
"fields": {
"raw": {
"type": "keyword"
}
},
"type": "text"
},
"multiField2": {
"fields": {
"raw": {
"type": "keyword"
}
},
"type": "text"
}
}
}
I should point out however that the default automapping for a C# string property in NEST 5.0 is to map it as a text field with a keyword sub field with ignore_above:256. NEST 5.0 was released to nuget earlier this week
client.Map<Sample1>(m => m
.AutoMap()
.DynamicTemplates(dt => dt
.DynamicTemplate("all_strings_multifields", t => t
.MatchMappingType("text")
.Mapping(tm => tm
.Text(mf => mf
.Index(true)
.Fields(mff => mff
.Keyword(s => s
.Name("raw")
)
)
)
)
)
)
);
produces
{
"dynamic_templates": [
{
"all_strings_multifields": {
"match_mapping_type": "text",
"mapping": {
"type": "text",
"fields": {
"raw": {
"type": "keyword"
}
},
"index": true
}
}
}
],
"properties": {
"iD": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
},
"doNotIndex": {
"type": "keyword",
"index": false
},
"multiField1": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
},
"multiField2": {
"fields": {
"keyword": {
"ignore_above": 256,
"type": "keyword"
}
},
"type": "text"
}
}
}
I am using Elasticsearch 2.3 - Nest API to search data. I am using attribute mapping for documents. I would like to know how to use Phonetic Analyzer using attribute mapping.
The document class:
[ElasticsearchType(Name = "contact", IdProperty = nameof(EntityId))]
public class ESContactSearchDocument
{
[String(Name = "entityid")]
public string EntityId { get; set; }
[String(Name = "id")]
public string Id { get; set; }
[String(Name = "name")]
public string Name { get; set; }
[String(Name = "domain")]
public string Domain { get; set; }
[String(Name = "description")]
public string Description { get; set; }
[String(Name = "email")]
public string Email { get; set; }
[String(Name = "firstname", Analyzer = "phonetic")]
public string FirstName { get; set; }
[String(Name = "lastname", Analyzer = "phonetic")]
public string LastName { get; set; }
}
Index creation and insertion:
var createIndexResponse = serviceClient.CreateIndex(indexName, c => c.Mappings(m => m.Map<ESContactSearchDocument>(d => d.AutoMap())));
var indexManyResponse = await serviceClient.IndexManyAsync(ESMapper.Map<TDocument, ESContactSearchDocument>(documentsBatch), indexName, typeName);
The ESMapper is only used to convert from one type to another.
Resultant mapping:
{
"contact-uklwcl072": {
"mappings": {
"contact": {
"properties": {
"description": {
"type": "string"
},
"domain": {
"type": "string"
},
"email": {
"type": "string"
},
"entityid": {
"type": "string"
},
"firstname": {
"type": "string"
},
"id": {
"type": "string"
},
"lastname": {
"type": "string"
},
"name": {
"type": "string"
}
}
}
}
}
}
I have also installed Phonetic Analysis Plugin
Take a look at the automapping documentation; you need to assign the name of the analyzer to the Analyzer property on the attribute
public class Document
{
[String(Analyzer="phonetic")]
public string Name {get; set;}
}
then call AutoMap() when mapping your document type e.g. when you create the index
client.CreateIndex("index-name", c => c
.Mappings(m => m
.Map<Document>(d => d
.AutoMap()
)
)
);
yields
{
"mappings": {
"document": {
"properties": {
"name": {
"type": "string",
"analyzer": "phonetic"
}
}
}
}
}
In between projects, started to play around with Elastic Search and NEST in preparation for some auto-completion features coming up.
I followed examples here and in the various docs ... and it seems like what I have should work. There must be some subtle difference with what i am doing.
This is the object that represents my document.
[ElasticType(IdProperty="KeyInstn")]
public class Instn
{
[ElasticProperty(Name = "keyinstn", Index = FieldIndexOption.Analyzed, Type = FieldType.Integer)]
public int KeyInstn { get; set; }
public string Type { get; set; }
[ElasticProperty(Name = "name", Index = FieldIndexOption.Analyzed, Type = FieldType.String)]
public string Name { get; set; }
[ElasticProperty(Name = "ticker", Index = FieldIndexOption.Analyzed, Type = FieldType.String)]
public string Ticker { get; set; }
[ElasticProperty(Name = "street", Index = FieldIndexOption.Analyzed, Type = FieldType.String)]
public string Street { get; set; }
[ElasticProperty(Name = "city", Index = FieldIndexOption.Analyzed, Type = FieldType.String)]
public string City { get; set; }
[ElasticProperty(Name = "state", Index = FieldIndexOption.Analyzed, Type = FieldType.String)]
public string State { get; set; }
[ElasticProperty(Name = "zip", Index = FieldIndexOption.Analyzed, Type = FieldType.String)]
public string Zip { get; set; }
[ElasticProperty(Name = "country", Index = FieldIndexOption.Analyzed, Type = FieldType.String)]
public string Country { get; set; }
[ElasticProperty(Name = "suggest", Type = FieldType.Completion)]
public CompletionField Suggest { get; set; }
}
This is the code that I use to create my index:
var index = client.CreateIndex("snl", c => c
.NumberOfReplicas(1)
.NumberOfShards(5)
.Settings(s => s
.Add("merge.policy.merge_factor", "10")
.Add("search.slowlog.threshold.fetch.warn", "1s")
)
.AddMapping<Instn>(m => m.MapFromAttributes()
.Properties(props => props
.Completion(s => s
.Name(p => p.Suggest)
.IndexAnalyzer("simple")
.SearchAnalyzer("simple")
.MaxInputLength(20)
.Payloads()
.PreservePositionIncrements()
.PreserveSeparators()
)
)
));
I am populating my documents using IndexMany in batches of 1000 documents.
I then perform my query:
var suggestResponse = this._client.Suggest<Instn>(s => s
.Index("snl")
.Completion("suggest", c => c
.Text(this.AutoCompleteText.Text)
.OnField("suggest")));
The error I receive is:
Field [suggest] is not a completion suggest field
Using SENSE, I pulls the mappings and notice that "suggest" doesn't have a type. The examples I followed indicate that the type should be 'completion.:'
Elasticsearch - Autocomplete with NEST
{
"snl": {
"mappings": {
"instn": {
"properties": {
"city": {
"type": "string"
},
"country": {
"type": "string"
},
"keyinstn": {
"type": "long"
},
"name": {
"type": "string"
},
"state": {
"type": "string"
},
"suggest": {
"properties": {
"input": {
"type": "string"
},
"output": {
"type": "string"
},
"payload": {
"type": "long"
},
"weight": {
"type": "double"
}
}
},
"type": {
"type": "string"
},
"zip": {
"type": "string"
}
}
}
}
}
}
In place of the CompletionObject I created myself, I also tried the intrinsic object, "SuggestField."
If anyone could point me in the right direction of what I am doing wrong, I'd appreciate it. FWIW "normal" searches work fine and return appropriate results.
Thanks.
UPDATE 1/26/2016
So, managed to get an index created that looks "better" ... I was able to get around the new error "analyzer on completion field must be set when search_analyzer is set" by omitting the search analyzer during creation and now my index looks like this:
{
"snl": {
"aliases": {},
"mappings": {
"instn": {
"properties": {
"city": {
"type": "string"
},
"country": {
"type": "string"
},
"keyinstn": {
"type": "long"
},
"name": {
"type": "string"
},
"state": {
"type": "string"
},
"suggest": {
"properties": {
"completion": {
"properties": {
"field": {
"type": "string"
}
}
},
"input": {
"type": "string"
},
"output": {
"type": "string"
},
"payload": {
"type": "long"
},
"text": {
"type": "string"
},
"weight": {
"type": "double"
}
}
},
"type": {
"type": "string"
},
"zip": {
"type": "string"
}
}
}
},
"settings": {
"index": {
"creation_date": "1453835225862",
"number_of_shards": "5",
"number_of_replicas": "1",
"uuid": "90QKK1OyRMKtwcyjAwPElA",
"version": {
"created": "2010199"
}
}
},
"warmers": {}
}
}
However, no completion searches return any results. Is it because the search analyzer omission? If so, how do I get around the "analyzer on completion field must be set when search_analyzer is set" error?