load specific fields in Elasticsearch Nest query - elasticsearch

the documentation seems to indicate i can return a subset of fields instead of the entire document. here's my code:
var result = client.Search<MyObject>(s => s
.Fields(f => f.Title)
.Query(q => q
.QueryString(qs => qs
.OnField("title")
.Query("the"))));
i'm searching on the word 'the' on the 'title' field and wanting to just return 'title'. my result.Documents object contains 10 objects that are each null.
i do see the values i want but it's deep in the search response:
result.Hits[0].Fields.FieldValues[0]...
is there a better way to get at the list of 'title' fields returned?
my mapping for the data (truncated) is this ...
{
"myidex": {
"mappings": {
"myobject": {
"properties": {
"title": {
"type": "string"
},
"artists": {
"properties": {
"id": {
"type": "string",
"index": "not_analyzed",
"analyzer": "fullTerm"
},
"name": {
"type": "string",
"index": "not_analyzed",
"analyzer": "fullTerm"
}
}
}
}
}
}
}
}
and my class objects are like this:
[Table("MyTable")]
[Serializable]
[ElasticType(Name="myobject")]
public class MyObject
{
[ElasticProperty]
public string Title { get; set; }
[JsonIgnore]
public string Artistslist { get; set; }
[ElasticProperty(Analyzer = "caseInsensitive")]
public List<Person> Artists { get; set; }
}
[Serializable]
public class Person
{
[ElasticProperty(Analyzer = "fullTerm", Index = FieldIndexOption.not_analyzed)]
public string Name { get; set; }
[ElasticProperty(Analyzer = "fullTerm", Index = FieldIndexOption.not_analyzed)]
public string Id { get; set; }
}
Artistslist comes from my data source (sql) then i parse it out into a new List object before indexing the data.

I think this deeply nested value is do to a change in Elasticsearch 1.0 and how partial fields are now returned as arrays (See 1.0 Breaking Changes - Return Values for details.). This is addressed in the NEST 1.0 Breaking Changes documentation; in the Fields() vs SourceIncludes() section. It shows an example of using a FieldValue helper method to gain access to these values. Based on that, try the following:
For all items:
foreach (var hit in result.Hits)
{
var title = hit.Fields.FieldValue<MyObject, string>(f => f.Title);
}
For a specific item:
var title = result.Hits.ElementAt(0)
.Fields.FieldValue<MyObject, string>(f => f.Title);
I know it is still a bit verbose but it should work for you and will handle the new array return formatting of Elasticsearch 1.0.

I found the solution in Nest's Github repo. They have created an issue about this problem. You should use FielddataFields instead of Fields.
https://github.com/elastic/elasticsearch-net/issues/1551
var result = client.Search<MyObject>(s => s
.FielddataFields(f => f.Title)
.Query(q => q
.QueryString(qs => qs
.OnField("title")
.Query("the"))));
and in response you see FieldSelections. You get the fields that you wanted.

Related

Abstract object not mapped correctly in Elasticsearch using Nest 7.0.0-alpha1

I am using NEST (.NET 4.8) to import my data, and I have a problem getting the mapping to work in NEST 7.0.0-alpha1.
I have the following class structure:
class LinkActor
{
public Actor Actor { get; set; }
}
abstract class Actor
{
public string Description { get; set; }
}
class Person : Actor
{
public string Name { get; set; }
}
I connect to Elasticsearch this way:
var connectionSettings = new ConnectionSettings(new Uri(connection));
connectionSettings.DefaultIndex(indexName);
var client = new ElasticClient(connectionSettings);
The actual data looks like this:
var personActor = new Person
{
Description = "Description",
Name = "Name"
};
var linkActor = new LinkActor
{
Actor = personActor
};
And the data is indexed like this:
result = client.IndexDocument(linkActor);
Using NEST 6.6 I am getting the following data in Elasticsearch 6.5.2:
"actor": {
"name": "Name",
"description": "Description"
}
However when using NEST 7.0.0-alpha1 I get the following data in Elasticsearch 7.0.0:
"actor": {
"description": "Description"
}
So the data from the concrete class is missing. I am obviously missing / not understanding some new mapping feature, but my attempts with AutoMap has failed:
client.Map<(attempt with each of the above classes)>(m => m.AutoMap());
Is is still possible to map the data from the concrete class in NEST 7.0.0-alpha1?
I found a workaround using the NEST.JsonNetSerializer (remember to install this), which allows me to pass a JObject directly:
Connect to Elasticsearch using a pool so you can add the JsonNetSerializer.Default:
var pool = new SingleNodeConnectionPool(new Uri(connection));
var connectionSettings = new ConnectionSettings(pool, JsonNetSerializer.Default);
connectionSettings.DefaultIndex(indexName);
var client = new ElasticClient(connectionSettings);
Convert the linkActor object from above to a JObject (JsonSerializerSettings omitted for clarity, add them to get CamelCasing):
var linkActorSerialized = JsonConvert.SerializeObject(linkActor);
var linkActorJObject = JObject.Parse(linkActorSerialized);
result = client.IndexDocument(linkActorJObject);
This gives the desired result:
"actor": {
"name": "Name",
"description": "Description"
}
It is a workaround, hopefully someone will be able to explain the mapping in the question.

In Nest 1.7.1 Delete or DeleteByQuery nothing works

In Nest 1.7.1 Delete or DeleteByQuery nothing works for me.
I am trying to delete below documents:
Article article1 = new Article()
{
Id = 1111,
Title = "Title - Test Elastic Search",
Summary = "Summary - Test Elastic Search",
Body = "Body - Test Elastic Search",
ArticleDate = _dateToday,
Author = new Author() { Id = 100, Name = "Mikey" },
};
Article article2 = new Article()
{
Id = 2222,
Title = "Title - Test Elastic Search",
Summary = "Summary - Test Elastic Search",
Body = "Body - Test Elastic Search",
ArticleDate = _dateToday,
Author = new Author() { Id = 100, Name = "Mikey" },
Published = true
};
I was expecting below queries would delete single document and all documents in an index but no query is deleting.
_elasticClient.Delete(article).Found;
_elasticClient.DeleteByQuery<Article>(q => q.Query(t => t.Term(m => m.OnField(f => f.Id).Value(articleId))))
.Found;
_elasticClient.DeleteByQuery<Article>(q => q.MatchAll()).IsValid;
Please correct me if i am doing anything wrong.
Here's a working example
void Main()
{
var settings = new ConnectionSettings(new Uri("http://localhost:9200"), "articles");
var client = new ElasticClient(settings);
if (client.IndexExists("articles").Exists)
{
client.DeleteIndex("articles");
}
client.CreateIndex("articles", c => c
.AddMapping<Article>(m => m
.MapFromAttributes()
)
);
var today = DateTime.Now.Date;
var article1 = CreateArticle(1111, today);
var article2 = CreateArticle(2222, today);
var article3 = CreateArticle(3333, today);
var article4 = CreateArticle(4444, today);
var bulkRequest = new BulkDescriptor();
bulkRequest.Index<Article>(i => i.Document(article1));
bulkRequest.Index<Article>(i => i.Document(article2));
bulkRequest.Index<Article>(i => i.Document(article3));
bulkRequest.Index<Article>(i => i.Document(article4));
bulkRequest.Refresh();
client.Bulk(bulkRequest);
var searchResponse = client.Search<Article>(q => q.MatchAll());
Console.WriteLine("Documents from search: {0}. Expect 4", searchResponse.Documents.Count());
client.Delete(article1, d => d.Refresh());
searchResponse = client.Search<Article>(q => q.MatchAll());
Console.WriteLine("Documents from search {0}. Expect 3", searchResponse.Documents.Count());
client.Delete(article2, d => d.Refresh());
searchResponse = client.Search<Article>(q => q.MatchAll());
Console.WriteLine("Documents from search {0}. Expect 2", searchResponse.Documents.Count());
client.DeleteByQuery<Article>(q => q.MatchAll());
searchResponse = client.Search<Article>(q => q.MatchAll());
Console.WriteLine("Documents from search {0}. Expect 0", searchResponse.Documents.Count());
}
private Article CreateArticle(int id, DateTime articleDate)
{
return new Article()
{
Id = id,
Title = "Title - Test Elastic Search",
Summary = "Summary - Test Elastic Search",
Body = "Body - Test Elastic Search",
ArticleDate = articleDate,
Author = new Author() { Id = 100, Name = "Mikey" },
Published = true
};
}
public class Article
{
public int Id { get; set;}
public string Title{ get; set;}
public string Summary { get; set;}
public string Body { get; set;}
public DateTime ArticleDate { get; set; }
public Author Author { get; set; }
public bool Published { get; set;}
}
public class Author
{
public int Id { get; set; }
public string Name { get; set;}
}
results in
Documents from search: 4. Expect 4
Documents from search 3. Expect 3
Documents from search 2. Expect 2
Documents from search 0. Expect 0
as expected.
Something to bear in mind is that Elasticsearch is eventually consistent meaning that a document that is indexed does not appear in search results until after a refresh interval (by default, 1 second); Likewise, with a delete query, a document marked for deletion will appear in search results until the refresh interval has elapsed.
A GET request on a given document with a given id will however return the document before the refresh interval.
If you need documents to be searchable (or to not show in search results after deletion), you can refresh the index after an operation as I did with the bulk and delete calls above, using .Refresh(). You might be tempted to call refresh after every operation, however I would recommend using it only when you really need to as it adds overhead to the cluster and called all the time would likely degrade performance.
I got it working finally.
The delete request which is sent via NEST in fiddler is DELETE /articlestest/article/_query and the query which worked in plugin is this DELETE /articlestest/articles/_query ( the document type name was misspelled in the code).That's the reason ,query was not deleting the documents via NEST.And the bad thing is ,it doesn't even complain about the non-existent document type :( It took me a while to found that issue.

Elasticsearch NEST 2 How to correctly map and use nested classes and bulk index

I have three main questions I need help answering.
How do you correctly map and store a nested map?
How do you search a nested part of a document?
How do you bulk index?
I'm using Nest version 2 and have been looking over the new documentation which can be found Here. The documentation has been useful in creating certain parts of the code but unfortunately doesn't explain how they fit together.
Here is the class I'm trying to map.
[ElasticsearchType(Name = "elasticsearchproduct", IdProperty = "ID")]
public class esProduct
{
public int ID { get; set; }
[Nested]
public List<PriceList> PriceList { get; set; }
}
[ElasticsearchType(Name = "PriceList")]
public class PriceList
{
public int ID { get; set; }
public decimal Price { get; set; }
}
and my mapping code
var node = new Uri(HOST);
var settings = new ConnectionSettings(node).DefaultIndex("my-application");
var client = new ElasticClient(settings);
var map = new CreateIndexDescriptor("my-application")
.Mappings(ms => ms
.Map<esProduct>(m => m
.AutoMap()
.Properties(ps => ps
.Nested<PriceList>(n => n
.Name(c => c.PriceList)
.AutoMap()
)
)
)
);
var response = client.Index(map);
This is the response I get:
Valid NEST response built from a succesful low level call on POST: /my-application/createindexdescriptor
So that seems to work. next index.
foreach (DataRow dr in ProductTest.Tables[0].Rows)
{
int id = Convert.ToInt32(dr["ID"].ToString());
List<PriceList> PriceList = new List<PriceList>();
DataRow[] resultPrice = ProductPriceTest.Tables[0].Select("ID = " + id);
foreach (DataRow drPrice in resultPrice)
{
PriceList.Add(new PriceList
{
ID = Convert.ToInt32(drPrice["ID"].ToString()),
Price = Convert.ToDecimal(drPrice["Price"].ToString())
}
esProduct product = new esProduct
{
ProductDetailID = id,
PriceList = PriceList
};
var updateResponse = client.Update<esProduct>(DocumentPath<esProduct>.Id(id), descriptor => descriptor
.Doc(product)
.RetryOnConflict(3)
.Refresh()
);
var index = client.Index(product);
}
}
Again this seems to work but when I come to search it does seem to work as expected.
var searchResults = client.Search<esProduct>(s => s
.From(0)
.Size(10)
.Query(q => q
.Nested(n => n
.Path(p => p.PriceList)
.Query(qq => qq
.Term(t => t.PriceList.First().Price, 100)
)
)
));
It does return results but I was expecting
.Term(t => t.PriceList.First().Price, 100)
to look move like
.Term(t => t.Price, 100)
and know that is was searching the nested PriceList class, is this not the case?
In the new version 2 documentation I can't find the bulk index section. I tried using this code
var descriptor = new BulkDescriptor();
***Inside foreach loop***
descriptor.Index<esProduct>(op => op
.Document(product)
.Id(id)
);
***Outside foreach loop***
var result = client.Bulk(descriptor);
which does return a success response but when I search I get no results.
Any help would be appreciated.
UPDATE
After a bit more investigation on #Russ advise I think the error must be with my bulk indexing of a class with a nested object.
When I use
var index = client.Index(product);
to index each product I can use
var searchResults = client.Search<esProduct>(s => s
.From(0)
.Size(10)
.Query(q => q
.Nested(n => n
.Path(p => p.PriceList)
.Query(qq => qq
.Term(t => t.PriceList.First().Price, 100)
)
)
)
);
to search and return results, but when I bulk index this no long works but
var searchResults = client.Search<esProduct>(s => s
.From(0)
.Size(10)
.Query(q => q
.Term(t => t.PriceList.First().Price, 100)
)
);
will work, code b doesn't work on the individual index method. Does anyone know why this has happened?
UPDATE 2
From #Russ suggested I have taken a look at the mapping.
the code I'm using to index is
var map = new CreateIndexDescriptor(defaultIndex)
.Mappings(ms => ms
.Map<esProduct>(m => m
.AutoMap()
.Properties(ps => ps
.Nested<PriceList>(n => n
.Name(c => c.PriceList)
.AutoMap()
)
)
)
);
var response = client.Index(map);
Which is posting
http://HOST/fresh-application2/createindexdescriptor {"mappings":{"elasticsearchproduct":{"properties":{"ID":{"type":"integer"},"priceList":{"type":"nested","properties":{"ID":{"type":"integer"},"Price":{"type":"double"}}}}}}}
and on the call to http://HOST/fresh-application2/_all/_mapping?pretty I'm getting
{
"fresh-application2" : {
"mappings" : {
"createindexdescriptor" : {
"properties" : {
"mappings" : {
"properties" : {
"elasticsearchproduct" : {
"properties" : {
"properties" : {
"properties" : {
"priceList" : {
"properties" : {
"properties" : {
"properties" : {
"ID" : {
"properties" : {
"type" : {
"type" : "string"
}
}
},
"Price" : {
"properties" : {
"type" : {
"type" : "string"
}
}
}
}
},
"type" : {
"type" : "string"
}
}
},
"ID" : {
"properties" : {
"type" : {
"type" : "string"
}
}
}
}
}
}
}
}
}
}
}
}
}
}
fresh-application2 returned mapping doesn't mention nested type at all, which I'm guessing is the issue.
The mapping my working nested query looks more like this
{
"my-application2" : {
"mappings" : {
"elasticsearchproduct" : {
"properties" : {
"priceList" : {
"type" : "nested",
"properties" : {
"ID" : {
"type" : "integer"
},
"Price" : {
"type" : "double"
}
}
},
"ID" : {
"type" : "integer"
},
}
}
}
}
}
This has the nested type returned. I think the one which isn't returning nested as a type is when I started using .AutoMap() , am I using it correctly?
UPDATE
I have fixed my mapping problem. I have changed my mapping code to
var responseMap = client.Map<esProduct>(ms => ms
.AutoMap()
.Properties(ps => ps
.Nested<PriceList>(n => n
.Name(c => c.PriceList)
.AutoMap()
)
)
);
Whilst you're developing, I would recommend logging out requests and responses to Elasticsearch so you can see what is being sent when using NEST; this'll make it easier to relate to the main Elasticsearch documentation and also ensure that the body of the requests and responses match your expectations (for example, useful for mappings, queries, etc).
The mappings that you have look fine, although you can forgo the attributes since you are using fluent mapping; there's no harm in having them there but they are largely superfluous (the type name for the esProduct is the only part that will apply) in this case because .Properties() will override inferred or attribute based mapping that is applied from calling .AutoMap().
In your indexing part, you update the esProduct and then immediately after that, index the same document again; I'm not sure what the intention is here but the update call looks superfluous to me; the index call will overwrite the document with the given id in the index straight after the update (and will be visible in search results after the refresh interval). The .RetryOnConflict(3) on the update will use optimistic concurrency control to perform the update (which is effectively a get then index operation on the document inside of the cluster, that will try 3 times if the version of the document changes in between the get and index). If you're replacing the whole document with an update i.e. not a partial update then the retry on conflict is not really necessary (and as per previous note, the update call in your example looks unnecssary altogether since the index call is going to overwrite the document with the given id in the index).
The nested query looks correct; You specify the path to the nested type and then the query to a field on the nested type will also include the path.I'll update the NEST nested query usage documentation to better demonstrate.
The bulk call looks fine; you may want to send documents in batches e.g. bulk index 500 documents at a time, if you need to index a lot of documents. How many to send in one bulk call is going to depend on a number of factors including the document size, how it is analyzed, performance of the cluster, so will need to experiment to get a good bulk size call for your circumstances.
I'd check to make sure that you are hitting the right index, that the index contains the count of documents that you expect and find a document that you know has a PriceList.Price of 100 and see what is indexed for it. It might be quicker to do this using Sense while you're getting up an running.

Nest Elasticsearch- searching front of some fields and back of others

I have a case where I need a partial match on the first part of some properties (last name and first name) and a partial match on the end of some other properties, and I'm wondering how to add both analyzers.
For example, if I have the first name of "elastic", I can currently search for "elas" and find it. But, if I have an account number of abc12345678, I need to search for "5678" and find all account numbers ending in that, but I can't have a first name search for "stic" find "elastic".
Here's a simplified example of my Person class:
public class Person
{
public string AccountNumber { get; set; }
[ElasticProperty(IndexAnalyzer = "partial_name", SearchAnalyzer = "full_name")]
public string LastName { get; set; }
[ElasticProperty(IndexAnalyzer = "partial_name", SearchAnalyzer = "full_name")]
public string FirstName { get; set; }
}
Here's the relevant existing code where I create the index, that currently works great for searching the beginning of a word:
//Set up analyzers on some fields to allow partial, case-insensitive searches.
var partialName = new CustomAnalyzer
{
Filter = new List<string> { "lowercase", "name_ngrams", "standard", "asciifolding" },
Tokenizer = "standard"
};
var fullName = new CustomAnalyzer
{
Filter = new List<string> { "standard", "lowercase", "asciifolding" },
Tokenizer = "standard"
};
var result = client.CreateIndex("persons", c => c
.Analysis(descriptor => descriptor
.TokenFilters(bases => bases.Add("name_ngrams", new EdgeNGramTokenFilter
{
MaxGram = 15, //Allow partial match up to 15 characters.
MinGram = 2, //Allow no smaller than 2 characters match
Side = "front"
}))
.Analyzers(bases => bases
.Add("partial_name", partialName)
.Add("full_name", fullName))
)
.AddMapping<Person>((m => m.MapFromAttributes()))
);
It seems like I could add another EdgeNGramTokenFilter, and make the Side = "back", but I don't want the first and last name searches to match back side searches. Can someone provide a way to do that?
Thanks,
Adrian
Edit
For completeness, this is the new decorator on the property that goes with the code in the accepted answer:
[ElasticProperty(IndexAnalyzer = "partial_back", SearchAnalyzer = "full_name")]
public string AccountNumber { get; set; }
You need to declare another analyzer (let's call it partialBack) specifically for matching from the back but you can definitely reuse the existing edgeNGram token filter, like this:
var partialBack = new CustomAnalyzer
{
Filter = new List<string> { "lowercase", "reverse", "name_ngrams", "reverse" },
Tokenizer = "keyword"
};
...
.Analyzers(bases => bases
.Add("partial_name", partialName)
.Add("partial_back", partialBack))
.Add("full_name", fullName))
)
The key here is the double use of the reverse token filter.
The string (abc12345678) is
first lowercased (abc12345678),
then reversed (87654321cba),
then edge-ngramed (87, 876, 8765, 87654, 876543, ...)
and finally the tokens are reversed again (78, 678, 5678, 45678, 345678, ...).
As you can see, the result is that the string is tokenized "from the back", so that a search for 5678 would match abc12345678.

How to index an object with completion fields

Following http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/search-suggesters-completion.html
How can I index/insert (I can do mapping) an object using Nest client library to be able to provide following options:
"input": ...,
"output": ...,
"payload" : ...,
"weight" : ...
I would like to be able to provide multiple values in 'input' option.
Can't find anyway of doing this using NEST.
Thank you
NEST provides the SuggestField type in order to assist in indexing completion suggestions. You don't necessarily need to use this type, you can provide your own that contains the expected completion fields (input, output, etc...), but the purpose of SuggestField is to make the whole process easier by already providing a baked in type.
Usage:
Add a suggest field to the document/type you are indexing:
public class MyType
{
public SuggestField Suggest { get; set; }
}
Your mapping should look something like:
client.Map<MyType>(m => m
.Properties(ps => ps
.Completion(c => c.Name(x => x.Suggest).Payloads(true))
)
);
Indexing example:
var myType = new MyType
{
Suggest = new SuggestField
{
Input = new [] { "Nevermind", "Nirvana" },
Output = "Nirvana - Nevermind",
Payload = new { id = 1234 },
Weight = 34
}
};
client.Index<MyType>(myType);
Hope that helps.

Resources