I have written some code using the Elasticsearch.Net & NEST client library that should index a document without using a POCO for mapping fields as I have many different documents.
Question 1) Is this correct way to do create an index, does the .AddMapping<string>(mapping => mapping.Dynamic(true)) create the mapping based on the document passed in?
var newIndex = client.CreateIndex(indexName, index => index
.NumberOfReplicas(replicas)
.NumberOfShards(shards)
.Settings(settings => settings
.Add("merge.policy.merge_factor", "10")
.Add("search.slowlog.threshold.fetch.warn", "1s")
)
.AddMapping<string>(mapping => mapping.Dynamic(true))
);
Question 2) Is this possible?
string document = "{\"name\": \"Mike\"}";
var newIndex = client.Index(document, indexSelector => indexSelector
.Index(indexName)
);
When I run code in "Question 2" it returns:
{"Unable to perform request: 'POST ' on any of the nodes after retrying 0 times."}
NEST only deals with typed objects in this case passing a string will cause it to index the document into /{indexName}/string/{id}.
Since it can't infer an id from string and you do not pass it one it will fail on that or on the fact that it can't serialize a string. I'll update the client to throw a better exception in this case.
If you want to index a document as string use the exposed Elasticsearch.NET client like so:
client.Raw.Index(indexName, typeName, id, stringJson);
If you want elasticsearch to come up with an id you can use
client.Raw.Index(indexName, type, stringJson);
client is the NESTclient and the Raw property is an Elasticsearch.Net client with the same connectionsettings.
Please note that I might rename Raw with LowLevel in the next beta update, still debating that.
Related
I'm searching a v5.5 elasticsearch cluster that's hosted using AWS's managed solution. I'm using a client to send search requests to the cluster but it's not finding any hits. I switched on the cluster lever logging and can see that the problem is that the type being searched is in lowercase (when the document types in the index are uppercase) so it can't match on any documents.
I'm passing a search descriptor object into the Nest client:
GetSearchDescriptor(SearchDescriptor<T> descriptor)
{
descriptor.Index(index)
.Type(documentType)
.Query(q => (q
.Bool(bq => bq
...
}
client.Search<T>(s => GetSearchDescriptor(s))
Where documentType is of type T e.g. Invoice.
When I look at the CloudWatch logs I can see that the request that's hitting the cluster is lowercase invoice (instead of uppercase Invoice). This doesn't match the document type hence doesn't show any results. When I do a kibana search using the exact json in the logs I get the correct results when using uppercase Invoice, but no results when using lowercase invoice.
Any ideas what could be going on here?
NEST's default inference for a type name from a POCO of type T, is to lowercase the typeof(T).Name value. You can easily change this behaviour for T or for all POCOs on ConnectionSettings
For all POCOs
var settings = new ConnectionSettings(new Uri("http://localhost:9200"))
.DefaultTypeNameInferrer(type => type.Name.ToUpperInvariant());
var client = new ElasticClient(settings);
For only T e.g. Invoice
var settings = new ConnectionSettings(new Uri("http://localhost:9200"))
.InferMappingFor<Invoice>(m => m
.TypeName("INVOICE")
);
var client = new ElasticClient(settings);
With the latter, you can also use InferMappingFor<T> to specify a default index name for T, and tell the client which property should be used to infer the id for the document.
Background
I'm using ElasticSearch as the search engine for a new ASP.Net Core 2.1 website I'm working on. I'm using the Nest API to integrate with it. I want to use the X.PagedList to handle the paging for me.
I've used this in other ASP.Net Core projects and it's worked well querying data in MS SQL Server.
Code
ISearchResponse<Foo> searchResponse =
_elasticSearchClient.Search<Foo>(s => s
.Query(q => q
.Bool(b => b.Filter(distanceFilters))
)
.Source(src => src
.Includes(i => i
.Fields(
f => f.Field1,
f => f.Field2,
f => f.Field3
)
)
)
.From(options.From)
.Size(options.Size)
);
var hitsMD = searchResponse.HitsMetadata;
var results = hitsMD?.Hits.Select(s => new Hit()
{
Index = s.Index,
Id = s.Id,
Score = s.Score,
Job = s.Source
}
).ToPagedList(PageNumber, PageSize);
Issue
When I call .ToPagedList on the search results returned by ElasticSearch, it only shows one page of results.
The issue is that ElasticSearch has its own paging mechanism so it's only returning one page of hits.
I had the idea that because ElasticSearch passes back the total number of hits I could tell the PagedList how many items are in the list by setting the PagedList.TotalItemCount property. However, I can't do this as it's a private set.
I've tried removing the from and size but this returns 10 hits which is ElasticSearch's default size which they obviously put in place for performance reasons.
Question
How can I make use of the X.PagedList package whilst integrating into ElasticSearch using the Nest API?
You've basically got all the pieces here already. All you're missing is StaticPagedList<T>. Since paging is already being handled by Elasticsearch, you need to simply define a static paging setup, i.e.:
var pagedResults = new StaticPagedList<Foo>(results, PageNumber, PageSize, total);
I am using elastic search for the first time and based on requirements i have some doubts and questions for scroll
To retrieve all data which are fulfilling all search criteria
1)I am trying to use scroll but i found while searching about it
https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking_21_search_changes.html
i found Search type scan is deprecated
but NEST is supporting it
so should i use "search type scan" or "sort by doc"? (I am using elastic search 2.4)
2)Can i use "sorting on any field" when using scrolling?
3)while doing clear scroll
var test2 = client.ClearScroll(x=>x.ScrollId(results.ScrollId));
Getting error as below:
Invalid NEST response built from a unsuccessful low level call on DELETE: /_search/scroll
Audit trail of this API call:
[1] BadResponse: Node: http://mydomain#localhost:9200/ Took: 00:00:00.0160110
OriginalException: System.Net.WebException: The remote server returned an error: (404) Not Found.
at System.Net.HttpWebRequest.GetResponse()
at Elasticsearch.Net.HttpConnection.Request[TReturn](RequestData requestData) in C:\Users\russ\source\elasticsearch-net-2.x\src\Elasticsearch.Net\Connection\HttpConnection.cs:line 141
Request:
{"scroll_id":["c2NhbjswOzE7dG90YWxfaGl0czoxMjs="]}
Response:
{}
so is it correct way of clearing scroll or not?
Update: : below is my code :
List<Object> indexedList = new List<Object>();
ISearchResponse<ListingSearch> listingResult =
client.Search<ListingSearch>(search => search
.Index(Constant.ES_INDEX)
.Type(Constant.ES_TYPE)
.From(listingSearch.StartIndex)
.Size(10)
.Source(s => s.Include(i => i.Fields(outpputFields)))
.Query(query => query.
Bool(boolean => boolean.
Must(
must => must.Term(t => t.Field("is_deleted").Value(false))
)
.Sort(x => x.Field("_doc", SortOrder.Ascending))
.Scroll("60s")
);
List<Object> indexedList = new List<Object>();
var results = client.Scroll<ListingSearch>("60s", listingResult.ScrollId);
while (results.Documents.Any())
{
foreach (var doc in results.Hits)
{
indexedList.Add(doc);
}
results = client.Scroll<ListingSearch>("60s", results.ScrollId);
}
var test2 = client.ClearScroll(x=>x.ScrollId(results.ScrollId));
//Clear Scroll
With above code I am getting data
but if i change size from 10 to 1000, getting no records.
Not sure if issue is the amount of data because my ES db has only 12-15 documents.
NEST 2.x versions have SearchType.Scan because NEST 2.x versions are compatible with all Elasticsearch 2.x versions, so the search type needs to exist when using NEST 2.x against Elasticsearch 2.0. Sending the search type through in later versions won't have any effect.
The most efficient way of retrieving documents with scroll is sorting by _doc but you can specify any sort parameters when scrolling.
When using the scroll API, you should use the scroll_id from the previous request in the next scroll call to fetch the next set of results. Once you have finished with a scroll, it is a good idea to clear it by calling ClearScroll() as you are doing. Your call looks correct; perhaps the scroll_id has already expired at the point you make the clear call?
I want to check the discrepancies between my current mapping (as in my C# code) and the mapping in the elasticsearch index.
With only:
var res = esClient.GetMapping<EsCompany>();
I get GetMappingResponse object in c#, I will have to compare field by field for equality. Even worse, each field has their own properties, I have to descend into those properties for further comparison.
In my application, I prefer obtaining the raw json of the mapping, and I can easily diff two json objects for equality.
I then tried this:
var res = esClient.Raw.IndicesGetMapping(myIndexName);
But when I read res.Response, I get an AmbiguousMatchException exception.
When you connect to Elasticsearch you can choose to expose the raw response like this:
var client = new ElasticClient(new ConnectionSettings().ExposeRawResponse());
Then you should be able to access the raw json via:
var json = res.ConnectionStatus.ResponseRaw;
I am integrating FOSElasticaBundle in my Symfony 2.3 project and I need to sort the results by their price property.
Here is my code:
$finder = $this->container->get('fos_elastica.finder.website.product');
$fieldTerms = new \Elastica\Query\Terms();
$fieldTerms->setTerms('taxon_ids', $taxon_ids_array);
$boolQuery->addMust($fieldTerms);
$resultSet = $finder->find($boolQuery);
How I can do this?
Thanks
Try create a \Elastica\Query object which also contains the sorting information, then send this to the finder:
$finder = $this->container->get('fos_elastica.finder.website.product');
$fieldTerms = new \Elastica\Query\Terms();
$fieldTerms->setTerms('taxon_ids', $taxon_ids_array);
$boolQuery->addMust($fieldTerms);
$finalQuery = new \Elastica\Query($boolQuery);
$finalQuery->setSort(array('price' => array('order' => 'asc')));
$resultSet = $finder->find($finalQuery);
Have a look at the elasticsearch docs on the sort parameter to see how to use it properly.
NOTE: \Elastica\Query is quite different to \Elastica\Query\AbstractQuery, the first encapsulates everything you could send to the _search API endpoint (facets, sorting, explain, etc...) The AbstractQuery represents a base type for each of the individual query types (range, fuzzy, terms, etc...).