I am trying to retrieve a single document with a secific name (exactly that name) using NEST 7.5.1 (.NET Core 3.1):
var queryByTerm = client.Search<SomeDto>(s =>s.Query(q => q.Term(p => p.NameField, "example name")));
But it does not return any documents (the call succeeds).
The actual query being sent (as seen in DebugInformation with .EnableDebugMode on client's ConnectionSettings):
{"query":{"term":{"nameField":{"value":"example name"}}}}
But it only works (in Kibana) when I add .keyword fo the nameField:
{"query":{"term":{"nameField.keyword":{"value":"example name"}}}}
Do I somehow have to force NEST to use nameField.keyword instead of nameField?
You can do this with .Suffix() extension method. Docs.
var queryByTerm = client.Search<SomeDto>(s =>s.Query(q => q.Term(p => p.NameField.Suffix("keyword"), "example name")));
Hope that helps.
I'm quite new to ElasticSearch. I'm trying to reindex a index in order to rename it. I'm using NEST API v5.4.
I saw this example:
var reindex =
elasticClient.Reindex<Customer>(r =>
r.FromIndex("customers-v1")
.ToIndex("customers-v2")
.Query(q => q.MatchAll())
.Scroll("10s")
.CreateIndex(i =>
i.AddMapping<Customer>(m =>
m.Properties(p =>
p.String(n => n.Name(name => name.Zipcode).Index(FieldIndexOption.not_analyzed))))));
Source: http://thomasardal.com/elasticsearch-migrations-with-c-and-nest/
However, I can't reproduce this using NEST 5.4. I think that is to version 2.4.
I check the breaking changes of ElasticSearch and try reindexing using this:
Source: https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/nest-breaking-changes.html
public method Nest.ReindexDescriptor..ctor Declaration changed (Breaking)
2.x: public .ctor(IndexName from, IndexName to) 5.x: public .ctor()
var reindex = new client.Reindex(oldIndexName, newIndexName);
But this did not work too.
I also search for documentation but i didn't find any code on c#, just JSON
Source: https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-reindex.html)
Can someone give me a example how to reindex using NEST 5.4 on C#?
Thanks in advance! :slight_smile:
After search for 2 long days I found out the solution to reindex a index. In order to solve future problems, I'll provide my solution.
Nest Version - 5.4
var reindex = client.Reindex<object>(r => r
.BackPressureFactor(10)
// ScrollAll - Scroll all the documents of the index and store it for 1minute
.ScrollAll("1m", 2, s => s
.Search(ss => ss
.Index(oldIndexName)
.AllTypes())
// there needs to be some degree of parallelism for this to work
.MaxDegreeOfParallelism(4))
.CreateIndex(c => c
// New index here
.Index(newIndexName)
.Settings(
// settings goes here)
.Mappings(
// mappings goes here))
.BulkAll(b => b
// New index here!
.Index(newIndexName)
.Size(100)
.MaxDegreeOfParallelism(2)
.RefreshOnCompleted()));
the ReIndex method returns a cold IObservable on which you have to call .Subscribe() to kick off everything.
So, you need to add it to your code:
var o = new ReindexObserver(
onError: (e) => { //do something },
onCompleted: () => { //do something });
reindex.Subscribe(o);
Useful links to check this are:
Documentation
Issue 2660 on GitHub
Issue 2771 on GitHub
Elasticsearch released their new Reindex API in Elasticsearch 2.3.0, does the current version of NEST (2.1.1) make use of this api yet? If not, are there plans to do so?
I am aware that the current version has a reindex method, but it forces you to create the new index. For my use case, the index already exists.
Any feedback/insights will be greately appricated. Thnx!
This kind of question is best asked on the github issues for NEST since the committers on the project will be able to best answer :)
A commit went in on 6 April to map the new Reindex API available in Elasticsearch 2.3.0, along with other features like the Task Management API and Update By Query. This made its way into NEST 2.3.0
NEST 2.x already contains a helper for doing reindexing that uses scan/scroll under the covers and returns an IObservable<IReindexResponse<T>> that can be used to observe progress
public class Document {}
var observable = client.Reindex<Document>("from-index", "to-index", r => r
// settings to use when creating to-index
.CreateIndex(c => c
.Settings(s => s
.NumberOfShards(5)
.NumberOfReplicas(2)
)
)
// query to optionally limit documents re-indexed from from-index to to-index
.Query(q => q.MatchAll())
// the number of documents to reindex in each request.
// NOTE: The number of documents in each request will actually be
// NUMBER * NUMBER OF SHARDS IN from-index
// since reindex uses scan/scroll
.Size(100)
);
ExceptionDispatchInfo e = null;
var waitHandle = new ManualResetEvent(false);
var observer = new ReindexObserver<Document>(
onNext: reindexResponse =>
{
// do something with notification. Maybe log total progress
},
onError: exception =>
{
e = ExceptionDispatchInfo.Capture(exception);
waitHandle.Set();
},
completed: () =>
{
// Maybe log completion, refresh the index, etc..
waitHandle.Set();
}
);
observable.Subscribe(observer);
// wait for the handle to be signalled
waitHandle.Wait();
// throw the exception if one was captured
e?.Throw();
Take a look at the ReIndex API tests for some ideas.
The new Reindex API is named client.ReIndexOnServer() in the client to differentiate it from the existing observable implementation.
Is it possible to get the raw search query from the NEST client?
var result = client.Search<SomeType>(s => s
.AllIndices()
.Type("SomeIndex")
.Query(query => query
.Bool(boolQuery => BooleanQuery(searchRequest, mustMatchQueries)))
);
I'd really like to debug why I am getting certain results.
The methods to do this seem to change with each major version, hence the confusing number of answers. If you want this to work in NEST 6.x, AND you want to see the deserialized request BEFORE it's actually sent, it's fairly easy:
var json = elasticClient.RequestResponseSerializer.SerializeToString(request);
If you're debugging in Visual Studio, it's handy to put a breakpoint right after this line, and when you hit it, hover over the json variable above and hit the magnifying glass thingy. You'll get a nice formatted view of the JSON.
You can get raw query json from RequestInformation:
var rawQuery = Encoding.UTF8.GetString(result.RequestInformation.Request);
Or enable trace on your ConnectionSettings object, so NEST will print every request to trace output
var connectionSettings = new ConnectionSettings(new Uri(elasticsearchUrl));
connectionSettings.EnableTrace(true);
var client = new ElasticClient(connectionSettings);
NEST 7.x
Enable debug mode when creating settings for a client:
var settings = new ConnectionSettings(connectionPool)
.DefaultIndex("index_name")
.EnableDebugMode()
var client = new ElasticClient(settings);
then your response.DebugInformation will contain information about request sent to elasticsearch and response from elasticsearch. Docs.
For NEST / Elasticsearch.NET v6.0.2, use the ApiCall property of the IResponse object. You can write a handy extension method like this:
public static string ToJson(this IResponse response)
{
return Encoding.UTF8.GetString(response.ApiCall.RequestBodyInBytes);
}
Or, if you want to log all requests made to Elastic, you can intercept responses with the connection object:
var node = new Uri("https://localhost:9200");
var pool = new SingleNodeConnectionPool(node);
var connectionSettings = new ConnectionSettings(pool, new HttpConnection());
connectionSettings.OnRequestCompleted(call =>
{
Debug.Write(Encoding.UTF8.GetString(call.RequestBodyInBytes));
});
In ElasticSearch 5.x, the RequestInformation.Request property does not exist in ISearchResponse<T>, but similar to the answer provided here you can generate the raw query JSON using the Elastic Client Serializer and a SearchDescriptor. For example, for the given NEST search query:
var results = elasticClient.Search<User>(s => s
.Index("user")
.Query(q => q
.Exists(e => e
.Field("location")
)
)
);
You can get the raw query JSON as follows:
SearchDescriptor<User> debugQuery = new SearchDescriptor<User>()
.Index("user")
.Query(q => q
.Exists(e => e
.Field("location")
)
)
;
using (MemoryStream mStream = new MemoryStream())
{
elasticClient.Serializer.Serialize(debugQuery, mStream);
string rawQueryText = Encoding.ASCII.GetString(mStream.ToArray());
}
Before making Request, from Nest Query - For Nest 5.3.0 :
var stream = new System.IO.MemoryStream();
elasticClient.Serializer.Serialize(query, stream );
var jsonQuery = System.Text.Encoding.UTF8.GetString(stream.ToArray());
Edit: It's changed from from Nest 6.x, and you can do below:
var json = elasticClient.RequestResponseSerializer.SerializeToString(request);
on nest version 6 use
connextionString.DisableDirectStreaming();
then on response.DebugInformation you can see all information.
Use result.ConnectionStatus.Request.
When using NEST 7 and you don't want to enable debug mode.
public static string GetQuery<T>(this IElasticClient client, SearchDescriptor<T> searchDescriptor) where T : class
{
using (System.IO.MemoryStream ms = new System.IO.MemoryStream())
{
client.RequestResponseSerializer.Serialize(searchDescriptor, ms);
return Encoding.UTF8.GetString(ms.ToArray());
}
}
While it's possible to get raw request/response through code, I find it much easier to analyze it with fiddler.
The reason is that I can easily analyze raw request, response, headers, Full URL, execution time - all together without any hassle of code changes.
Here's some reference links in case someone unfamiliar with fiddler wants to check details:
#1 https://www.elastic.co/guide/en/elasticsearch/client/net-api/current/logging-with-fiddler.html
#2 NEST 1.0: See request on Fiddler
#3 https://newbedev.com/how-to-get-nest-to-work-with-proxy-like-fiddler
How about using Fiddler ?! :)
I'm doing a small research about a client for elastic search in .net and I found that NEST is one of the most supported solutions for this matter.
I was looking at Nest's docummentation and I couldn´t find a way to output a raw json from a query and avoid the serialization into an object, because I'm using angularJs in the front end I don´t want to overload the process of sending the information to the client with some unnecessary steps.
......and also I'd like to know how can I overrdide the serialization process?
I found that NEST uses Json.NET which I would like to change for the servicestack json serielizer.
thanks!
Hi Pedro you can do this with NEST
var searchDescriptor = new SearchDescriptor<ElasticSearchProject>()
.Query(q=>q.MatchAll());
var request = this._client.Serializer.Serialize(searchDescriptor);
ConnectionStatus result = this._client.Raw.SearchPost(request);
Assert.NotNull(result);
Assert.True(result.Success);
Assert.IsNotEmpty(result.Result);
This allows you to strongly type your queries, but return the string .Result which is the raw response from elasticsearch as string to your
request can be an object or the string so if you are OK with the internal json serialize just pass searchDescriptor directly
Use RequestResponseSerializer instead of Serializer.
var searchDescriptor = ...;
...
byte[] b = new byte[60000];
using (MemoryStream ms = new MemoryStream(b))
{
this._client.RequestResponseSerializer.Serialize(searchDescriptor , ms);
}
var rawJson = System.Text.Encoding.Default.GetString(b);