Nest ElasticClient with multiple indexes to index a document - elasticsearch

At first I had 1 index and my elasticclient was setup like below in my startup.cs
public static IServiceCollection AddElasticClient(this IServiceCollection services)
{
var elasticSettings = services.BuildServiceProvider().GetService<IOptions<ElasticSettings>>().Value;
var settings = new ConnectionSettings(new Uri(elasticSettings.Uri));
settings
.ThrowExceptions(elasticSettings.ThrowExceptions)
.PrettyJson(elasticSettings.PrettyJson)
.DefaultIndex(elasticSettings.Index)
.BasicAuthentication(elasticSettings.Username, elasticSettings.Password)
.DefaultMappingFor<CorrelationContext>(ms => ms.Ignore(p => p.DgpHeader));
var client = new ElasticClient(settings);
services.AddSingleton<IElasticClient>(client);
return services;
}
My writer looks like
public class ElasticWriter : IElasticWriter
{
private readonly IElasticClient _elasticClient;
public ElasticWriter(IElasticClient elasticClient)
{
_elasticClient = elasticClient ?? throw new ArgumentNullException(nameof(elasticClient));
}
public void Write(AuditElasticDoc doc)
{
var indexResponse = _elasticClient.IndexDocument(doc);
if (!indexResponse.IsValid)
{
throw indexResponse.OriginalException ?? new Exception("Invalid Elastic response when writing document.");
}
}
}
Now there is a new requirement by which they can provide the name of the index to write to.
All authentication data of the different indexes are provided through config settings, so I have everything available at startup.
The document type is always the same.
I found examples of specifying the index when querying but not when indexing.
Can I provide multiple indexes in my ElasticClient and specify the index when executing the IndexDocument?
Or do I need a separate client for each index?
If the latter, is there a way I can still use DI to inject the client in my writer or do I have to create one there at the spot?
Thx.
I'm using Nest 7.6.1

Instead of using IndexDocument, you can use IndexAsync method which will allow you to control additional request parameters
var indexResponse = await _elasticClient.IndexAsync(doc, descriptor => descriptor.Index("other"));
IndexDocument is a wrapper method, hiding the complexity of indexing documents from the clients. Have a look.
Request auth configuration
var indexResponse = await _elasticClient.IndexAsync(doc,
descriptor => descriptor
.Index("other")
.RequestConfiguration(rq => rq.BasicAuthentication("user", "pass")));

Related

Mock Elastic Search response in.Net

I have Elastic Search Nest library code and need to mock the response i am getting from elastic search index.
var obj = service.Search<TestDocument>(new student().Query());
var Name= obj.Aggs.Terms("Name");
For Testing :
I am creating the Nest object after doing quick watch but facing issue -Aggregations - is a internal protected property and i am not able to set this value.
new Nest.KeyedBucket<object>
{
Key="XYZ school",
KeyAsString=null,
Aggregations=new Dictionary<string, IAggregationContainer>{}
}
Please suggest solution or any other approach i can use to mock elastic search nest object .
If you really want to stub the response from the client, you could do something like the following with Moq
var client = new Mock<IElasticClient>();
var searchResponse = new Mock<ISearchResponse<object>>();
var aggregations = new AggregateDictionary(new Dictionary<string, IAggregate> {
["Name"] = new BucketAggregate
{
Items = new List<KeyedBucket<object>>
{
new Nest.KeyedBucket<object>(new Dictionary<string, IAggregate>())
{
Key = "XYZ school",
KeyAsString = null,
DocCount = 5
}
}.AsReadOnly()
}
});
searchResponse.Setup(s => s.Aggregations).Returns(aggregations);
client.Setup(c => c.Search<object>(It.IsAny<Func<SearchDescriptor<object>, ISearchRequest>>()))
.Returns(searchResponse.Object);
var response = client.Object.Search<object>(s => s);
var terms = response.Aggregations.Terms("Name");
Another way would be to use the InMemoryConnection and return known JSON in response to a request..
For testing purposes however, it may be better to have an instance of Elasticsearch running, and perform integration tests against it. Take a look at Elastic.Xunit which provides an easy way to spin up an Elasticsearch cluster for testing purposes. This is used by the client in integration tests.
You can get Elastic.Xunit from the Appveyor feed.

"Clone" index mappings

I have an index which I will be reindexing. At the moment I want to create a new index, which should contain the exact same mappings that can be found in the original index.
I've got this:
var srcMappings = client.GetMapping(new GetMappingRequest((Indices)sourceIndexName)).Mappings;
And I try to create an index:
var response = client.CreateIndex(destinationIndex, c => c
.Settings(...my settings ...)
.Mappings(... what here? ...)
);
What exactly should I pass to the .Mappings(...) above so that the mappings from the source index are replicated into the target index? I don't want to explicitly 'know' about the types.
I am trying to use Nest.
Alternatively, is there a Reindex API which would take the destination index name and create the index for me, together with the mappings of the source?
You can get the mappings from one index and use them to create the mappings in another index with
var client = new ElasticClient();
var getIndexResponse = client.GetIndex("assignments");
var createIndexResponse = client.CreateIndex("assignments2", c => c
.Mappings(m => Promise.Create(getIndexResponse.Indices["assignments"].Mappings))
);
You'll need an IPromise<T> implementation to do so
public class Promise
{
public static IPromise<TValue> Create<TValue>(TValue value) where TValue : class =>
new Promise<TValue>(value);
}
public class Promise<T> : IPromise<T> where T : class
{
public T Value { get; }
public Promise(T value) => Value = value;
}
The Promise is needed in some places in NEST's fluent API implementation where values are additive and a final value needs to be returned at a later point.
You can also do the same using the object initializer syntax and no Promise<T>
var createIndexResponse = client.CreateIndex(new CreateIndexRequest("assignments2")
{
Mappings = getIndexResponse.Indices["assignments"].Mappings
});
Alternatively, is there a Reindex API which would take the destination index name and create the index for me, together with the mappings of the source?
There are two Reindex APIs within NEST; an Observable implementation that has been around since NEST 1.x, and the Reindex API as available within Elasticsearch since 2.3 (known as ReindexOnServer in NEST). The former Observable implementation can create the destination index for you, although it will copy all settings, mappings and aliases. The latter Reindex API does not create the destination index as part of the operation, so it needs to be set up before starting the reindex process.

How to use ES Java API to create a new type of an index

I have succeed create an index use Client , the code like this :
public static boolean addIndex(Client client,String index) throws Exception {
if(client == null){
client = getSettingClient();
}
CreateIndexRequestBuilder requestBuilder = client.admin().indices().prepareCreate(index);
CreateIndexResponse response = requestBuilder.execute().actionGet();
return response.isAcknowledged();
//client.close();
}
public static boolean addIndexType(Client client, String index, String type) throws Exception {
if (client == null) {
client = getSettingClient();
}
TypesExistsAction action = TypesExistsAction.INSTANCE;
TypesExistsRequestBuilder requestBuilder = new TypesExistsRequestBuilder(client, action, index);
requestBuilder.setTypes(type);
TypesExistsResponse response = requestBuilder.get();
return response.isExists();
}
however, the method of addIndexType is not effected, the type is not create .
I don't know how to create type ?
You can create types when you create the index by providing a proper mapping configuration. Alternatively a type gets created when you index a document of a certain type. However the first suggestion is the better one, because then you can control the full mapping of that type instead of relying on dynamic mapping.
You can set types in the following way:
// JSON schema is the JSON mapping which you want to give for the index.
JSONObject builder = new JSONObject().put(type, JSONSchema);
// And then just fire the below command
client.admin().indices().preparePutMapping(indexName)
.setType(type)
.setSource(builder.toString(), XContentType.JSON)
.execute()
.actionGet();

Bulk Update on ElasticSearch using NEST

I am trying to replacing the documents on ES using NEST. I am seeing the following options are available.
Option #1:
var documents = new List<dynamic>();
`var blkOperations = documents.Select(doc => new BulkIndexOperation<T>`(doc)).Cast<IBulkOperation>().ToList();
var blkRequest = new BulkRequest()
{
Refresh = true,
Index = indexName,
Type = typeName,
Consistency = Consistency.One,
Operations = blkOperations
};
var response1 = _client.Raw.BulkAsync<T>(blkRequest);
Option #2:
var descriptor = new BulkDescriptor();
foreach (var eachDoc in document)
{
var doc = eachDoc;
descriptor.Index<T>(i => i
.Index(indexName)
.Type(typeName)
.Document(doc));
}
var response = await _client.Raw.BulkAsync<T>(descriptor);
So can anyone tell me which one is better or any other option to do bulk updates or deletes using NEST?
You are passing the bulk request to the ElasticsearchClient i.e. ElasticClient.Raw, when you should be passing it to ElasticClient.BulkAsync() or ElasticClient.Bulk() which can accept a bulk request type.
Using BulkRequest or BulkDescriptor are two different approaches that are offered by NEST for writing queries; the former uses an Object Initializer Syntax for building up a request object while the latter is used within the Fluent API to build a request using lambda expressions.
In your example, BulkDescriptor is used outside of the context of the fluent API, but both BulkRequest and BulkDescriptor implement IBulkRequest so can be passed to ElasticClient.Bulk(IBulkRequest).
As for which to use, in this case it doesn't matter so whichever you prefer.

PrepareResponse().AsActionResult() throws unsupported exception DotNetOpenAuth CTP

Currently I'm developing an OAuth2 authorization server using DotNetOpenAuth CTP version. My authorization server is in asp.net MVC3, and it's based on the sample provided by the library. Everything works fine until the app reaches the point where the user authorizes the consumer client.
There's an action inside my OAuth controller which takes care of the authorization process, and is very similar to the equivalent action in the sample:
[Authorize, HttpPost, ValidateAntiForgeryToken]
public ActionResult AuthorizeResponse(bool isApproved)
{
var pendingRequest = this.authorizationServer.ReadAuthorizationRequest();
if (pendingRequest == null)
{
throw new HttpException((int)HttpStatusCode.BadRequest, "Missing authorization request.");
}
IDirectedProtocolMessage response;
if (isApproved)
{
var client = MvcApplication.DataContext.Clients.First(c => c.ClientIdentifier == pendingRequest.ClientIdentifier);
client.ClientAuthorizations.Add(
new ClientAuthorization
{
Scope = OAuthUtilities.JoinScopes(pendingRequest.Scope),
User = MvcApplication.LoggedInUser,
CreatedOn = DateTime.UtcNow,
});
MvcApplication.DataContext.SaveChanges();
response = this.authorizationServer.PrepareApproveAuthorizationRequest(pendingRequest, User.Identity.Name);
}
else
{
response = this.authorizationServer.PrepareRejectAuthorizationRequest(pendingRequest);
}
return this.authorizationServer.Channel.PrepareResponse(response).AsActionResult();
}
Everytime the program reaches this line:
this.authorizationServer.Channel.PrepareResponse(response).AsActionResult();
The system throws an exception which I have researched with no success. The exception is the following:
Only parameterless constructors and initializers are supported in LINQ to Entities.
The stack trace: http://pastebin.com/TibCax2t
The only thing I've done differently from the sample is that I used entity framework's code first approach, an I think the sample was done using a designer which autogenerated the entities.
Thank you in advance.
If you started from the example, the problem Andrew is talking about stays in DatabaseKeyNonceStore.cs. The exception is raised by one on these two methods:
public CryptoKey GetKey(string bucket, string handle) {
// It is critical that this lookup be case-sensitive, which can only be configured at the database.
var matches = from key in MvcApplication.DataContext.SymmetricCryptoKeys
where key.Bucket == bucket && key.Handle == handle
select new CryptoKey(key.Secret, key.ExpiresUtc.AsUtc());
return matches.FirstOrDefault();
}
public IEnumerable<KeyValuePair<string, CryptoKey>> GetKeys(string bucket) {
return from key in MvcApplication.DataContext.SymmetricCryptoKeys
where key.Bucket == bucket
orderby key.ExpiresUtc descending
select new KeyValuePair<string, CryptoKey>(key.Handle, new CryptoKey(key.Secret, key.ExpiresUtc.AsUtc()));
}
I've resolved moving initializations outside of the query:
public CryptoKey GetKey(string bucket, string handle) {
// It is critical that this lookup be case-sensitive, which can only be configured at the database.
var matches = from key in db.SymmetricCryptoKeys
where key.Bucket == bucket && key.Handle == handle
select key;
var match = matches.FirstOrDefault();
CryptoKey ck = new CryptoKey(match.Secret, match.ExpiresUtc.AsUtc());
return ck;
}
public IEnumerable<KeyValuePair<string, CryptoKey>> GetKeys(string bucket) {
var matches = from key in db.SymmetricCryptoKeys
where key.Bucket == bucket
orderby key.ExpiresUtc descending
select key;
List<KeyValuePair<string, CryptoKey>> en = new List<KeyValuePair<string, CryptoKey>>();
foreach (var key in matches)
en.Add(new KeyValuePair<string, CryptoKey>(key.Handle, new CryptoKey(key.Secret, key.ExpiresUtc.AsUtc())));
return en.AsEnumerable<KeyValuePair<string,CryptoKey>>();
}
I'm not sure that this is the best way, but it works!
It looks like your ICryptoKeyStore implementation may be attempting to store CryptoKey directly, but it's not a class that is compatible with the Entity framework (due to not have a public default constructor). Instead, define your own entity class for storing the data in CryptoKey and your ICryptoKeyStore is responsible to transition between the two data types for persistence and retrieval.

Resources