Elastic Search Custom Create Index using Java High Level Rest Client - spring-boot

createIndexWithCustomMappings(String indexName, String fieldsMapping){CreateIndexResponse createIndexResponse = client.admin().indices()
.prepareCreate(index).setSettings(fieldsMapping).execute().get();}
I have a code which creates the index in elastic search in a spring boot application. Currently the client used is transport client which is now depreciated as per elastic search documentation and now is replaced by High Level Rest Client.
For Creating Index using High Level Rest Client. I have seen this code.
CreateIndexRequest request = new CreateIndexRequest(indexName);
CreateIndexResponse createIndexResponse = client.indices().create(request, RequestOptions.DEFAULT);
Here fieldsMapping is a json file which has details regarding analyzer, tokenizer, filter and is passed as String to this method. I am not able to find methods in java rest high level client to incorporate setSettings(fieldsMapping).execute().get() as done above with transport client.
Any Idea on how this setSettings(fieldMappings) can work java high level rest client

You can use the implementation from the ElasticsearchRestTemplate itself.
Using Elasticsearch 6.x:
This is how you create the index with settings:
#Override
public boolean createIndex(String indexName, Object settings) {
CreateIndexRequest request = new CreateIndexRequest(indexName);
if (settings instanceof String) {
request.settings(String.valueOf(settings), Requests.INDEX_CONTENT_TYPE);
} else if (settings instanceof Map) {
request.settings((Map) settings);
} else if (settings instanceof XContentBuilder) {
request.settings((XContentBuilder) settings);
}
try {
return client.indices().create(request, RequestOptions.DEFAULT).isAcknowledged();
} catch (IOException e) {
throw new ElasticsearchException("Error for creating index: " + request.toString(), e);
}
}
This is how you update the mappings for the index:
#Override
public boolean putMapping(String indexName, String type, Object mapping) {
Assert.notNull(indexName, "No index defined for putMapping()");
Assert.notNull(type, "No type defined for putMapping()");
PutMappingRequest request = new PutMappingRequest(indexName).type(type);
if (mapping instanceof String) {
request.source(String.valueOf(mapping), XContentType.JSON);
} else if (mapping instanceof Map) {
request.source((Map) mapping);
} else if (mapping instanceof XContentBuilder) {
request.source((XContentBuilder) mapping);
}
try {
return client.indices().putMapping(request, RequestOptions.DEFAULT).isAcknowledged();
} catch (IOException e) {
throw new ElasticsearchException("Failed to put mapping for " + indexName, e);
}
}
Using Elasticsearch 7.x:
You need to create a variable IndexCoordinates.of("indexName")
Get the IndexOperations from the ElasticSearchTemplate for that index
Create your index via the indexOperations variable like this:
IndexOperations indexOperations = elasticsearchTemplate.indexOps(indexCoordinates);
String indexSettings = "" //Pass json string here
String mappingJson = "" //Pass json string here
Document mapping = Document.parse(mappingJson);
Map<String, Object> settings = JacksonUtil.fromString(indexSettings, new TypeReference<>() {});
indexOperations.create(settings, mapping);
indexOperations.refresh(); //(Optional) refreshes the doc count
It really depends on which spring-data-elasticsearch you are using. Feel free to checkout the documentation as well:
https://docs.spring.io/spring-data/elasticsearch/docs/current/reference/html/#new-features
Hope this helps with your elasticsearch journey! Feel free to ask more questions regarding the java implementation :)

Related

Spring WebFlux check user exists

I want to check that the user has not been created yet before creating a new one, if there is then create an error... I found a similar question, but I can't remake it =(
Spring WebFlux: Emit exception upon null value in Spring Data MongoDB reactive repositories?
public Mono<CustomerDto> createCustomer(Mono<CustomerDto> dtoMono) {
//How Create Mono error???
Mono<Customer> fallback = Mono.error(new DealBoardException("Customer with email: " + dtoMono ???));
return dtoMono.map(customerConverter::convertDto) //convert from DTO to Document
.map(document -> {
customerRepository.findByEmailOrPhone(document.getEmail(), document.getPhone())
})
.switchIfEmpty() //How check such customer doesn't exists?
.map(document -> { //Filling in additional information from other services
var customerRequest = customerConverter.convertDocumentToStripe(document);
var customerStripe = customerExternalService.createCustomer(customerRequest);
document.setCustomerId(customerStripe.getId());
return document;
})
.flatMap(customerRepository::save) //Save to MongoDB
.map(customerConverter::convertDocument); //convert from Document to Dto
}
public Mono<User> create(String username, String password) {
User user = new User();
user.setUsername(username);
user.setPassword(encoder.encode(password));
return userRepo.existsUserByUsername(username)
.flatMap(exists -> (exists) ? Mono.error(UserAlreadyExists::new) : userRepo.save(user));
}
Add the following index declaration on the top of your Customer class:
#CompoundIndex(def = "{'email': 1, 'phone': 1}", unique = true)
That will prevent duplicate entries to be inserted in the database.
You can catch your org.springframework.dao.DuplicateKeyException with the following construct:
customerService.save(newCustomer).onErrorMap(...);
Ref: MongoDB Compound Indexes official documentation

Nest ElasticClient with multiple indexes to index a document

At first I had 1 index and my elasticclient was setup like below in my startup.cs
public static IServiceCollection AddElasticClient(this IServiceCollection services)
{
var elasticSettings = services.BuildServiceProvider().GetService<IOptions<ElasticSettings>>().Value;
var settings = new ConnectionSettings(new Uri(elasticSettings.Uri));
settings
.ThrowExceptions(elasticSettings.ThrowExceptions)
.PrettyJson(elasticSettings.PrettyJson)
.DefaultIndex(elasticSettings.Index)
.BasicAuthentication(elasticSettings.Username, elasticSettings.Password)
.DefaultMappingFor<CorrelationContext>(ms => ms.Ignore(p => p.DgpHeader));
var client = new ElasticClient(settings);
services.AddSingleton<IElasticClient>(client);
return services;
}
My writer looks like
public class ElasticWriter : IElasticWriter
{
private readonly IElasticClient _elasticClient;
public ElasticWriter(IElasticClient elasticClient)
{
_elasticClient = elasticClient ?? throw new ArgumentNullException(nameof(elasticClient));
}
public void Write(AuditElasticDoc doc)
{
var indexResponse = _elasticClient.IndexDocument(doc);
if (!indexResponse.IsValid)
{
throw indexResponse.OriginalException ?? new Exception("Invalid Elastic response when writing document.");
}
}
}
Now there is a new requirement by which they can provide the name of the index to write to.
All authentication data of the different indexes are provided through config settings, so I have everything available at startup.
The document type is always the same.
I found examples of specifying the index when querying but not when indexing.
Can I provide multiple indexes in my ElasticClient and specify the index when executing the IndexDocument?
Or do I need a separate client for each index?
If the latter, is there a way I can still use DI to inject the client in my writer or do I have to create one there at the spot?
Thx.
I'm using Nest 7.6.1
Instead of using IndexDocument, you can use IndexAsync method which will allow you to control additional request parameters
var indexResponse = await _elasticClient.IndexAsync(doc, descriptor => descriptor.Index("other"));
IndexDocument is a wrapper method, hiding the complexity of indexing documents from the clients. Have a look.
Request auth configuration
var indexResponse = await _elasticClient.IndexAsync(doc,
descriptor => descriptor
.Index("other")
.RequestConfiguration(rq => rq.BasicAuthentication("user", "pass")));

How to use ES Java API to create a new type of an index

I have succeed create an index use Client , the code like this :
public static boolean addIndex(Client client,String index) throws Exception {
if(client == null){
client = getSettingClient();
}
CreateIndexRequestBuilder requestBuilder = client.admin().indices().prepareCreate(index);
CreateIndexResponse response = requestBuilder.execute().actionGet();
return response.isAcknowledged();
//client.close();
}
public static boolean addIndexType(Client client, String index, String type) throws Exception {
if (client == null) {
client = getSettingClient();
}
TypesExistsAction action = TypesExistsAction.INSTANCE;
TypesExistsRequestBuilder requestBuilder = new TypesExistsRequestBuilder(client, action, index);
requestBuilder.setTypes(type);
TypesExistsResponse response = requestBuilder.get();
return response.isExists();
}
however, the method of addIndexType is not effected, the type is not create .
I don't know how to create type ?
You can create types when you create the index by providing a proper mapping configuration. Alternatively a type gets created when you index a document of a certain type. However the first suggestion is the better one, because then you can control the full mapping of that type instead of relying on dynamic mapping.
You can set types in the following way:
// JSON schema is the JSON mapping which you want to give for the index.
JSONObject builder = new JSONObject().put(type, JSONSchema);
// And then just fire the below command
client.admin().indices().preparePutMapping(indexName)
.setType(type)
.setSource(builder.toString(), XContentType.JSON)
.execute()
.actionGet();

Missing document in Elasticsearch when using BulkProcessor

I'm using a [java] kafka-producer to push data to kafka-topic x and a [java] high level consumer/bulkProcessor to read from topic x and index data to elasticsearch. The producer pushes 10 docs each time. When I start my java code for bulkProcessor for the first time after running producer, I see only 9 records being pushed to ES, all with "_version": 1. The 10th record is not in ES.
But somehow, beforeBulk() and afterBulk() methods show the follwoing results.
Going to execute new bulk composed of 10 actions
Executed bulk composed of 10 actions
This moment onwards, if I remove the elasticsearch index and use the producer, I see 10 records consistently. I have no idea why this is happening. Any help is appreciated.
Note: ES version 2.2.0
Kafka: 0.9.0.0
EDIT [Added relevant code]
public Consumer(KafkaStream a_stream, int a_threadNumber, String esHost, String esCluster, int bulkSize, String topic) {
/*Create transport client*/
BulkProcessor bulkProcessor;
this.bulkProcessor = BulkProcessor.builder(client, new BulkProcessor.Listener() {
public void beforeBulk(long executionId, BulkRequest request) {
System.out.format("Going to execute new bulk composed of %d actions\n", request.numberOfActions());
}
public void afterBulk(long executionId, BulkRequest request, BulkResponse response) {
System.out.format("Executed bulk composed of %d actions\n", response.getItems().length);
}
public void afterBulk(long executionId, BulkRequest request, Throwable failure) {
System.out.format("Error executing bulk", failure);
}
}).setBulkActions(bulkSize)
.setBulkSize(new ByteSizeValue(200, ByteSizeUnit.MB))
.setFlushInterval(TimeValue.timeValueSeconds(1))
.build();
}
public void run() {
ConsumerIterator<byte[], byte[]> it = m_stream.iterator();
while (it.hasNext()) {
byte[] x = it.next().message();
try {
bulkProcessor.add(new IndexRequest(index, type, id.toString()).source(modifyMsg(x).toString()));
}
catch (Exception e) {
logger.warn("bulkProcessor failed: " + m_threadNumber + e.getMessage());
}
}
logger.info("Shutting down Thread: " + m_threadNumber);
}
Docs going to ES are of the following form:
{"index":"temp1","type":"temp2","id":"0","event":"we're doomed"}
{"index":"temp1","type":"temp2","id":"1","event":"we're doomed"}
{"index":"temp1","type":"temp2","id":"2","event":"we're doomed"}
...
{"index":"temp1","type":"temp2","id":"9","event":"we're doomed"}
[EDIT]
If I add the following line in my run() method the problem is gone.
public void run() {
...
bulkProcessor.add(new IndexRequest("")); //Added this line
while (it.hasNext()) {
...
}
...
}
I feel like such a fool. In the line bulkProcessor.add(new IndexRequest(index, type, id.toString()).source(modifyMsg(x).toString())); the method modifyMsg() was initializing index, type and id, which was set to empty string in the constructor. That's why my first index request was failing as it had invalid index name.

PrepareResponse().AsActionResult() throws unsupported exception DotNetOpenAuth CTP

Currently I'm developing an OAuth2 authorization server using DotNetOpenAuth CTP version. My authorization server is in asp.net MVC3, and it's based on the sample provided by the library. Everything works fine until the app reaches the point where the user authorizes the consumer client.
There's an action inside my OAuth controller which takes care of the authorization process, and is very similar to the equivalent action in the sample:
[Authorize, HttpPost, ValidateAntiForgeryToken]
public ActionResult AuthorizeResponse(bool isApproved)
{
var pendingRequest = this.authorizationServer.ReadAuthorizationRequest();
if (pendingRequest == null)
{
throw new HttpException((int)HttpStatusCode.BadRequest, "Missing authorization request.");
}
IDirectedProtocolMessage response;
if (isApproved)
{
var client = MvcApplication.DataContext.Clients.First(c => c.ClientIdentifier == pendingRequest.ClientIdentifier);
client.ClientAuthorizations.Add(
new ClientAuthorization
{
Scope = OAuthUtilities.JoinScopes(pendingRequest.Scope),
User = MvcApplication.LoggedInUser,
CreatedOn = DateTime.UtcNow,
});
MvcApplication.DataContext.SaveChanges();
response = this.authorizationServer.PrepareApproveAuthorizationRequest(pendingRequest, User.Identity.Name);
}
else
{
response = this.authorizationServer.PrepareRejectAuthorizationRequest(pendingRequest);
}
return this.authorizationServer.Channel.PrepareResponse(response).AsActionResult();
}
Everytime the program reaches this line:
this.authorizationServer.Channel.PrepareResponse(response).AsActionResult();
The system throws an exception which I have researched with no success. The exception is the following:
Only parameterless constructors and initializers are supported in LINQ to Entities.
The stack trace: http://pastebin.com/TibCax2t
The only thing I've done differently from the sample is that I used entity framework's code first approach, an I think the sample was done using a designer which autogenerated the entities.
Thank you in advance.
If you started from the example, the problem Andrew is talking about stays in DatabaseKeyNonceStore.cs. The exception is raised by one on these two methods:
public CryptoKey GetKey(string bucket, string handle) {
// It is critical that this lookup be case-sensitive, which can only be configured at the database.
var matches = from key in MvcApplication.DataContext.SymmetricCryptoKeys
where key.Bucket == bucket && key.Handle == handle
select new CryptoKey(key.Secret, key.ExpiresUtc.AsUtc());
return matches.FirstOrDefault();
}
public IEnumerable<KeyValuePair<string, CryptoKey>> GetKeys(string bucket) {
return from key in MvcApplication.DataContext.SymmetricCryptoKeys
where key.Bucket == bucket
orderby key.ExpiresUtc descending
select new KeyValuePair<string, CryptoKey>(key.Handle, new CryptoKey(key.Secret, key.ExpiresUtc.AsUtc()));
}
I've resolved moving initializations outside of the query:
public CryptoKey GetKey(string bucket, string handle) {
// It is critical that this lookup be case-sensitive, which can only be configured at the database.
var matches = from key in db.SymmetricCryptoKeys
where key.Bucket == bucket && key.Handle == handle
select key;
var match = matches.FirstOrDefault();
CryptoKey ck = new CryptoKey(match.Secret, match.ExpiresUtc.AsUtc());
return ck;
}
public IEnumerable<KeyValuePair<string, CryptoKey>> GetKeys(string bucket) {
var matches = from key in db.SymmetricCryptoKeys
where key.Bucket == bucket
orderby key.ExpiresUtc descending
select key;
List<KeyValuePair<string, CryptoKey>> en = new List<KeyValuePair<string, CryptoKey>>();
foreach (var key in matches)
en.Add(new KeyValuePair<string, CryptoKey>(key.Handle, new CryptoKey(key.Secret, key.ExpiresUtc.AsUtc())));
return en.AsEnumerable<KeyValuePair<string,CryptoKey>>();
}
I'm not sure that this is the best way, but it works!
It looks like your ICryptoKeyStore implementation may be attempting to store CryptoKey directly, but it's not a class that is compatible with the Entity framework (due to not have a public default constructor). Instead, define your own entity class for storing the data in CryptoKey and your ICryptoKeyStore is responsible to transition between the two data types for persistence and retrieval.

Resources