want to delete time based indices on elastic search older than some specific (let's say 30) days using date math expression in JAVA.
Trying to implement following approach in our spring boot application using Transaport Client but getting index not exist exception.
https://www.elastic.co/guide/en/elasticsearch/reference/current/date-math-index-names.html
When the same URI encoded index name (which is passed to DELETE INDEX API) is used to search index using query GET uri_encoded_index_name on kibana, It shows that index exists.
Is there something I am missing out ?
Is there any better approach to do that without using curator and wilcard characters?
code snippet:
String indexName = "<" + indexNameStaticPart + "{now/d-30d{MMddYYYY}}>";
String encodedIndexName = UriEncoder.encode( indexName ).replace( "/", "%2F" );
AcknowledgedResponse response = client.admin().indices().delete( new DeleteIndexRequest( encodedIndexName ) ).actionGet();
encodedIndexName : %3Cstring__string_string__%7Bnow%2Fd-30d%7BMMddyyyy%7D%7D%3E
kibana:
GET encodedIndexName
DELETE encodedIndexName
Try it without URI encoding..
https://discuss.elastic.co/t/delete-time-based-indices-on-elastic-search-older-than-30-days-using-date-math-expression-in-java-gives-indexnotexists-exception/171365
Related
I'm running an updateByQuery operation in ElasticSearch using Spring Data ElasticSearch (Spring Boot parent v2.6.1, Elastic Search 7.15.2).
In my ES index I have stored 2 documents.
When I give a non-existent document in the search, it doesn't give me any error, because of which I'm not able to distinguish whether the update actually ran or not.
Updates for a document that exists work fine. I'd like to figure a way such that if no rows are edited, I can log it.
What should I look at? What should I change so that I can get some message to understand if there was an update?
Here's my code snippet:
UpdateByQueryRequest request = new UpdateByQueryRequest('index');
Map<String, Object> data = new HashMap<>();
data.put("marks", "30");
data.put("name", "timmy");
data.put("roll_number", "10");
request.setScript(
new Script(
ScriptType.INLINE, "painless",
"if (ctx._source.name == params.name && ctx._source.roll_number == params.roll_number) {ctx._source.marks=params.marks;}",
data));
BulkByScrollResponse resp = globalClient.updateByQuery(request, RequestOptions.DEFAULT);
log.info("response: {}",resp.getStatus());
I've added the response status as well. What I find weird is that in case of both existent and non-existent document, the updated parameter count is 2, same as the number of documents I have in my index.
response in case of non-existent record:
response: BulkIndexByScrollResponse[sliceId=null,updated=2,created=0,deleted=0,batches=1,versionConflicts=0,noops=0,retries=0,throttledUntil=0s]
response in case of existing record:
response: BulkIndexByScrollResponse[sliceId=null,updated=2,created=0,deleted=0,batches=1,versionConflicts=0,noops=0,retries=0,throttledUntil=0s]
This is pure Elasticsearch code, nothing from Spring Data Elasticsearch .
Where do you specify the query? I don't see any in your code. That means that all documents will be updated - 2 in your case.
I am new in writing aggregate queries in Mongo DB + Spring
Scenario: We are storing birthDate(Jjava.uti.Date) in mongo db which got stored as ISO date. Now we are trying to look for the records which are matching with the dayOfMonth and Month only. So that we can corresponding object from the list.
I had gone through few solutions and here is the way I am trying but this is giving me a null set of records.
Aggregation agg = Aggregation.newAggregation(
Aggregation.project().andExpression("dayOfMonth(birthDate)").as("day").andExpression("month(birthDate)")
.as("month"),
Aggregation.group("day", "month"));
AggregationResults<Employee> groupResults = mongoTemplate.aggregate(agg, Employee.class, Employee.class);
I also tried applying a a query with the help of Criteria but this is also giving me a Employee object which all null content.
Aggregation agg = Aggregation.newAggregation(Aggregation.match(Criteria.where("birthDate").lte(new Date())), Aggregation.project().andExpression("dayOfMonth(birthDate)").as("day").andExpression("month(birthDate)")
.as("month"),
Aggregation.group("day", "month"));
AggregationResults<Employee> groupResults = mongoTemplate.aggregate(agg, Employee.class, Employee.class);
I must missing some important thing which is giving me these null data.
Additional Info: Employee object has only birthDate(Date) and email(String) in it
Please try to specify the fields to be included in the $project stage.
project("birthDate", "...").andExpression("...
The _id field is, by default, included in the output documents. To include any other fields from the input documents in the output documents, you must explicitly specify the inclusion in $project.
see: MongoDBReference - $project (aggregation)
I've created DATAMONGO-2200 to add an option to project directly onto the fields of a given domain type via something like project(Employee.class).
I have some custom data(let's call Camera) in my ElasticSearch, the data showed in Kibana is like
And I tried to delete data by Query according to the accepted answer in this article ElasticSearch Delete by Query, my code is like
String query = "{\"Name\":\"test Added into Es\"}";
DeleteByQuery delete = new DeleteByQuery.Builder(query).addIndex(this._IndexName).addType(this._TypeName).build();
JestResult deleteResult = this._JestClient.execute(delete);
And the result is 404 Not Found.
Its obvious that there exist one Camera data in ElasticSearch which Name match the query, so I believe the 404 is caused by other reason.
Did I do anything wrong? Should I change the query string?
The query needs to be a real query, not a partial document
Try with this instead
String query = "{\"query\": { \"match\": {\"Name\":\"test Added into Es\"}}}";
I'm facing some strange issue with "doc." keyword on Nest C# Elasticsearch.
I'm using Couchbase, and I have a class where one of its fields is an array of objects
I try to search inside this array for a specific value.
Something like this:
string mailFilesKey = string.Empty;
ISearchResponse<object> result = _mainManager.Client.Search<object>(c => c
.Type("MailFiles")
.Query(q =>
q.Term("SentFile_Id", fileId))
.Size(1));
Now, this thing actually works. But when I do this one, it doesn't work:
q.Term("doc.SentFile_Id", fileId))
Why?
haha ok nice one. I had this thing long time ago when i started to use Nest and elastic. If you have the object then you can use lambda expressions
like f=>f.SentFile_Id.
Now when you use a string to get the name of the field in nest you must know that all fields, index name, types in elastic are stored with lowercase first letter. So you should use this : q.Term("sentFile_Id", fileId))
Should work just fine.
I am using elastic search multi get for reading documents after bulk update. Its returning some document sources as null.
MultiGetRequestBuilder builder = client.prepareMultiGet();
builder.setRefresh(true);
builder.add(indexName, type, idsList);
MultiGetResponse multiResponse = builder.execute().actionGet();
for (MultiGetItemResponse response : multiResponse.getResponses())
{
String customerJson = response.getResponse().getSourceAsString();
System.out.println("customerJson::" + customerJson);
}
Any issues in my code? Thanks in advance.
When you say "some return sources as null", I assume the get response is marking them as not existing..?
If that's the case, then maybe :
some indexation request in the bulk are failing dur to mapping/random error.
You need to refresh your index between the indexation and the multiget (i.e : your docs are not available for search yet)
transportClient.admin().indices().prepareRefresh(index).execute();
good luck
EDIT : You answered your own question in the comment, but for readability's sake : when using get or multiget, if a routing key was used when indexing, it must be specified again during the get, else, a wrong shard is determined using default routing and the get fails.