I'm running an updateByQuery operation in ElasticSearch using Spring Data ElasticSearch (Spring Boot parent v2.6.1, Elastic Search 7.15.2).
In my ES index I have stored 2 documents.
When I give a non-existent document in the search, it doesn't give me any error, because of which I'm not able to distinguish whether the update actually ran or not.
Updates for a document that exists work fine. I'd like to figure a way such that if no rows are edited, I can log it.
What should I look at? What should I change so that I can get some message to understand if there was an update?
Here's my code snippet:
UpdateByQueryRequest request = new UpdateByQueryRequest('index');
Map<String, Object> data = new HashMap<>();
data.put("marks", "30");
data.put("name", "timmy");
data.put("roll_number", "10");
request.setScript(
new Script(
ScriptType.INLINE, "painless",
"if (ctx._source.name == params.name && ctx._source.roll_number == params.roll_number) {ctx._source.marks=params.marks;}",
data));
BulkByScrollResponse resp = globalClient.updateByQuery(request, RequestOptions.DEFAULT);
log.info("response: {}",resp.getStatus());
I've added the response status as well. What I find weird is that in case of both existent and non-existent document, the updated parameter count is 2, same as the number of documents I have in my index.
response in case of non-existent record:
response: BulkIndexByScrollResponse[sliceId=null,updated=2,created=0,deleted=0,batches=1,versionConflicts=0,noops=0,retries=0,throttledUntil=0s]
response in case of existing record:
response: BulkIndexByScrollResponse[sliceId=null,updated=2,created=0,deleted=0,batches=1,versionConflicts=0,noops=0,retries=0,throttledUntil=0s]
This is pure Elasticsearch code, nothing from Spring Data Elasticsearch .
Where do you specify the query? I don't see any in your code. That means that all documents will be updated - 2 in your case.
Related
want to delete time based indices on elastic search older than some specific (let's say 30) days using date math expression in JAVA.
Trying to implement following approach in our spring boot application using Transaport Client but getting index not exist exception.
https://www.elastic.co/guide/en/elasticsearch/reference/current/date-math-index-names.html
When the same URI encoded index name (which is passed to DELETE INDEX API) is used to search index using query GET uri_encoded_index_name on kibana, It shows that index exists.
Is there something I am missing out ?
Is there any better approach to do that without using curator and wilcard characters?
code snippet:
String indexName = "<" + indexNameStaticPart + "{now/d-30d{MMddYYYY}}>";
String encodedIndexName = UriEncoder.encode( indexName ).replace( "/", "%2F" );
AcknowledgedResponse response = client.admin().indices().delete( new DeleteIndexRequest( encodedIndexName ) ).actionGet();
encodedIndexName : %3Cstring__string_string__%7Bnow%2Fd-30d%7BMMddyyyy%7D%7D%3E
kibana:
GET encodedIndexName
DELETE encodedIndexName
Try it without URI encoding..
https://discuss.elastic.co/t/delete-time-based-indices-on-elastic-search-older-than-30-days-using-date-math-expression-in-java-gives-indexnotexists-exception/171365
I have some custom data(let's call Camera) in my ElasticSearch, the data showed in Kibana is like
And I tried to delete data by Query according to the accepted answer in this article ElasticSearch Delete by Query, my code is like
String query = "{\"Name\":\"test Added into Es\"}";
DeleteByQuery delete = new DeleteByQuery.Builder(query).addIndex(this._IndexName).addType(this._TypeName).build();
JestResult deleteResult = this._JestClient.execute(delete);
And the result is 404 Not Found.
Its obvious that there exist one Camera data in ElasticSearch which Name match the query, so I believe the 404 is caused by other reason.
Did I do anything wrong? Should I change the query string?
The query needs to be a real query, not a partial document
Try with this instead
String query = "{\"query\": { \"match\": {\"Name\":\"test Added into Es\"}}}";
I am using elastic search for the first time and based on requirements i have some doubts and questions for scroll
To retrieve all data which are fulfilling all search criteria
1)I am trying to use scroll but i found while searching about it
https://www.elastic.co/guide/en/elasticsearch/reference/current/breaking_21_search_changes.html
i found Search type scan is deprecated
but NEST is supporting it
so should i use "search type scan" or "sort by doc"? (I am using elastic search 2.4)
2)Can i use "sorting on any field" when using scrolling?
3)while doing clear scroll
var test2 = client.ClearScroll(x=>x.ScrollId(results.ScrollId));
Getting error as below:
Invalid NEST response built from a unsuccessful low level call on DELETE: /_search/scroll
Audit trail of this API call:
[1] BadResponse: Node: http://mydomain#localhost:9200/ Took: 00:00:00.0160110
OriginalException: System.Net.WebException: The remote server returned an error: (404) Not Found.
at System.Net.HttpWebRequest.GetResponse()
at Elasticsearch.Net.HttpConnection.Request[TReturn](RequestData requestData) in C:\Users\russ\source\elasticsearch-net-2.x\src\Elasticsearch.Net\Connection\HttpConnection.cs:line 141
Request:
{"scroll_id":["c2NhbjswOzE7dG90YWxfaGl0czoxMjs="]}
Response:
{}
so is it correct way of clearing scroll or not?
Update: : below is my code :
List<Object> indexedList = new List<Object>();
ISearchResponse<ListingSearch> listingResult =
client.Search<ListingSearch>(search => search
.Index(Constant.ES_INDEX)
.Type(Constant.ES_TYPE)
.From(listingSearch.StartIndex)
.Size(10)
.Source(s => s.Include(i => i.Fields(outpputFields)))
.Query(query => query.
Bool(boolean => boolean.
Must(
must => must.Term(t => t.Field("is_deleted").Value(false))
)
.Sort(x => x.Field("_doc", SortOrder.Ascending))
.Scroll("60s")
);
List<Object> indexedList = new List<Object>();
var results = client.Scroll<ListingSearch>("60s", listingResult.ScrollId);
while (results.Documents.Any())
{
foreach (var doc in results.Hits)
{
indexedList.Add(doc);
}
results = client.Scroll<ListingSearch>("60s", results.ScrollId);
}
var test2 = client.ClearScroll(x=>x.ScrollId(results.ScrollId));
//Clear Scroll
With above code I am getting data
but if i change size from 10 to 1000, getting no records.
Not sure if issue is the amount of data because my ES db has only 12-15 documents.
NEST 2.x versions have SearchType.Scan because NEST 2.x versions are compatible with all Elasticsearch 2.x versions, so the search type needs to exist when using NEST 2.x against Elasticsearch 2.0. Sending the search type through in later versions won't have any effect.
The most efficient way of retrieving documents with scroll is sorting by _doc but you can specify any sort parameters when scrolling.
When using the scroll API, you should use the scroll_id from the previous request in the next scroll call to fetch the next set of results. Once you have finished with a scroll, it is a good idea to clear it by calling ClearScroll() as you are doing. Your call looks correct; perhaps the scroll_id has already expired at the point you make the clear call?
I'm building a native query but I only want to return certain fields, all of which are held within a parent field. I think I am looking for the QueryBuilders or NativeSearchQueryBuilder equivalent of the REST API's _source. Here's a code example:
NativeSearchQueryBuilder sb = new NativeSearchQueryBuilder()
.withIndices("myIndex")
.withTypes("myType")
.withQuery(QueryBuilders.queryStringQuery("parent.field2:Foo*"));
.withFields("parent.field1");
I'd expect this to return a list of only parent.field1 that are associated with objects that have parent.field2 like Foo*. But it returns nothing.
Thanks for any help!
After some research, I found the answer is in NativeSearchQueryBuilder. I was just using an older version of spring-data elastic search, so I could not see this method: withSourceFilter. The way to do this is:
NativeSearchQueryBuilder sb = new NativeSearchQueryBuilder()
.withIndices("myIndex")
.withTypes("myType")
.withQuery(QueryBuilders.queryStringQuery("parent.field2:Foo*"));
.withSourceFilter(new FetchSourceFilter(<String array of includes>, null));
FetchSourceFilter takes 2 arguments, a String[] array of includes and one of excludes. In my example, I'd have an array like new String[]{"parent.field1"} passed to FetchSourceFilter, which in turn is passed to withSourceFilter. The search above will then return (once built and ran) a list of parent.field1 with parent.field2 like Foo*.
The version I upgraded to was spring-data-elasticsearch 2.0.2.
I am using elastic search multi get for reading documents after bulk update. Its returning some document sources as null.
MultiGetRequestBuilder builder = client.prepareMultiGet();
builder.setRefresh(true);
builder.add(indexName, type, idsList);
MultiGetResponse multiResponse = builder.execute().actionGet();
for (MultiGetItemResponse response : multiResponse.getResponses())
{
String customerJson = response.getResponse().getSourceAsString();
System.out.println("customerJson::" + customerJson);
}
Any issues in my code? Thanks in advance.
When you say "some return sources as null", I assume the get response is marking them as not existing..?
If that's the case, then maybe :
some indexation request in the bulk are failing dur to mapping/random error.
You need to refresh your index between the indexation and the multiget (i.e : your docs are not available for search yet)
transportClient.admin().indices().prepareRefresh(index).execute();
good luck
EDIT : You answered your own question in the comment, but for readability's sake : when using get or multiget, if a routing key was used when indexing, it must be specified again during the get, else, a wrong shard is determined using default routing and the get fails.