I'm using the Nest client to programmatically execute requests against an Elasticsearch index. I need to use the UpdateByQuery API to update existing data in my index. To improve performance on large data sets, the recommended approach is to use slicing. In my case I'd like to use the automatic slicing feature documented here.
I've tested this out in the Kibana dev console and it works beautifully. I'm struggling on how to set this property in code through the Nest client interface. here's a code snippet:
var request = new Nest.UpdateByQueryRequest(indexModel.Name);
request.Conflicts = Elasticsearch.Net.Conflicts.Proceed;
request.Query = filterQuery;
// TODO Need to set slices to auto but the current client doesn't allow it and the server
// rejects a value of 0
request.Slices = 0;
var elasticResult = await _elasticClient.UpdateByQueryAsync(request, cancellationToken);
The comments on that property indicate that it can be set to "auto", but it expects a long so that's not possible.
// Summary:
// The number of slices this task should be divided into. Defaults to 1, meaning
// the task isn't sliced into subtasks. Can be set to `auto`.
public long? Slices { get; set; }
Setting to 0 just throws an error on the server. Has anyone else tried doing this? Is there some other way to configure this behavior? Other APIs seem to have the same problem, like ReindexOnServerAsync.
This was a bug in the spec and an unfortunate consequence of generating this part of the client from the spec.
The spec has been fixed and the change will be reflected in a future version of the client. For now though, it can be set with the following
var request = new Nest.UpdateByQueryRequest(indexModel.Name);
request.Conflicts = Elasticsearch.Net.Conflicts.Proceed;
request.Query = filterQuery;
((IRequest)request).RequestParameters.SetQueryString("slices", "auto");
var elasticResult = await _elasticClient.UpdateByQueryAsync(request, cancellationToken);
Related
I have a question regarding a small issue that I'm having. I've created a widget that will live on the Service Portal to allow an admin to Accept or Reject requests.
The data for the widget is pulling from the Approvals (approval_approver) table. Under my GlideRecord, I have a query that checks for the state as requested. (Ex. addQuery('state', 'requested'))
To narrow down the search, I tried entering addQuery('sys_id', current.sys_id). When I use this query, my script breaks and I get an error on the Service Portal end.
Here's a sample of the GlideRecord script I've written to Accept.
[//Accept Request
if(input && input.action=="acceptApproval") {
var inRec1 = new GlideRecord('sysapproval_approver');
inRec1.addQuery('state', 'requested');
//inRec1.get('sys_id', current.sys_id);
inRec1.query();
if(inRec1.next()) {
inRec1.setValue('state', 'Approved');
inRec1.setValue('approver', gs.getUserID());
gs.addInfoMessage("Accept Approval Processed");
inRec1.update();
}
}][1]
I've research the web, tried using $sp.getParameter() as a work-around and no change.
I would really appreciate any help or insight on what I can do different to get script to work and filter the right records.
If I understand your question correctly, you are asking how to get the sysId of the sysapproval_approver record from the client-side in a widget.
Unless you have defined current elsewhere in your server script, current is undefined. Secondly, $sp.getParameter() is used to retrieve URL parameters. So unless you've included the sysId as a URL parameter, that will not get you what you are looking for.
One pattern that I've used is to pass an object to the client after the initial query that gets the list of requests.
When you're ready to send input to the server from the client, you can add relevant information to the input object. See the simplified example below. For the sake of brevity, the code below does not include error handling.
// Client-side function
approveRequest = function(sysId) {
$scope.server.get({
action: "requestApproval",
sysId: sysId
})
.then(function(response) {
console.log("Request approved");
});
};
// Server-side
var requestGr = new GlideRecord();
requestGr.addQuery("SOME_QUERY");
requestGr.query(); // Retrieve initial list of requests to display in the template
data.requests = []; // Add array of requests to data object to be passed to the client via the controller
while(requestsGr.next()) {
data.requests.push({
"number": requestsGr.getValue("number");
"state" : requestsGr.getValue("state");
"sysId" : requestsGr.getValue("sys_id");
});
}
if(input && input.action=="acceptApproval") {
var sysapprovalGr = new GlideRecord('sysapproval_approver');
if(sysapprovalGr.get(input.sysId)) {
sysapprovalGr.setValue('state', 'Approved');
sysapprovalGr.setValue('approver', gs.getUserID());
sysapprovalGr.update();
gs.addInfoMessage("Accept Approval Processed");
}
...
This question was previously posted here but I am repeating the question and a summary of the discussion here to have a complete question.
I have set up a ODataController in ASP.NET WebAPI with DocumentDB/CosmosDB as a backend. It works quite ok, I return the result of CreateDocumentQuery and $select,$top,$orderby and $filter work fine.
However, $skip does not. I know that this is a planned feature (Vote here) but I would like to know if there is any temporary workaround.
My plan was to have a "continuationToken" parameter to my OData controller action. I would then pass this using FeedOptions.RequestContinuation to CreateDocumentQuery. However, CreateDocumentQuery does not return a response token. I then tried with ReadDocumentFeedAsync, which does return a response token, but it does not return a IQueryable.
This is an example of api code inside a ODataController:
public IQueryable<Result> FindResults(Uri collectionUri, string continuationToken)
{
FeedOptions feedOptions = new FeedOptions();
feedOptions.MaxItemCount = 100;
feedOptions.RequestContinuation = continuationToken;
var query = client.CreateDocumentQuery<Result>(collectionUri, queryString, feedOptions).AsDocumentQuery();
return query as IQueryable<Result>;
}
However, what would the client code look like? I.e. what is the http-request required to keep the IQueryable alive to be able to use the continuationToken when paging?
I guess I could store the result of ReadDocumentFeedAsync in memory and return a IQueryable to that, but is there a better solution?
Did you look at the PageResult object?
For non-OData formats, it is still possible to support next-page links and inline count, by wrapping the query results in a PageResult object. However, it requires a bit more code. Here is an example:
public PageResult<Product> Get(ODataQueryOptions<Product> options)
{
ODataQuerySettings settings = new ODataQuerySettings()
{
PageSize = 5
};
IQueryable results = options.ApplyTo(_products.AsQueryable(), settings);
return new PageResult<Product>(
results as IEnumerable<Product>,
Request.GetNextPageLink(),
Request.GetInlineCount());
}
https://learn.microsoft.com/en-us/aspnet/web-api/overview/odata-support-in-aspnet-web-api/supporting-odata-query-options
Is there any way to track changes to Metadata, like new fields, new entities and so on?
It is difficult to control a very large project in the same environment, so sometimes there are some customization that should not be deployed to productions (Mostly are mistakes or test in a development environment).
And there is a way to know who did that customization?
I am looking to know every possible change, not any in particular.
You have to use the RetrieveMetadataChangesRequest and it is not possible to know who made the change.
This is available only from Microsoft Dynamics CRM 2011 Update Rollup 12
This request is intended to be used to cache information from the metadata and be able to work offline, but we can use it to track changes to metadata in complex projects and complex teams
Examples on internet are not very friendly so this is how you can use the request:
The request can be completed only with filling one parameter
RetrieveMetadataChangesRequest req = new RetrieveMetadataChangesRequest()
{
ClientVersionStamp = null
};
var response = (RetrieveMetadataChangesResponse)service.Execute(req);
The first time you executed this request ClientVersionStamp needs to be null, because there was no request made to the metadata before and there is no ClientVersionStamp. This parameter is the last time you query for metadata changes and if it is null it will bring all customization from all time, so probably this request won't complete on time so we need to tune up.
var EntityFilter = new MetadataFilterExpression(LogicalOperator.And);
EntityFilter.Conditions.Add(new MetadataConditionExpression("SchemaName", MetadataConditionOperator.Equals, "ServiceAppointment"));
var entityQueryExpression = new EntityQueryExpression()
{
Criteria = EntityFilter
};
RetrieveMetadataChangesRequest req = new RetrieveMetadataChangesRequest()
{
Query = entityQueryExpression,
ClientVersionStamp = null
};
var response = (RetrieveMetadataChangesResponse)service.Execute(req);
This will query all metadata changes for "ServiceAppointment", feel free to use the entity you want, but what we need is the ServerTimeStamp from the response, it will looks like "22319800!09/13/2017 16:17:46", if you try to send this time stamp first, it will throw an exception, so it is necessary to query first to get a server time stamp.
Now you can use the request and the time stamp to retrieve all new changes since "22319800!09/13/2017 16:17:46"
RetrieveMetadataChangesRequest req = new RetrieveMetadataChangesRequest()
{
Query = entityQueryExpression,
ClientVersionStamp = #"22319800!09/13/2017 16:17:46"
};
var response = (RetrieveMetadataChangesResponse)service.Execute(req);
You can filter the query to match your needs, only search for specific entities, labels, relationship, keys and attributes or specific properties.
EntityQueryExpression entityQueryExpression = new EntityQueryExpression()
{
Criteria = EntityFilter,
Properties = EntityProperties,
RelationshipQuery = new RelationshipQueryExpression()
{
Properties = RelationshipProperties,
Criteria = RelationshipFilter
},
AttributeQuery = new AttributeQueryExpression()
{
Properties = AttributeProperties,
Criteria = AttributeFilter
}
};
Use this request and implement it the way you need.
A couple more options:
Register a plugin on Publish and Publish All, and track who did
the publish and when. That may help you narrow down who was making
changes, although someone could technically make a change without
publishing it, so not perfect information.
If you're using Dynamics OnPremise, the Metadata tables sometimes store information about who made a change that is not visible through a Metadata retrieve. I've found this to be very spotty though, not all Metadata has a Modified By user stored.
EDIT:
This is basically what I want to do, only in Java
Using ElasticSearch, we add documents to an index bypassing IndexRequest items to a BulkRequestBuilder.
I would like for the documents to be dropped from the index after some time has passed (time to live/ttl)
This can be done either by setting a default for the index, or on a per-document basis. Either approach is fine by me.
The code below is an attempt to do it per document. It does not work. I think it's because TTL is not enabled for the index. Either show me what Java code I need to add to enable TTL so the code below works, or show me different code that enables TTL + sets default TTL value for the index in Java I know how to do it from the REST API but I need to do it from Java code, if at all possible.
logger.debug("Indexing record ({}): {}", id, map);
final IndexRequest indexRequest = new IndexRequest(_indexName, _documentType, id);
final long debug = indexRequest.ttl();
if (_ttl > 0) {
indexRequest.ttl(_ttl);
System.out.println("Setting TTL to " + _ttl);
System.out.println("IndexRequest now has ttl of " + indexRequest.ttl());
}
indexRequest.source(map);
indexRequest.operationThreaded(false);
bulkRequestBuilder.add(indexRequest);
}
// execute and block until done.
BulkResponse response;
try {
response = bulkRequestBuilder.execute().actionGet();
Later I check in my unit test by polling this method, but the document count never goes down.
public long getDocumentCount() throws Exception {
Client client = getClient();
try {
client.admin().indices().refresh(new RefreshRequest(INDEX_NAME)).actionGet();
ActionFuture<CountResponse> response = client.count(new CountRequest(INDEX_NAME).types(DOCUMENT_TYPE));
CountResponse countResponse = response.get();
return countResponse.getCount();
} finally {
client.close();
}
}
After a LONG day of googling and writing test programs, I came up with a working example of how to use ttl and basic index/object creation from the Java API. Frankly most of the examples in the docs are trivial, and some JavaDoc and end-to-end examples would go a LONG way to help those of us who are using the non-REST interfaces.
Ah well.
Code here: Adding mapping to a type from Java - how do I do it?
I'm doing a small research about a client for elastic search in .net and I found that NEST is one of the most supported solutions for this matter.
I was looking at Nest's docummentation and I couldn´t find a way to output a raw json from a query and avoid the serialization into an object, because I'm using angularJs in the front end I don´t want to overload the process of sending the information to the client with some unnecessary steps.
......and also I'd like to know how can I overrdide the serialization process?
I found that NEST uses Json.NET which I would like to change for the servicestack json serielizer.
thanks!
Hi Pedro you can do this with NEST
var searchDescriptor = new SearchDescriptor<ElasticSearchProject>()
.Query(q=>q.MatchAll());
var request = this._client.Serializer.Serialize(searchDescriptor);
ConnectionStatus result = this._client.Raw.SearchPost(request);
Assert.NotNull(result);
Assert.True(result.Success);
Assert.IsNotEmpty(result.Result);
This allows you to strongly type your queries, but return the string .Result which is the raw response from elasticsearch as string to your
request can be an object or the string so if you are OK with the internal json serialize just pass searchDescriptor directly
Use RequestResponseSerializer instead of Serializer.
var searchDescriptor = ...;
...
byte[] b = new byte[60000];
using (MemoryStream ms = new MemoryStream(b))
{
this._client.RequestResponseSerializer.Serialize(searchDescriptor , ms);
}
var rawJson = System.Text.Encoding.Default.GetString(b);