I have template_update_delay=24h for caching the templates for 24 hours. If my URLTemplateLoader gets IOException due to temporary outage(http status 429) then freemarker.cache.TemplateCache will call storeNegativeLookup and cached the exception too.
cachedTemplate.templateOrException = e
// Template source was removed
if (!newLookupResult.isPositive()) {
if(debug) {
LOG.debug(debugName + " no source found.");
}
storeNegativeLookup(tk, cachedTemplate, null);
return null;
}
private void storeNegativeLookup(TemplateKey tk,
CachedTemplate cachedTemplate, Exception e) {
cachedTemplate.templateOrException = e;
cachedTemplate.source = null;
cachedTemplate.lastModified = 0L;
storeCached(tk, cachedTemplate);
}
Later even if the URL endpoint is up and available, freemarker.cache.TemplateCache:getTemplate() will keep picking the cachedTemplate with the IOException and will keep rethrowing the exception till the cache is not expired.
else if(t instanceof IOException) {
rethrown = true;
throwLoadFailedException((IOException)t);
}
This is causing application of fail all the time ((.
How can I force Freemarker to retry fetching the template from the source instead of cache if there was an exception happened last time?
Failed lookups are cached for the same reason successful ones are. A failed lookup can take as much or even more time (if for example a connection timeout is involved) as a successful one, and so can jam the application if it's for a frequently used template (by consuming the whole thread pool for example).
The problem is that there's only a single templateUpdateDelay. But it's possible that you want to use a different one depending what the cause of the error was. You should open a feature request for that on Jira.
What can you do right now though. You can catch the exception thrown by Configuration.getTemplate, and walk the cause trace to find out if the root cause is an exception that you don't want to cache, and then call Configuration.removeTemplateFromCache (but consider if you should only do it once in a certain time interval). I'm not sure if the exception with HTTP 429 can be recognized reliably out-of-the-box, or you have to customize the TemplateLoader so that the thrown exception contains the necessary information.
Related
I have a code similar to the one below. Every-time a DBLock appears, I want to get an alert in Dynatrace creating a problem so that I can see it on the dashboard and possibly get an email notification also. The DB lock would appear if the update count is greater than 1.
private int removeDBLock(DataSource dataSource) {
int updateCount = 0;
final Timestamp lastAllowedDBLockTime = new Timestamp(System.currentTimeMillis() - (5 * 60 * 1000));
final String query = format(RELEASE_DB_CHANGELOCK, lastAllowedDBLockTime.toString());
try (Statement stmt = dataSource.getConnection().createStatement()) {
updateCount = stmt.executeUpdate(query);
if(updateCount>0){
log.error("Stale DB Lock found. Locks Removed Count is {} .",updateCount);
}
} catch (SQLException e) {
log.error("Error while trying to find and remove Db Change Lock. ",e);
}
return updateCount;
}
I tried using the event API to trigger an event on my host mentioned here and was successful in raising a problem alert on my dashboard.
https://www.dynatrace.com/support/help/dynatrace-api/environment-api/events/post-event/?request-parameters%3C-%3Ejson-model=json-model
but this would mean injecting an api call in my code just for monitoring, any may lead to more external dependencies and hence more chance of failure.
I also tried creating a custom service detection by adding the class containing this method and the method itself in the custom service. But I do not know how I can link this to an alert or a event that creates a problem on the dashboard.
Are there any best practices or solutions on how I can do this in Dynatrace. Any leads would be helpful.
I would take a look at Custom Services for Java which will cause invocations of the method to be monitored in more detail.
Maybe you can extract a method which actually throws the exception and the outer method which handles it. Then it should be possible to alert on the exception.
There are also some more ways to configure the service via settings, i.e. raise an error based on a return value directly.
See also documentation:
https://www.dynatrace.com/support/help/how-to-use-dynatrace/transactions-and-services/configuration/define-custom-services/
https://www.dynatrace.com/support/help/technology-support/application-software/java/configuration-and-analysis/define-custom-java-services/
I'm trying to simulate a 2 minute time out on my Plugin by using Thread.Sleep(120000) to generate a Time Out but when that exception arises my try catch can't seem to catch them, even the finally block is skipped.
How would i properly catch it so that i can perform on creating a case record on this error?
I've tried using different catch exceptions to no avail. I've even tried profiling the plugin but it also won't profile the plugin due to the error.
protected override void ExecuteCrmPlugin(LocalPluginContext localContext) {
IPluginExecutionContext context = localContext.PluginExecutionContext;
IOrganizationService service = localContext.OrganizationService;
ITracingService tracer = localContext.TracingService;
try {
if (context.InputParameters.Contains("Target") && context.InputParameters["Target"] is Entity) {
Entity entity = (Entity)context.InputParameters["Target"];
if (context.MessageName.ToLower() != "create")
return;
if (entity.LogicalName.ToLower() == Contact.EntityLogicalName) {
tracer.Trace("Sleep for 2 min");
Thread.Sleep(120000);
}
}
}
catch (System.ServiceModel.FaultException<System.TimeoutException ex) {
throw new InvalidPluginExecutionException("An error occurred in the plug-in.", ex);
}
catch (System.ServiceModel.FaultException<Microsoft.Xrm.Sdk.OrganizationServiceFault ex) {
throw new InvalidPluginExecutionException("An error occurred in the plug-in.", ex);
}
catch (TimeoutException e) {
throw new InvalidPluginExecutionException("A timeout has occurred during the execution of the plugin.", e);
}
catch (FaultException ex) {
throw new InvalidPluginExecutionException("Err occurred.", ex);
}
catch (Exception ex) {
tracer.Trace(ex.ToString());
throw;
}
finally {
tracer.Trace("Finally");
}
}
In the Base Class I also have the same catch blocks.
The error is:
Unhandled exception:
Exception type: System.ServiceModel.FaultException`1[Microsoft.Xrm.Sdk.OrganizationServiceFault]
Message: An unexpected error occurred from ISV code. (ErrorType = ClientError) Unexpected exception from plug-in (Execute): MyPlugin.Plugins.PreCreateContact: System.TimeoutException: Couldn’t complete execution of the MyPlugin.Plugins.PreCreateContact plug-in within the 2-minute limit.Detail:
<OrganizationServiceFault xmlns:i="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://schemas.microsoft.com/xrm/2011/Contracts">
<ActivityId>397e0f4c-2e16-43ea-9368-ea76607820a5</ActivityId>
<ErrorCode>-2147220956</ErrorCode>
<ErrorDetails xmlns:d2p1="http://schemas.datacontract.org/2004/07/System.Collections.Generic" />
<HelpLink i:nil="true" />
<Message>An unexpected error occurred from ISV code. (ErrorType = ClientError) Unexpected exception from plug-in (Execute): MyPlugin.Plugins.PreCreateContact: System.TimeoutException: Couldn’t complete execution of the MyPlugin.Plugins.PreCreateContact plug-in within the 2-minute limit.</Message>
<Timestamp>2019-07-17T00:49:48.360749Z</Timestamp>
<ExceptionRetriable>false</ExceptionRetriable>
<ExceptionSource>PluginExecution</ExceptionSource>
<InnerFault i:nil="true" />
<OriginalException>System.TimeoutException: Couldn’t complete execution of the MyPlugin.Plugins.PreCreateContact plug-in within the 2-minute limit.
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at Microsoft.Crm.Sandbox.SandboxAppDomainHelper.Execute(IOrganizationServiceFactory organizationServiceFactory, Dictionary`2 sandboxServices, String pluginTypeName, String pluginConfiguration, String pluginSecureConfig, IPluginExecutionContext requestContext, Boolean enablePluginStackTrace, Boolean chaosFailAppDomain, String crashOnPluginExceptionMessage)
at Microsoft.Crm.Sandbox.SandboxAppDomainHelper.Execute(IOrganizationServiceFactory organizationServiceFactory, Dictionary`2 sandboxServices, String pluginTypeName, String pluginConfiguration, String pluginSecureConfig, IPluginExecutionContext requestContext, Boolean enablePluginStackTrace, Boolean chaosFailAppDomain, String crashOnPluginExceptionMessage)
at Microsoft.Crm.Sandbox.SandboxWorker.<>c__DisplayClass3_0.<Execute>b__0()</OriginalException>
<TraceText>
Entered MyPlugin.Plugins.PreCreateContact.Execute(), Correlation Id: 0c2b0dd3-d27c-46ea-a7e2-90c0729b326e, Initiating User: 61e01dfa-668a-e811-8107-123456
</TraceText>
</OrganizationServiceFault>
I guess once the plugin hits the 2-minute timeout, our ability to process anything is over. And that makes sense - if the system allowed us to run a catch block after the 2-minute timeout that catch block could run for another minute, or five.
One approach I've taken to allow a plugin or custom workflow process something that might take longer than the 2-minute timeout is to use a cancellation token to gracefully shut down before the timeout. Rather than catch the platform's timeout exception, you can throw your own at 115 seconds and cancel gracefully.
I've even gone so far as to preserve the state of the execution as XML stored in an annotation and retrigger the plugin up to the infinite loop limit of 7 times. And beyond that I've created four processes that each recur every hour, staggered by 15 minutes - i.e. 15, 30, 45, and 60 minutes after the main process. (A process can recur hourly without triggering an infinite loop exception). These processes catch any jobs that have preserved their state after 7 runs and retrigger them for another 7. Using this method I have had workflows complete calculations that took hours.
Mind you, this could be considered abuse of the asynchronous processing system. I'll admit it's a stretch, but the use case needed to handle big calculations entirely inside of Dynamics. Of course the standard approach to "escape" the sandbox timeout is to move processing to an external service (e.g. Azure API, Azure Functions, Flow, Logic Apps, etc.)
We're using PushStreamContent to stream some large lumps with Content-Disposition headers set and the like. As a number of people have discovered, the drawback is what happens when something goes wrong in the streaming?
At the very least, we were trying to get the error logged on our side so someone could follow up.
Recently, I ran into a weird situation. Putting a try/catch around the streaming function worked well enough for errors encountered before you actually started streaming (i.e. errors in sql queries and the like), but if the error occurred later (like in the serialization), the catch block doesn't fire.
Would anyone have any idea why that is?
e.g.
HttpResponseMessage response = new HttpResponseMessage(HttpStatusCode.OK);
try
{
response.Content = new PushStreamContent((stream, content, context) =>
{
using (XmlWriter rWriter = PrepForXmlOutput(stream))
{
rpt.GenerateXmlReport(rWriter, reportParams, true);
}
}, "EventReport", extension);
}
catch (Exception e)
{
// The first step of GenerateXmlReport() is to run the sql;
// if the error happens there, this fires and will log the exception
// if the error happens later, during the result serialization, this does NOT fire
Log.Error(e);
}
return response;
Hate it when I see the answer just after I hit Post.
Try/catch around the outside only covers until I return the HttpResponseMessage. When/where I get the exception depends on how far the inner method gets before that return happens.
The try/catch needed to be on the inner call (the one where all the work happens) to cover the whole lifecycle.
I have a CRM plugin registered on Create (synchronous, post-operation) of a custom entity that performs some actions, and I want the Create operation to succeed in spite of errors in the plugin. For performance reasons, I also want the plugin to fire immediately when a record is created, so making the plugin asynchronous is undesirable. I've implemented this by doing something like the following:
public class FooPlugin : IPlugin
{
public FooPlugin(string unsecureInfo, string secureInfo) { }
public void Execute(IServiceProvider serviceProvider)
{
try
{
// Boilerplate
var context = (IPluginExecutionContext) serviceProvider.GetService(typeof (IPluginExecutionContext));
var serviceFactory = (IOrganizationServiceFactory) serviceProvider.GetService(typeof (IOrganizationServiceFactory));
IOrganizationService service = serviceFactory.CreateOrganizationService(context.UserId);
// Additional validation omitted
var targetEntity = (Entity) context.InputParameters["Target"];
UpdateFrobber(service, (EntityReference)targetEntity["new_frobberid"]);
CreateFollowUpFlibber(service, targetEntity);
CloseTheEntity(service, targetEntity);
}
catch (Exception ex)
{
// Send an email but do not re-throw the exception
// because we don't want a failure to roll-back the transaction.
try
{
SendEmailForException(ex, context);
}
catch { }
}
}
}
However, when an error occurs (e.g. in UpdateFrobber(...)), the service client receives this exception:
System.ServiceModel.FaultException`1[Microsoft.Xrm.Sdk.OrganizationServiceFault]:
There is no active transaction. This error is usually caused by custom plug-ins
that ignore errors from service calls and continue processing.
Server stack trace:
at System.ServiceModel.Channels.ServiceChannel.HandleReply(ProxyOperationRuntime operation, ref ProxyRpc rpc)
at System.ServiceModel.Channels.ServiceChannel.Call(String action, Boolean oneway, ProxyOperationRuntime operation, Object[] ins, Object[] outs, TimeSpan timeout)
at System.ServiceModel.Channels.ServiceChannelProxy.InvokeService(IMethodCallMessage methodCall, ProxyOperationRuntime operation)
at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(ref MessageData msgData, Int32 type)
at Microsoft.Xrm.Sdk.IOrganizationService.Create(Entity entity)
at Microsoft.Xrm.Sdk.Client.OrganizationServiceProxy.CreateCore(Entity entity)
at Microsoft.Xrm.Sdk.Client.OrganizationServiceProxy.Create(Entity entity)
at Microsoft.Xrm.Client.Services.OrganizationService.<>c__DisplayClassd.<Create>b__c(IOrganizationService s)
at Microsoft.Xrm.Client.Services.OrganizationService.InnerOrganizationService.UsingService(Func`2 action)
at Microsoft.Xrm.Client.Services.OrganizationService.Create(Entity entity)
at MyClientCode() in MyClientCode.cs: line 100
My guess is that this happens because UpdateFrobber(...) uses the IOrganizationService instance derived from the plugin, so any CRM service calls that it makes participate in the same transaction as the plugin, and if those "child" operations fail, it causes the entire transaction to rollback. Is this correct? Is there a "safe" way to ignore an error from a "child" operation in a synchronous plugin? Perhaps a way of instantiating an IOrganizationService instance that doesn't re-use the plugin's context?
In case it's relevant, we're running CRM 2013, on-premises.
You cannot ignore unhandled exceptions from child plugins when your plugin is participating in a database transaction.
However, when your plugin is operating On Premise in partial trusted mode, you can actually create a OrganizationServiceProxy instance of your own and use that to access CRM. Be sure you reference the server your plugin is executing on to avoid "double hop" problems.
If really needed, I would create an ExecuteMultipleRequest with ContinueOnError = true, for your email you could just check the ExecuteMultipleResponse...
But it looks a bit overkill.
You can catch exceptions if running in async mode. Be sure to verify your mode when catching the exception.
Sample Code:
try
{
ExecuteTransactionResponse response =
(ExecuteTransactionResponse)service.Execute(exMultReq);
}
catch (Exception ex)
{
errored = true;
if (context.Mode == 0) //0 sync, 1 Async.
throw new InvalidPluginExecutionException(
$"Execute Multiple Transaction
Failed.\n{ex.Message}\n{innermessage}", ex);
}
if(errored == true)
{
//Do more stuff to handle it, such as Log the failure.
}
It is not possible to do so for a synchronous plugin.
A more detailed summary, explaining the execution mode and use case can be found on my blog: https://helpfulbit.com/handling-exceptions-in-plugins/
Cheers.
I am writing a RESTful web service using spring 3 and I noticed that when I implemented my DAO's (I am using spring-jdbc for database access), the exceptions that get thrown are pretty generic, so i am not able to identify if the exception occurred because my database is down or my query failed.
sample code:
try {
Q q = jdbcTemplate.queryForObject(MY_QUERY, new Object[]{id}, new MyMapper());
return q;
} catch (DataAccessException e) {
// What is this exception ? database down ? query failed ?
}
Unless I know what exception is this during runtime, I can't send back reasonable error message to service client.
Any help is appreciated. Thanks
You shouldn't be trying to catch every single possible exception; code so that you don't run into the possibility of multiple user or programmer error type exceptions. Generally there are three type of exceptions that occur in every DAO regardless of what you do. (1) your database is down... imo if this is the issue you've got bigger problems. (2) user authentication error which is easy enough to catch and deal with (however you should probably be handling that situation on your RESTful front-end. (3) improper data. If you have bad data, just send the attempted data back and the reason for the exception.
try {
Q q = jdbcTemplate.queryForObject(MY_QUERY, new Object[]{id}, new MyMapper());
return q;
} catch (DataAccessException e) {
throw new DaoException("Could not retrieve q with ID: " + qID, e);
}
The method queryForObject only throws DataAccessException or IncorrectResultSizeDataAccessException. You shouldn't catch exception due to database connection failure at this level. However, in your DAO class, you normally have code to establish the dataSource which can be injected by Spring's IoC container. That part of code will throw exception if the database connection fails. You should catch DB failure exception there.