I'm finding a lot of OperationCanceledExceptions in my API log. I think it is related to high load conditions, but that could just be because under higher load there's a higher number of calls and therefore a higher number of exceptions. The stack trace of the exception does not reach any controller code; it passes through some of my OWIN middleware but dies before it reaches any controller. And it's the same stack trace for many calls to different API endpoints:
System.OperationCanceledException: The operation was canceled.
at System.Threading.CancellationToken.ThrowOperationCanceledException()
at System.Net.Http.HttpContentExtensions.<ReadAsAsyncCore>d__0`1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.ModelBinding.FormatterParameterBinding.<ExecuteBindingAsyncCore>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Web.Http.Controllers.HttpActionBinding.<ExecuteBindingAsyncCore>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Controllers.ActionFilterResult.<ExecuteAsync>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Filters.AuthorizationFilterAttribute.<ExecuteAuthorizationFilterAsyncCore>d__2.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__1.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at MyDomain.Api.Handlers.ApiCultureHandler.<SendAsync>d__0.MoveNext() in D:\Bamboo\xml-data\build-dir\API-R5V3-JOB1\src\Api\Handlers\ApiCultureHandler.cs:line 50
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.HttpServer.<SendAsync>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at System.Web.Http.Owin.HttpMessageHandlerAdapter.<InvokeCore>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Owin.Cors.CorsMiddleware.<Invoke>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Owin.Host.SystemWeb.IntegratedPipeline.IntegratedPipelineContextStage.<RunApp>d__5.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Owin.Security.Infrastructure.AuthenticationMiddleware`1.<Invoke>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at Microsoft.Owin.Security.Infrastructure.AuthenticationMiddleware`1.<Invoke>d__0.MoveNext()
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at DependencyResolution.Api.Middleware.AuditLogMiddleware.<Invoke>d__1.MoveNext() in D:\Bamboo\xml-data\build-dir\API-R5V3-JOB1\src\DependencyResolution\Api\Middleware\AuditLogMiddleware.cs:line 23
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at DependencyResolution.Api.Middleware.ShardingMiddleware.<Invoke>d__2.MoveNext() in D:\Bamboo\xml-data\build-dir\API-R5V3-JOB1\src\DependencyResolution\Api\Middleware\ShardingMiddleware.cs:line 77
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at DependencyResolution.Api.Middleware.StructureMapMiddleware.<Invoke>d__2.MoveNext() in D:\Bamboo\xml-data\build-dir\API-R5V3-JOB1\src\DependencyResolution\Api\Middleware\StructureMapMiddleware.cs:line 29
--- End of stack trace from previous location where exception was thrown ---
at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)
at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)
at DependencyResolution.Api.Middleware.ExceptionHandlerMiddleware.<Invoke>d__3.MoveNext() in D:\Bamboo\xml-data\build-dir\API-R5V3-JOB1\src\DependencyResolution\Api\Middleware\ExceptionHandlerMiddleware.cs:line 33
This is very disturbing, as I'm not sure how to debug this.
Additional possibly useful information: at the time these errors were happening, one of my API servers was exhibiting what I call the "Bart Simpson" pattern in CPU usage. CPU would shoot up to the 95-100% range for about 20-30 seconds, then it would drop off to 5-10% for another 20-30 seconds, then back up again. The cycle repeated indefinitely. Because we were in production, I didn't have time to investigate much further, I killed that server and let our autoscaling rules bring up a new server, which then worked fine. But I have no guarantee that the Bart Simpson pattern won't affect any other API servers we bring up, and I have no idea what might cause it.
Any ideas?
Since there is no other error in your code in the call stack, this is likely a false alarm.
If the client closes the connection before the server responds, the operation will be cancelled.
Let's say, for example, that your client has a heartbeat endpoint that it invokes every 5 seconds, and the call takes 250 ms. Every time a client shuts down, there is a 5% chance of getting an operation cancelled. There are many normal scenarios where a connection may be cut off by the client before a response is received. OperationCanceledException by itself should not be a cause for alarm. However, do consider the frequency in relation to the number of clients and your specific application as well.
As discussed, it is possible that somewhere, in one of those custom middle-ware listed in the stack trace, there may be a mixing of async await and blocking calls (.Result or .Wait()) that may be leading to a deadlock that is causing the operation to hang, timeout and cancel.
The stack trace snippet has already provided a list of possible suspects.
It is suggested to start by inspecting the Invoke of the listed middle ware that handle the request pipeline.
Related
I want skip all exceptions (using AlwaysSkipPolicy) and then handle all skipped exceptions with errors in StepListener.
I want create summary message at the end of step with written/read items and when any exception occured then about how many and with what exception that items was skipped.
When I using skip policy always true + step listener, I have 0 exceptions in "failureExceptions". When I turned off skip policy i got exception there but then when exception occured, job stoping.
SkipListener is what you are looking for. It allows you to intercept skipped items during all phases of a chunk-oriented step (ie read, process and write). This listener gives you access to the skipped item and the exception that caused it to be skipped, so you should be able to implement the reporting you need with this listener.
To debug possible issues with some new method, I'd like to log the stack trace upon encountering a StandardError.
In my case the stack traces are extremely long and only the last few calls are of importance to me (to debug a possible issue, I need to know who was the caller).
Is there a way to log a truncated version of the stack trace association with the exception?
E.g.
def some_unreliable_method
begin
# do unreliable things
rescue => e
log(
error_message: e.message,
backtrace: e.backtrace # the full stack trace is too long
)
# recover from exception
end
end
e.backtrace is just array and you can get only last N numbers of backtrace's lines:
# get last 10 lines
e.backtrace[0, 10].join("\n\t")
Instead of log(error_message: e.message, backtrace: e.backtrace), just logger.error(e)
If that's still too long, take a look at config/initializers/backtrace_silencers.rb. It's well documented.
I'm trying to figure out whether using Window's
SetUnfilteredExceptionHandler will catch every exception in the process, or whether AddVectoredExceptionHandler will catch it before.
It seems that MSDN doesn't provide any info about the priority or the order of execution.
the first is called Vectored Exception Handling (VEH)
Vectored handlers are called in the order that they were added, after
the debugger gets a first chance notification, but before the system
begins unwinding the stack.
then (if exception not handled) called Frame-based Exception Handling (SEH)
the last (top level) exception handler in SEH can be supersede by SetUnhandledExceptionFilter
Enables an application to supersede the top-level exception handler of
each thread of a process.
lpTopLevelExceptionFilter is called last
When my WebApi project has an exception, the browser get's a response with the stack trace!
This is happening even when compilation debug set to false, how can I prevent this?
e.g: {"message":"An error has occurred.","exceptionMessage":"Object reference not set to an instance of an object.","exceptionType":"System.NullReferenceException","stackTrace":" at MvcApi.Helpers.ApiErrorHandler.OnException(HttpActionExecutedContext context) in c:\\dev\\Foundation\\Foundation.Api\\Helpers\\Filters\\ApiErrorHandler.cs:line 16\r\n at System.Web.Http.Filters.ExceptionFilterAttribute.OnExceptionAsync(HttpActionExecutedContext actionExecutedContext, CancellationToken cancellationToken)\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Tracing.Tracers.ExceptionFilterAttributeTracer.<>c__DisplayClass4.<<OnExceptionAsyncCore>b__1>d__6.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Tracing.ITraceWriterExtensions.<TraceBeginEndAsyncCore>d__24.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Filters.ExceptionFilterAttribute.<ExecuteExceptionFilterAsyncCore>d__0.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Controllers.ExceptionFilterResult.<ExecuteAsync>d__0.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Tracing.Tracers.HttpControllerTracer.<ExecuteAsyncCore>d__5.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Web.Http.Tracing.ITraceWriterExtensions.<TraceBeginEndAsyncCore>d__181.MoveNext()\r\n--- End of stack trace from previous location where exception was thrown ---\r\n at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task)\r\n at System.Runtime.CompilerServices.TaskAwaiter1.GetResult()\r\n at System.Web.Http.Dispatcher.HttpControllerDispatcher.<SendAsync>d__0.MoveNext()"}
The error details are driven by the the IncludeErrorDetailPolicy enumeration on HttpConfiguration. By default the value of this is LocalOnly so you should be seeing these detailed exception message only in local environments. Are you noticing a different behavior?
I am having big problems in fetching exception details from Remote EJBs...
I have a pipeline pattern implemented with Remote EJBs, where one EJB calls other EJBs to do certain tasks. What I can't figure out is how to get more meaningful exceptions when some problem occures.
Here is an example of a "very useful" output I typically get from the EJB container (Glassfish):
2010-11-18 12:22:36,974 DEBUG [CustomerDetectionFilter.java:48 [eef819a4debb613d0670dab35c39] - Checking customer details ...
2010-11-18 12:22:37,026 ERROR [ServiceCore.java:104] [eef819a4debb613d0670dab35c39] - nested exception is: java.rmi.ServerException: RemoteException occurred in server thread; nested exception is:
java.rmi.RemoteException: null; nested exception is:
javax.transaction.TransactionRolledbackException: CORBA TRANSACTION_ROLLEDBACK 9998 Maybe; nested exception is:
org.omg.CORBA.TRANSACTION_ROLLEDBACK: vmcid: 0x2000 minor code: 1806 completed: Maybe
javax.ejb.EJBException: nested exception is: java.rmi.ServerException: RemoteException occurred in server thread; nested exception is:
java.rmi.RemoteException: null; nested exception is:
javax.transaction.TransactionRolledbackException: CORBA TRANSACTION_ROLLEDBACK 9998 Maybe; nested exception is:
org.omg.CORBA.TRANSACTION_ROLLEDBACK: vmcid: 0x2000 minor code: 1806 completed: Maybe
at com.my.remote._CustomerDetectionFilterRemote_Wrapper.execute(com/my/remote/_CustomerDetectionFilterRemote_Wrapper.java) ~[project-common.jar:na]
So in this case I would know that this is the cause:
java.rmi.RemoteException: null
... but I have no way of getting more useful information for debugging.
Can you recommend a way of excepting handling where the caller of remote EJB needs to get more details about the problem that occured? AppServer is Glassfish 2.1.
Thank you,
Bozo
There are various ways for handling exceptions. We can configure them to get most information about root cause from it.
Below link might provide useful information & topic is discussed briefly.
Best practices in EJB exception handling - http://www.ibm.com/developerworks/java/library/j-ejbexcept.html