Azure AppFabric caching. ErrorCode<ERRCA0017>:SubStatus<ES0006> - asp.net-mvc-3

there is a very strange thing going on
i have a web site made in MVC 3 deployed in an Azure WebRole
it was running fine and originally was made in VS2010
now i use VS2012
i decided to make some minor changes and it appeared to be working but keeps break down at random intervals
this is what is happening
after random intervals i get this error ErrorCode<ERRCA0017>:SubStatus<ES0006>
after 2-3 minutes of getting this error sites appears to be working fine
i havent changed anything from the previous version of the site related to caching
this error does nt happens when i test it in staging or in local environment
no where in my code im accessing this cache programmatically
i have copy-pasted entire configuration as it is from the azure portal and i know it it correct because these are the same configuration settings that were working before
i just using Azure AppFabric Caching because i need to save sessions some where. so the only place it is ever mentioned is in my websites web config and as i have told all the configuration can be assumed to be correct.
this is the stacktrace if this can be of some help
[DataCacheException: ErrorCode<ERRCA0017>:SubStatus<ES0006>:There is a temporary failure. Please retry later. (One or more specified cache servers are unavailable, which could be caused by busy network or servers. For on-premises cache clusters, also verify the following conditions. Ensure that security permission has been granted for this client account, and check that the AppFabric Caching Service is allowed through the firewall on all cache hosts. Also the MaxBufferSize on the server must be greater than or equal to the serialized object size sent from the client.)]
Microsoft.ApplicationServer.Caching.DataCache.ThrowException(ResponseBody respBody) +616
Microsoft.ApplicationServer.Caching.DataCache.ExecuteAPI(RequestBody reqMsg, IMonitoringListener listener) +29
Microsoft.ApplicationServer.Caching.DataCache.InternalGetAndLock(String key, TimeSpan timeout, DataCacheLockHandle& lockHandle, String region, Boolean lockKey, IMonitoringListener listener) +227
Microsoft.ApplicationServer.Caching.<>c__DisplayClass78.<GetAndLock>b__77() +103
Microsoft.ApplicationServer.Caching.DataCache.GetAndLock(String key, TimeSpan timeout, DataCacheLockHandle& lockHandle) +258
Microsoft.Web.DistributedCache.<>c__DisplayClass31`1.<PerformCacheOperation>b__30() +19
Microsoft.Web.DistributedCache.DataCacheRetryWrapper.PerformCacheOperation(Action action) +208
Microsoft.Web.DistributedCache.DataCacheForwarderBase.GetAndLock(String key, TimeSpan timeout, DataCacheLockHandle& lockHandle) +190
Microsoft.Web.DistributedCache.BlobBasedSessionStoreProvider.GetItem(HttpContextBase context, String id, Boolean acquireWriteLock, Boolean& locked, TimeSpan& lockAge, Object& lockId, SessionStateActions& actions) +593
Microsoft.Web.DistributedCache.DistributedCacheSessionStateStoreProvider.GetItemExclusive(HttpContext context, String id, Boolean& locked, TimeSpan& lockAge, Object& lockId, SessionStateActions& actions) +125
System.Web.SessionState.SessionStateModule.GetSessionStateItem() +178
System.Web.SessionState.SessionStateModule.BeginAcquireState(Object source, EventArgs e, AsyncCallback cb, Object extraData) +1076
System.Web.AsyncEventExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() +115
System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) +375
UPDATE:-
suddenly i discovered a new error which seems to me as one of the inner errors many sites talk about
error description
A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond [IP-Address]:[PORT]
and this is the new stack trace
[SocketException (0x274c): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond [IP-ADDRESS]:[PORT]]
System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress) +305
System.Net.Sockets.Socket.Connect(EndPoint remoteEP) +162
System.ServiceModel.Channels.SocketConnectionInitiator.Connect(Uri uri, TimeSpan timeout) +774
[EndpointNotFoundException: Could not connect to [SOCKET-ENDPOINT]. The connection attempt lasted for a time span of 00:00:21.0308462. TCP error code 10060: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond 111.221.89.111:22233. ]
System.Runtime.Remoting.Proxies.RealProxy.EndInvokeHelper(Message reqMsg, Boolean bProxyCase) +518
System.Runtime.Remoting.Proxies.RemotingProxy.Invoke(Object NotUsed, MessageData& msgData) +401
Microsoft.ApplicationServer.Caching.OpenDelegate.EndInvoke(IAsyncResult result) +0
Microsoft.ApplicationServer.Caching.ChannelContainer.Opened(IAsyncResult ar) +122

There was an interesting blog post about this error over a year ago: Azure AppFabric Caching - ErrorCode:SubStatus : What to do?.
Before intializing the DataCacheFactory modify the
DataCacheFactoryConfiguration.ChannelOpenTimeout of the
dataCacheFactoryConfiguration instance that you are using to a larger
value of 2 minutes. (can't use configuration to set the value, since
20 seconds in the allowed limit) . Also, this recommendation is only
for debugging purposes and ideally should not be needed in production
environments. This lets the underlying layer to respond back with the actual error
that occurred. Now, if you check the inner exception, you should see
what caused the request to fail.
Please follow these steps to find the inner exception which will explain why the request fails.
Note: If this is caused by a transient fault, you should consider using the Transient Fault Handling Application Block (which also supports AppFabric Cache) in order to implement a retry policy.

Related

Connection refused error between containers when running a simple project with docker compose in visual studio 2019

I am following this tutorial step by step exactly. Very simple one having .net core frontend and backend(webapi) projects.
But the communication between the frontend and the api does not happen. I always get
An unhandled exception occurred while processing the request.
SocketException: Connection refused
System.Net.Http.ConnectHelper.ConnectAsync(string host, int port, CancellationToken
cancellationToken)
HttpRequestException: Connection refused
System.Net.Http.ConnectHelper.ConnectAsync(string host, int port, CancellationToken
cancellationToken)
Looks like I am missing something trivial here. But no matter what I tweak, I get the same communication exception. Driving me nuts now.
I have uploaded the full solution to github here, in case if you want to take a look.
Its a 3.1 project and ensured as closed to the tutorial as possible. Even added a bridge network to the compose file after looking at some posts.
Here are some screen shots.
Finally here is the exception message.
UPDATE
So based on the suggestion of Exploding Kitten in the comment, I had removed the call to
// app.UseHttpsRedirection();
in startup.cs of the Api project and things started working fine.
Or we can have a conditional as follows.
if (Configuration["DOTNET_RUNNING_IN_CONTAINER"] != "true")
app.UseHttpsRedirection();
We can detect if an app is running in the container using the env var DOTNET_RUNNING_IN_CONTAINER. Take a look at this.

Why are some WCF Named Pipe clients getting a TimeoutException

I have a WCF name pipe service, to communicate 2 processes on windows.
I've tested it on my machine and it worked, I've developed an installer for it, and it's working for most users, but some users are getting a TimeoutException with this message:
"request operation sent to net.pipe://localhost/myService did not receive a reply within the configured timeout (00:00:10). The time allotted to this operation may have been a portion of a longer timeout. This may be because the service is still processing the operation or because the service was unable to send a reply message. Please consider increasing the operation timeout (by casting the channel/proxy to IContextChannel and setting the OperationTimeout property) and ensure that the service is able to connect to the client."
What can be the reason for this error? Is it a security issue? Where can I set security settings for named pipes?
Thanks
Some users may not receive a reply in time due to network problems, which may cause a timeout.
You can configure the timeout setting in configuration file by setting the send and receive timeout on binding element.
Apply the binding configuration on both service and client-side:
<netNamedPipeBinding>
<binding sendTimeout="00:20:00" receiveTimeout="00:20:00">
<reliableSession inactivityTimeout="00:20:00"/>
</binding>
</netNamedPipeBinding>

CRM OnPrem--> Plugin--> Connect to External (SOAP) Webservice Error with connection

I am trying to connect from CRM Plugin to External SOAP API. It gives me error.
I am using Isolation Mode as None.
When I connect the same API using Console APP it works fine. I executed the Console APP from Server and it connects and responds Fine.
I googled a bit got some leads but probably I lack capability to understand/ pinpoint the issue.
Code Snippet below
WebRequest request = WebRequest.Create("URL");
request.Method = "GET";
//request.Credentials = new NetworkCredential("USERNAME", "PASSWORD");
WebResponse response = request.GetResponse();
HttpWebResponse webresponse = (HttpWebResponse)response;
if (webresponse.StatusCode == HttpStatusCode.OK)
{
tracing.Trace($" Resposne is correct i.e OK");
}
When I debug using Plugin Registration Tool, Tool breaks up and exits.
Error from Tracing:
A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.ServicePoint.ConnectSocketInternal(Boolean connectFailure, Socket s4, Socket s6, Socket& socket, IPAddress& address, ConnectSocketState state, IAsyncResult asyncResult, Exception& exception)
--- End of inner exception stack trace ---
at System.Net.HttpWebRequest.GetResponse()
Reason why it was not working:
Customer (OnPrem) uses Proxy to connect to Web.
When I was running Console App on server it was running on my Logged on context (outside CRM) i.e. user who logged on Server.
However, When I tried to call website from CRM may it be from Isolation Mode = None OR Sandbox, The services that runs under specific user did not have proxy added to them.
For Example running plugin under Isolation Mode = NONE i.e. directly under the context of APPPool Service "crmtestappserv" did not have proxy and even using mere Internet Explored under the context of same user was not able to open google.com unless proxies were Added.
Steps Taken to Solve the issue: In our plugin Code, when we make an HTTP request we add proxy to the request. This solved our issue.

Test Rig Azure Controller cannot connect to Visual Studio Client

I am deploying a test rig as per the directions from http://msdn.microsoft.com/en-us/library/windowsazure/hh674491.aspx with the controller and agents hosted on windows azure. I have followed all the directions properly and my VS client is able to see the controller and agents. However when i execute a load test, the test remains in an pending state and I see the below error in the Controller's eventlog. My problem is similar to one found at: http://social.msdn.microsoft.com/Forums/en-US/windowsazureconnectivity/thread/4040619f-5667-4445-804a-84f37d823c9f/. Please help!
(QTController.exe, PID 692, Thread 11) ControllerDeployment.DoDeployment: System.Net.Sockets.SocketException (0x80004005): A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond [xxx:xxx:xxxx:xxx:xxxx:xxxx:xxxx:220]:6915
Server stack trace:
at System.Net.Sockets.Socket.DoConnect(EndPoint endPointSnapshot, SocketAddress socketAddress)
at System.Net.Sockets.Socket.Connect(EndPoint remoteEP)
at System.Runtime.Remoting.Channels.RemoteConnection.CreateNewSocket(EndPoint ipEndPoint)
at System.Runtime.Remoting.Channels.RemoteConnection.CreateNewSocket()
at System.Runtime.Remoting.Channels.SocketCache.GetSocket(String machinePortAndSid, Boolean openNew)
at System.Runtime.Remoting.Channels.Tcp.TcpClientTransportSink.SendRequestWithRetry(IMessage msg, ITransportHeaders requestHeaders, Stream requestStream)
at System.Runtime.Remoting.Channels.Tcp.TcpClientTransportSink.ProcessMessage(IMessage msg, ITransportHeaders requestHeaders, Stream requestStream, ITransportHeaders& responseHeaders, Stream& responseStream)
at System.Runtime.Remoting.Channels.BinaryClientFormatterSink.SyncProcessMessage(IMessage msg)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at Microsoft.VisualStudio.TestTools.Common.FileCopyService.get_DeploymentFlags()
at Microsoft.VisualStudio.TestTools.Controller.ControllerExecution.b__6(FileCopyService fileCopyService)
at Microsoft.VisualStudio.TestTools.Execution.RemoteObjectContainer1.InvokeAsRemoteUser(Action1 invoke)
at Microsoft.VisualStudio.TestTools.Controller.ControllerExecution.DoDeployment()
Based on the error message it looks like the controller is not able to talk to the agent. Have you set up a connect group as mentioned below:
"http://msdn.microsoft.com/en-us/library/windowsazure/hh674497.aspx"
Creating a Connect Group
The Connect feature of Azure allows you to create a Virtual Private Network. Members of the group can include on-premise computers as well as Azure role instances. For the Load Test solution, the Connect group allows communication between the test controller and the agents.
Attempted this as well, and couldn't get it to work. Ended up having to move the Controller into Azure as well.
I followed the documentation as far as what ports the agent needed, and opened those up in the Azure firewall. However, after doing a network capture, IIRC it also used some dynamic ports that weren't going to work to open up.
So I found that I had to move Visual Studio, the Load Agent Controller, and the Load Agents themselves all into Azure to utilize this in the cloud.

Impersonation okay local but not on server

I am having a problem with I think impersonation, but I could be mistaken.
I have an IBM XMS consumer, consuming data from a queue. Works fine. In order to be granted access to the queue the service needs to be running as specific user, for this I have implemented .Net impersonation. On all systems I am able to get the queue running, the problem occurs when there is a need to restart.
To test restart I have placed a simple if(messages == 500) throw... which triggers cancellation, disconnection and reconnection.
I installed the application as a Windows service to my local machine, it connects (using impersonation), reaches 500, throws, disconnects, reconnects and continues to process messages. However when I run the same code on our server it fails while trying to create a queue connection. Here is the error:
7765 [20] ERROR SIS (null) - Unable to connect to Websphere MQ
IBM.XMS.XMSException: CWSMQ0006E: An exception was received during the
call to the method ConnectionFactory.CreateConnection:
System.ComponentModel.Win32Exception (0x80004005): Access is denied
at System.Diagnostics.ProcessManager.OpenProcess(Int32 processId,
Int32 access, Boolean throwIfExited) at
System.Diagnostics.NtProcessManager.GetModuleInfos(Int32 processId,
Boolean firstModuleOnly) at
System.Diagnostics.ProcessManager.GetModuleInfos(Int32 processId)
at System.Diagnostics.Process.get_Modules() at
IBM.WMQ.CommonServices.TraceEnvironment() at
IBM.WMQ.CommonServices.CreateCommonServices() at
IBM.WMQ.CommonServices.TraceEnabled() at IBM.WMQ.MQBase..ctor()
at IBM.WMQ.Nmqi.Phconn..ctor(NmqiEnvironment env) at
IBM.WMQ.Nmqi.NmqiEnvironment.NewPhconn() at
IBM.XMS.Client.WMQ.WmqConnection..ctor(NmqiEnvironment nmqiEnv, NmqiMQ
nmqiMQ, XmsPropertyContext connectProps) at
IBM.XMS.Client.WMQ.Factories.WmqConnectionFactory.CreateV7ProviderConnection(XmsPropertyContext
connectionProps). During execution of the specified method an
exception was thrown by another component. See the linked exception
for more information. at
IBM.XMS.Client.WMQ.Factories.WmqConnectionFactory.CreateProviderConnection(XmsPropertyContext
connectionProps) at
IBM.XMS.Client.Impl.XmsConnectionFactoryImpl.CreateConnection(String
userID, String password) at
IBM.XMS.Client.Impl.XmsConnectionFactoryImpl.CreateConnection() at
Spin.TradingServices.DataAcquisition.Domain.DataProcessing.SIS.SISMQConnection.Connect(ImpersonationCredentials
impersonationCredentials)
Linked Exception : System.ComponentModel.Win32Exception (0x80004005):
Access is denied at
System.Diagnostics.ProcessManager.OpenProcess(Int32 processId, Int32
access, Boolean throwIfExited) at
System.Diagnostics.NtProcessManager.GetModuleInfos(Int32 processId,
Boolean firstModuleOnly) at
System.Diagnostics.ProcessManager.GetModuleInfos(Int32 processId)
at System.Diagnostics.Process.get_Modules() at
IBM.WMQ.CommonServices.TraceEnvironment() at
IBM.WMQ.CommonServices.CreateCommonServices() at
IBM.WMQ.CommonServices.TraceEnabled() at IBM.WMQ.MQBase..ctor()
at IBM.WMQ.Nmqi.Phconn..ctor(NmqiEnvironment env) at
IBM.WMQ.Nmqi.NmqiEnvironment.NewPhconn() at
IBM.XMS.Client.WMQ.WmqConnection..ctor(NmqiEnvironment nmqiEnv, NmqiMQ
nmqiMQ, XmsPropertyContext connectProps) at
IBM.XMS.Client.WMQ.Factories.WmqConnectionFactory.CreateV7ProviderConnection(XmsPropertyContext
connectionProps)
Looks like the user id being impersonated does not have enough permissions and not a problem with XMS.
Here is a link that may be of help to you.
Access is denied at System.Diagnostics.Process.StartWithCreateProcess(ProcessStartInfo startInfo)
and this also.
It seems that on the IBM site there is a fix available for this error:
http://www-01.ibm.com/support/docview.wss?uid=swg1IC90674
(please use the Wayback Machine https://archive.org/web/ if the URL is no longer available)
They also say that there is a local fix:
Granting "Query process information and read process virtual
memory" rights to the impersonated user so that OpenProcess
succeeds.

Resources