are not permitted to be deserialized at this security level - .net-remoting

Question

I have an application that works everywhere else except between 2 server. Web and App. The Web uses .NET Remoting to the App server.
We have typeFilterLevel= Full and remoting works on Global.asax but not on .aspx pages.
Below is the error.
Raw url: /welcome.aspx
System.Security.SecurityException: Type System.Runtime.Remoting.ObjRef and the types derived from it (such as System.Runtime.Remoting.ObjRef) are not permitted to be deserialized at this security level.
Server stack trace:
at System.Runtime.Serialization.FormatterServices.CheckTypeSecurity(Type t, TypeFilterLevel securityLevel)
at System.Runtime.Serialization.Formatters.Binary.ObjectReader.CheckSecurity(ParseRecord pr)
at System.Runtime.Serialization.Formatters.Binary.ObjectReader.ParseObject(ParseRecord pr)
at System.Runtime.Serialization.Formatters.Binary.__BinaryParser.Run()
at System.Runtime.Serialization.Formatters.Binary.ObjectReader.Deserialize(HeaderHandler handler, __BinaryParser serParser, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
at System.Runtime.Serialization.Formatters.Binary.BinaryFormatter.Deserialize(Stream serializationStream, HeaderHandler handler, Boolean fCheck, Boolean isCrossAppDomain, IMethodCallMessage methodCallMessage)
at System.Runtime.Remoting.Channels.CoreChannel.DeserializeBinaryRequestMessage(String objectUri, Stream inputStream, Boolean bStrictBinding, TypeFilterLevel securityLevel)
at System.Runtime.Remoting.Channels.BinaryServerFormatterSink.ProcessMessage(IServerChannelSinkStack sinkStack, IMessage requestMsg, ITransportHeaders requestHeaders, Stream requestStream, IMessage& responseMsg, ITransportHeaders& responseHeaders, Stream& responseStream)
Exception rethrown at [0]:
at System.Runtime.Remoting.Proxies.RealProxy.HandleReturnMessage(IMessage reqMsg, IMessage retMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at System.Runtime.Remoting.Activation.IActivator.Activate(IConstructionCallMessage msg)
at System.Runtime.Remoting.Messaging.ClientContextTerminatorSink.SyncProcessMessage(IMessage reqMsg)
at System.Runtime.Remoting.Activation.ActivationServices.Activate(RemotingProxy remProxy, IConstructionCallMessage ctorMsg)
at System.Runtime.Remoting.Proxies.RemotingProxy.InternalActivate(IConstructionCallMessage ctorMsg)
at System.Runtime.Remoting.Proxies.RealProxy.PrivateInvoke(MessageData& msgData, Int32 type)
at OnSIS.Common.Utilities..ctor()
at OnSIS.Web.Welcome..ctor()
at ASP.welcome_aspx..ctor()
at __ASP.FastObjectFactory_app_web_welcome_aspx_cdcab7d2_gkkoykn5.Create_ASP_welcome_aspx()
at System.Web.Compilation.BuildManager.CreateInstanceFromVirtualPath(VirtualPath virtualPath, Type requiredBaseType, HttpContext context, Boolean allowCrossApp)
at System.Web.UI.PageHandlerFactory.GetHandlerHelper(HttpContext context, String requestType, VirtualPath virtualPath, String physicalPath)
at System.Web.HttpApplication.MapHttpHandler(HttpContext context, String requestType, VirtualPath path, String pathTranslated, Boolean useAppConfig)
at System.Web.HttpApplication.MapHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute()
at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
The Zone of the assembly that failed was:
MyComputer 2017/07/17 09:43:25
Again this all works perfectly fine exact same code in production servers and a dozen test server. But our Production Support server decided it just does not want to work anymore. I am looking for what to check that could have gone wrong on either the web or app server to cause the above error.
REMOTECONFIG(Webserver)
<?xml version="1.0" encoding="utf-8"?>
<configuration>
<system.runtime.remoting>
<application>
<channels>
<channel ref="http" useDefaultCredentials="true" port="0">
<clientProviders>
<formatter ref="binary" typeFilterLevel="Full" />
</clientProviders>
</channel>
</channels>
<client url="http://10.10.10.10/AppServer">
<activated type="MyApp.Common.Utilities,MyApp.Common" />
</client>
</application>
</system.runtime.remoting>
</configuration>
WEBCONFIG(APPserver)
<?xml version="1.0"?>
<configuration>
<system.runtime.remoting>
<application>
<service>
<activated type="MyApp.Common.Utilities,MyApp.Common"/>
</service>
<channels>
<channel ref="http"/>
<serverProviders>
<formatter ref="binary" typeFilterLevel="Full" />
</serverProviders>
</channels>
</application>
</system.runtime.remoting>
</configuration>

Answer 1

This issue is caused by v6 of the New Relic monitoring agent. Specifically, a new DLL to support tracing for async code (something not previously supported by New Relic's agent).
If uninstalling the agent or reverting to an earlier version isn't possible, the async wrapper can be deleted (C:\Program Files\New Relic.NET Agent\Extensions\NewRelic.Providers.CallStack.AsyncLocal.dll). This will restore .NET Remoting functionality, though at the cost of New Relic not being able to instrument async methods in your code.

Answer 2

After 2 weeks of searching we finally found out what the issues was. Our Operations team installed some monitoring software I believe was New Relic. They did not tell anyone nor did they keep track of or tell anyone what they were doing. We uninstalled the software and the environment is back up and running. This is what happens when you make several changes to an environments all at once. You loose track of what you did and what caused the environment to go down.