Debug AIR Application on Client Machine - debugging

I have an AIR application that is causing me problems on a single client machine, and on a specific user account.
I have a debug interface that I developed to print traces to but apparently you can not get stack traces when not in a debugging environment.
I am seeing "TypeError #2007" with no additional information. I tried to output the error text using e.getStackTrace() and it returns null.
How can I get more information about this error on a client machine?

In this instance I was able to isolate the problem using a bunch of different try..catch statements and a lot of trace statements.
It took a lot of persistence and a lot of back and forth installing many different versions on the client machine and testing.
Obviously not an optimal solution in most cases, but one that worked.

Related

Troubleshooting MVC4 Web API Performance Issues

I have an asp.net mvc4 web api interface that gets about 54k requests a day.
http://myserv.x.com/api/123/getstuff?whatstuff=thisstuff
I have 3 web servers behind a load balancer that are setup to handle the http requests.
On average response times are ~300ms. However, lately something has gone awry (or maybe it has always been there) as there is sporadic behavior of response times coming back in 10-20sec. This would be for the same request hitting the same server directly instead of through the load balancer.
GIVEN:
- System has been passed down to me so there may be gaps with IIS confiuration, etc,.
- Database: SQL Server 2008R2
- Web Servers: Windows Server 2008R2 Enterprise SP1
- IIS 7.5
- Using MemoryCache aggressively with Model and Business Objects with eviction set to 2hrs
- Looked at the logs but really don't see anything significantly relevant
- One application pool...no other LOB applications running on this server
Assumptions & Ask:
Somehow I'm thinking that something is recycling the application pool or IIS worker threads are shutting down and restarting thus causing each new request to warmup and recache itself. It's so sporadic that it's tough to trouble shoot right now. The same request to the same server comes back fast as expected (back to back N requests) since it was cached in about 300ms....but wait about 5-10-20min and that same request to the same server takes 16seconds.
I have limited tracing to go by as these are prod systems so I can only expose so much logging details. Any help and information attacking this or similar behavior somebody else has run into is appreciated. Thx
UPDATE:
The w3wpe.exe process grows to ~3G. Somehow it gets wiped out and the PID changes so itself or something is killing it every 3-4min I see tons of warnings in my webserver (IIS) log:
A process serving application pool 'MyApplication' suffered a fatal
communication error with the Windows Process Activation Service. The
process id was '1732'. The data field contains the error number.
After 4-5 days of assessing IIS and configuration vs internal code issues I finally found the issue with little to no help with windbg or debugdiag IIS tools. Those tools contain so much information even with mini dumps or log trace stacks that they can be red herrings. Best bet was to reproduce it by setting up a "copy intelligently" instance of a production system, which we did not have at the time and took a bit for ops to set something up.
Needless to say the problem had to do with over cacheing business objects. There was one race condition where updates on a certain table were updating an attribute to that corresponding business object (updates were coming from multiple servers) which was causing an OOC stackoverflow that pretty much caused the cacheing to recursively cache itself to death thus causing the w3wp.exe process to die and psuedo-recycle itself. It was one of those edge cases that was incredibly hard to test and repro in a non-production environment.

Using ODBC Trace or Oracle Trace to find cause of error?

I have a third party Windows service which controls/monitors equipment and updates an Oracle database. Their services occasionally report an error about a row/column in the database being "bad" but do not give the underlying database error, and their services need to be restarted and everything is fine. The current suspicion is that something from our applications/services which read/write to those same tables/rows are interfering - i.e. some kind of blocking/locking. I suspect that there is some sort of leak in their system since it happens about once a week, but our systems never need any re-starting like this.
I attempted to have the DBA run a trace run in Oracle (10g), but this managed to make our apps unable to access the Oracle database. Our systems access Oracle in .NET, either using the Oracle ODP client or Microsoft client (older programs) and on this same server (either web apps or services) or from other control workstations. The third-party services connects to Oracle via ODBC on this server. I also attempted to run a ODBC trace (since that would only be activity from the third-party service), but didn't get anything in the trace file at all.
So I'm trying to find a way to either get ODBC tracing working or what I need to look out for so that the Oracle trace doesn't kill my server.
I'm looking for the undserlying error which Oracle is returning to the thrid-party service so I can tell if we are interfering with their access to the data in some way.
If a block in the database is corrupted "Bad" this should show up in the alert logs as an ORA-01578 error. I would search the archive log for the ORA- error and then compare that with the time stamp on the client error being reported. This is making the assumption of the definition of "Bad". It would be better to have the exact error messages posted.
Blanket tracing in the database is a tricky thing as it will tend to affect the performance of your entire application. And leaving it on for an entire week may not be feasible. I have also found in one case (cant remember the exact circumstance) where turning on tracing fixed the error.
One method I have used in the past is to add the sql statement to alter the session and turn on sqltrace. This is predicated on the ability to modify the code in some way. Depending on the application this may or may not be possible.
Another method would be to work with the DBA to identify the session and turn on sql trace for that session. Also if you can identify the offending sql statements and parameter values you may be able to replicate the problem outside the service.
I have found most ORM's avoid passing the ORA- error back. However it is typically logged in the application server layer with the associated ORM error.
I have used these method and variations of these method to trouble shoot errors in the application. I hope this is useful.

"Replay" the steps needed to recreate an error

I am going to create a typical business application that will be used by a few hundred consultants. Normally, the consultants would be presented with an error message with a standard text. As the application will be a complicated one with lots of changes being made to it constantly I would like the following:
When an error message is presented, the user has the option to "send" the error message to the developers. The developers should be able to open the incoming file in i.e. Eclipse and debug the steps of the last 10 minutes of work step by step (one line at a time if they want to). Everything should be transparent, meaning that they for example should be able to see the return values of calls to the database.
Are there any solutions that offer such functionality today, my preferred language is Python or also Java. I know that there will be a huge performance hit because of such functionality, but that is acceptable as this kind of software is not performance sensitive.
It would be VERY nice if the database also had a cronology so that one could query the database for values that existed at the exact time that a specific line of code was run in the application, leading up to the bug.
You should try to use logging, e.g. commit logs from the DB and logging the user interactions with the application, if it is a web application you can start with the log files from the webserver. Make sure that the logfiles include all submitted data such as the complete GET url with parameters and POST with entity body. You can configure the web server to generate such logs when necesary.
Then you build a test client that can parse the log files and re-create all the user interaction that caused the problem to appear. If you suspect race conditions you should log with high precision (ms resolution) and make sure that the test client can run through the same sequences over and over again to stress those critical parts.
Replay (as your title suggests) is the best way to reproduce an error, just collect all the data needed to recreate the input that generated a specific state/situation. Do not focus on internal structures and return values, when it comes to hunting down an error or a bug you should not work in forensic mode, e.g. trying to analyse the cause of the crash by analyzing the wreck, you should crash the plane over and over again and add more and more logging/or use a debugger until you know that goes wrong.

Error 1053: the service did not respond to the start or control request in a timely fashion

I have recently inherited a couple of applications that run as windows services, and I am having problems providing a gui (accessible from a context menu in system tray) with both of them.
The reason why we need a gui for a windows service is in order to be able to re-configure the behaviour of the windows service(s) without resorting to stopping/re-starting.
My code works fine in debug mode, and I get the context menu come up, and everything behaves correctly etc.
When I install the service via "installutil" using a named account (i.e., not Local System Account), the service runs fine, but doesn't display the icon in the system tray (I know this is normal behavior because I don't have the "interact with desktop" option).
Here is the problem though - when I choose the "LocalSystemAccount" option, and check the "interact with desktop" option, the service takes AGES to start up for no obvious reason, and I just keep getting
Could not start the ... service on Local Computer.
Error 1053: the service did not respond to the start or control request in a timely fashion.
Incidentally, I increased the windows service timeout from the default 30 seconds to 2 minutes via a registry hack (see http://support.microsoft.com/kb/824344, search for TimeoutPeriod in section 3), however the service start up still times out.
My first question is - why might the "Local System Account" login takes SOOOOO MUCH LONGER than when the service logs in with the non-LocalSystemAccount, causing the windows service time-out? what's could the difference be between these two to cause such different behavior at start up?
Secondly - taking a step back, all I'm trying to achieve, is simply a windows service that provides a gui for configuration - I'd be quite happy to run using the non-Local System Account (with named user/pwd), if I could get the service to interact with the desktop (that is, have a context menu available from the system tray). Is this possible, and if so how?
Any pointers to the above questions would be appreciated!
After fighting this message for days, a friend told me that you MUST use the Release build. When I InstallUtil the Debug build, it gives this message. The Release build Starts fine.
If you continue down the road of trying to make your service interact with the user's desktop directly, you'll lose: even under the best of circumstances (i.e. "before Vista"), this is extremely tricky.
Windows internally manages several window stations, each with their own desktop. The window station assigned to services running under a given account is completely different from the window station of the logged-on interactive user. Cross-window station access has always been frowned upon, as it's a security risk, but whereas previous Windows versions allowed some exceptions, these have been mostly eliminated in Vista and later operating systems.
The most likely reason your service is hanging on startup, is because it's trying to interact with a nonexistent desktop (or assumes Explorer is running inside the system user session, which also isn't the case), or waiting for input from an invisible desktop.
The only reliable fix for these issues is to eliminate all UI code from your service, and move it to a separate executable that runs inside the interactive user session (the executable can be started using the global Startup group, for example).
Communication between your UI code and your service can be implemented using any RPC mechanism: Named Pipes work particularly well for this purpose. If your communications needs are minimal, using application-defined Service Control Manager commands might also do the trick.
It will take some effort to achieve this separation between UI and service code: however, it's the only way to make things work reliably, and will serve you well in the future.
ADDENDUM, April 2010: Since this question remains pretty popular, here's a way to fix another common scenario that causes "service did not respond..." errors, involving .NET services that don't attempt any funny stuff like interacting with the desktop, but do use Authenticode signed assemblies: disable the verification of the Authenticode signature at load time in order to create Publisher evidence, by adding the following elements to your .exe.config file:
<configuration>
<runtime>
<generatePublisherEvidence enabled="false"/>
</runtime>
</configuration>
Publisher evidence is a little-used Code Access Security (CAS) feature: only in the unlikely event that your service actually relies on the PublisherMembershipCondition will disabling it cause issues. In all other cases, it will make the permanent or intermittent startup failures go away, by no longer requiring the runtime to do expensive certificate checks (including revocation list lookups).
I faced this problem because of a missing framework on the box running my service. The box had .NET 4.0 and the service was written on top of .NET 4.5.
I installed the following download on the box, restarted, and the service started up fine:
http://www.microsoft.com/en-us/download/details.aspx?id=30653
To debug the startup of your service, add the following to the top of the OnStart() method of your service:
while(!System.Diagnostics.Debugger.IsAttached) Thread.Sleep(100);
This will stall the service until you manually attach the Visual Studio Debugger using Debug -> Attach to Process...
Note: In general, if you need a user to interact with your service, it is better to split the GUI components into a separate Windows application that runs when the user logs in. You then use something like named pipes or some other form of IPC to establish communication between the GUI app and your service. This is in fact the only way that this is possible in Windows Vista.
In service class within OnStart method don't do huge operation, OS expect short amount of time to run service, run your method using thread start:
protected override void OnStart(string[] args)
{
Thread t = new Thead(new ThreadStart(MethodName)); // e.g.
t.Start();
}
I'm shooting blind here, but I've very often found that long delays in service startups are directly or indirectly caused by network function timeouts, often when attemting to contact a domain controller when looking up account SIDs - which happens very often indirectly via GetMachineAccountSid() whether you realize it or not, since that function is called by the RPC subsystem.
For an example on how to debug in such situations, see The Case of the Process Startup Delays on Mark Russinovich's blog.
If you are using Debug code as below in your service the problem may arise.
#if(!DEBUG)
ServiceBase[] ServicesToRun;
ServicesToRun = new ServiceBase[]
{
new EmailService()
};
ServiceBase.Run(ServicesToRun);
#else
//direct call function what you need to run
#endif
To fix this, while you build your windows service remove #if condition because it didn't work as it is.
Please use argument for debug mode instead as below.
if (args != null && args.Length > 0)
{
_isDebug = args[0].ToLower().Contains("debug");
}
In my case the problem was missing version of .net framework.
My service used
<startup>
<supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.5" />
</startup>
But .net Framework version of server was 4, so by changing 4.5 to 4 the problem fixed:
<startup>
<supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.0" />
</startup>
Copy the release DLL or get the dll from release mode rather than Debug mode and paste it to installation folder,,it should work
I was running into a similar problem with a Service I was writing. It worked fine then one day I started getting the timeout on Start errors. It happened in one &/or both Release and Debug depending on what was going on. I had instantiated an EventLogger from System.Diagnostics, but whatever error I was seeing must have been happening before the Logger was able to write...
If you are not aware of where to look up the EventLogs, in VS you can go to your machine under the Server Explorer. I started poking around in some of the other EventLogs besides those for my Service. Under Application - .NETRuntime I found the Error logs pertinent to the error on startup. Basically, there were some exceptions in my service's constructor (one turned out to be an exception in the EventLog instance setup - which explained why I could not see any logs in my Service EventLog). On a previous build apparently there had been other errors (which had caused me to make the changes leading to the error in the EventLog set up).
Long story short - the reason for the timeout may be due to various exceptions/errors, but using the Runtime EventLogs may just help you figure out what is going on (especially in the instances where one build works but another doesn't).
Hope this helps!
Install the debug build of the service and attach the debugger to the service to see what's happening.
I want to echo mdb's comments here. Don't go this path. Your service is not supposed to have a UI... "No user interaction" is like the definining feature of a service.
If you need to configure your service, write another application that edits the same configuration that the service reads on startup. But make it a distinct tool -- when you want to start the service, you start the service. When you want to configure it, you run the configuration tool.
Now, if you need realtime monitoring of the service, then that's a little trickier (and certainly something I've wished for with services). Now you're talking about having to use interprocess communications and other headaches.
Worst of all, if you need user interaction, then you have a real disconnect here, because services don't interact with the user.
In your shoes I would step back and ask why does this need to be a service? And why does it need user interaction?
These two requirements are pretty incompatible, and that should raise alarms.
I had this problem and it drove me nuts for two days…
If your problem similar to mine:
I have settings “User settings” in my windows service, so the service can do self-maintenance, without stopping and starting the service. Well, the problem is with the “user settings”, where the config file for these settings is saved in a folder under the user-profile of the user who is running the windows service under the service-exe file version.
This folder for some reason was corrupted. I deleted the folder and service start working back again happily as usual…
I had this problem, it took about a day to fix. For me the problem was that my code skipped the "main content" and effectively ran a couple of lines then finished. And this caused the error for me. It is a C# console application which installs a Windows Service, as soon as it tried to run it with the ServiceController (sc.Run() ) then it would give this error for me.
After I fixed the code to go to the main content, it would run the intended code:
ServiceBase.Run(new ServiceHost());
Then it stopped showing up.
As lots of people have already said, the error could be anything, and the solutions people provide may or may not solve it. If they don't solve it (like the Release instead of Debug, adding generatePublisherEvidence=false into your config, etc), then chances are that the problem is with your own code.
Try and get your code to run without using sc.Run() (i.e. make the code run that sc.Run() would have executed).
This problem usually occurs when there is some reference missing on your assembly and usually the binding fails at the run time.
to debug put Thread.Sleep(1000) in the main(). and put a break point in the next line of execution.
Then start the process and attach the debugger to the process while it is starting. Press f5 after it hit the break point. It will throw the exception of missing assembly or reference.
Hopefully this will resolve this error.
Once try to run your exe file. I had the same problem, but when I ran it direct by double click on the exe file, I got a message about .Net framework version, because I was released the service project with a framework which it wasn't installed on target machine.
Took me hours, should have seen the event viewer get_AppSettings().
A change in the app config, caused the problem.
Adding 127.0.0.1 crl.microsoft.com to the "Hosts" file solved our issue.
My issue was due to target framework mentioned in windows service config was
<startup>
<supportedRuntime version="v4.0" sku=".NETFramework,Version=v4.6"/>
</startup>
and my server in which I tried to install windows service was not supported for this .Net version.
Changing which , I could able to resolve the issue.
I had a similar issue, steps I followed:
Put a Debugger.Launch() in the windows service constructor
Followed step by step to see where it got stuck
My issue wasn't due to any error.
I had a BlockingCollection.GetConsumingEnumerable() in the way that caused the windows service to wait.
I had this problem too. I made it to work by changing Log On account to Local System Account. In my project I had it setup to run as Local Service account. So when I installed it, by default it was using Local Service. I'm using .net 2.0 and VS 2005. So installing .net 1.1 SP1 wouldn't have helped.
Both Local System Account and Local Service would not work for me, i then set it to Network Service and this worked fine.
In my case, I had this trouble due to a genuine error. Before the service constructor is called, one static constructor of member variable was failing:
private static OracleCommand cmd;
static SchedTasks()
{
try
{
cmd = new OracleCommand("select * from change_notification");
}
catch (Exception e)
{
Log(e.Message);
// "The provider is not compatible with the version of Oracle client"
}
}
By adding try-catch block I found the exception was occuring because of wrong oracle version. Installing correct database solved the problem.
I also faced similar problem and found that there was issue loading assembly. I was receiving this error immediately when trying to start the service.
To quickly debug the issue, try to run service executable via command prompt using ProcDump http://technet.microsoft.com/en-us/sysinternals/dd996900. It shall provide sufficient hint about exact error.
http://bytes.com/topic/net/answers/637227-1053-error-trying-start-my-net-windows-service helped me quite a bit.
This worked for me. Basically make sure the Log on user is set to the right one. However it depends how the account infrastructure is set. In my example it's using AD account user credentials.
In start up menu search box search for 'Services'
-In Services find the required service
-right click on and select the Log On tab
-Select 'This account' and enter the required content/credentials
-Ok it and start the service as usual
In case you have a windows form used for testing, ensure that the startup object is still the service and not the windows form
We have Log4Net configured to log to a database table. The table had grown so large that the service was timing out trying to log messages.
open the services window as administrator,Then try to start the service.That worked for me.
Build project in Release Mode.
Copy all Release folder files to source path.
Execute Window service using command prompt window in administrative access.
Never delete files from source path.
At lease this works for me.
Release build did not work for me, however, I looked through my event viewer and Application log and saw that the Windows Service was throwing a security exception when it was trying to create an event log. I fixed this by adding the event source manually with administration access.
I followed this guide from Microsoft:
open registry editor, run --> regedit
Locate the following registry subkey:
HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\Eventlog\Application
Right-click the Application subkey, point to New, and then click Key.
Type event source name used in your windows service for the key name.
Close Registry Editor.

Meaning/cause of RPC Exception 'No interfaces have been exported.'

We have a fairly standard client/server application built using MS RPC. Both client and server are implemented in C++. The client establishes a session to the server, then makes repeated calls to it over a period of time before finally closing the session.
Periodically, however, especially under heavy load conditions, we are seeing an RPC exception show up with code 1754: RPC_S_NOTHING_TO_EXPORT.
It appears that this happens in the middle of a session. The user is logged on for a while, making successful calls, then one of the calls inexplicably returns this error. As far as we can tell, the server receives no indication that anything went wrong - and it definitely doesn't see the call the client made.
The error code appears to have permanent implications, as well. Having the client retry the connection doesn't work, either. However, if the user has multiple user sessions active simultaneously between the same client and server, the other connections are unaffected.
In essence, I have two questions:
Does anyone know what RPC_S_NOTHING_TO_EXPORT means? The MSDN documentation simply says: "No interfaces have been exported." ... Huh? The session was working fine for numerous instances of the same call up until this point...
Does anyone have any ideas as to how to identify the real problem? Note: Capturing network traffic is something we would rather avoid, if possible, as the problem is sporadic enough that we would likely go through multiple gigabytes of traffic before running into an occurrence.
Capturing network traffic would be one of the best ways to tackle this issue. If you can't do that, could you dump the client process and debug with WinDBG or Visual Studio? Perhaps compare a dump when operating normally versus in the error state?

Resources