I need to debug a dll that's getting loaded on a specific application pool process. I'm using WinDbg, and so far I have successfully found the correct w3wp.exe process. Problem is, the app pool recycles itself, sometimes before it reaches what I want to find (a very elusive second chance exception). Then, I need to start over.
How do I configure WinDbg to automatically attach to a w3wp.exe process that's connected to a specific app pool, every time it starts?
Another solution would be to get the crash dump - I tried using ADPlus.exe for this but it also needs to be started on a process and I didn't find a way to re-run it automatically on only the process I need AND every time it starts.
So to sum it up, I need a way to get a crash dump from a w3wp.exe process that's connected to a specific IIS App Pool when it crashes on a second chance exception, while the process gets restarted once in a while (not enough time for me to run the debug tools manually each time).
Eventually, I discovered this awesome tool called DebugDiag, which is an official Microsoft tool. It has a nice interface that allows collecting dumps on certain events, such as exceptions, and creates very useful logs.
On top of that - it has a special section specifically for IIS debugging, which allowed me to select the app pool I was interested in.
Download available here.
Related
Very simple question from new user of multiple processes debugging.
Whenever I read MSDN articles regarding this topic, I read:
"You can attach to multiple processes when you are debugging, but only one process is active in the debugger at any given time"
If you can debug only single process actively, seemingly attaching multiple processes is not very useful if I can only activate a single process in the debugger.
Please correct me if I am not understanding the statement correct.
It means that while debugging, VS studio will switch between processes being debugged to gather information needed for debugging, otherwise would be a mess to understand what is happening to a process if multiple processes information are shown together.
An example to see it happening is creating a solution with two web applications and start debugging, VS itself will run both and attach to then, if you put a breakpoint in one endpoint of each app and call one endpoint, let's say here on app1, VS will stop on that breakpoint, and then call the endpoint on app2 before continuing the debugging on app1, visual Studio Will only switch to the other breakpoint when the first one gets released, than VS Will switch to app2 breakpoint.
We are trying to troubleshoot a nasty problem on a production server where the server will start misbehaving after running for awhile.
Diagnostics have led us to believe there may be a bug in a DLL that is used by one of the processes running on this server that is resulting in a global atom leak. The assumed vector is a process that is calling RegisterClass without a corresponding UnregisterClass (and the class name is using a random number as part of the name, so it's a different class name each time the process starts).
This article provided some information: https://blogs.msdn.microsoft.com/ntdebugging/2012/01/31/identifying-global-atom-table-leaks/
But we are reluctant to attempt kernel mode debugging on a production server, so we have tried installing windbg and using the !gatom command to list atoms for a given session.
I use windbg to attach to a process in one of the sessions (these processes are running as Windows Services if that matters), then invoke the !gatom command. The returned atom list doesn't have any window classes in it.
Then I read this: https://blogs.msdn.microsoft.com/oldnewthing/20150429-00/?p=44984
and it sounds like there is a separate atom table for windows classes. And no way to query it. I was hoping that we'd be able to actually see how many windows class atoms have been registered, and see if that list gets bigger over time, indicating a leak.
The documentation on !gatom is sparse, and I'm hoping I can get some expert confirmation or recommendations on how to proceed.
Does anyone have any ideas on how we can get at the list of registered Windows classes on a production server?
More detail about what happens when the server starts to misbehave:
We run many instances (>50) of the same application as separately registered services running from isolated executables and DLLs - so each of those 50 instances has their own private executables and DLLs.
During their normal run, the processes unload and reload a DLL (about every hour). There is a windows class used that's part of a "session handle" used by the DLL (the session handle is part of the registered windows class name), and that session handle is unique each time the DLL is loaded. So every hour, there is an additional Window class registration, made by a DLL (our service stays running).
After some period of time, the system will get into a state where further attempts to load the DLL in question fail. This may happen for one of the services, then gradually over time, other services will start to have the same problem.
When this happens, restarting the service does not fix the problem. The only way that we've found to get things running properly again is to reboot the server.
We are monitoring memory commit load, and we are well within the virtual memory of the server. We are even within the physical memory size.
I just did a code review the vendor of the DLL, and it looks like they are not actually calling RegisterClass from the DLL itself (they only make one RegisterClass call from the DLL, and it's a static string - not a different class name for each session). The DLL launches an EXE, and that EXE is the one that registers the session specific class name. Their EXE does call UnregisterClass (and even if it didn't, the EXE is terminated when we unload their DLL, so it seems that this may not be what is going on).
I am now out of bullets on this one. The behavior seems like some sort of resource leak or pool exhaustion. The next time this happens, I will try connecting to the failing process with windbg and see what the application atom pool looks like - but I'm not hopeful that is going to shed any light.
Update: The excellent AtomTableMonitor tool has narrowed the problem to rogue RegisterWindowMessage. I'm going to ask a more specific question focused on this exact issue: Diagnosing RegisterWindowsMessage leak
You may try using this standalone global atom monitor
The application appears to have capabilities to monitor atoms in services
that run in a different session
btw if you have narrowed it to RegisterWindowMessage
then spy++ can log the Registered messages system wide along with thread and process
spy++ (i am using it from vs2015 community)
ctrl+m select all windows in system
in the messages tab clear all and select registered
and start logging
you can also save the log (it is plain text in-spite of strange extension )
powershell -c "gc spy++.sxl -Tail 3"
<000152> 001F01A4 P message:0xC1B2 [Registered:"nsAppShell:EventID"] wParam:00000000 lParam:06EDFCE0 time:4:2
7:49.584 point:(408, 221)
<000153> 001F01A4 P message:0xC1B2 [Registered:"nsAppShell:EventID"] wParam:00000000 lParam:06EDFCE0 time:4:2
7:49.600 point:(408, 221)
<000154> 001F01A4 P message:0xC1B2 [Registered:"nsAppShell:EventID"] wParam:00000000 lParam:06EDFCE0 time:4:2
7:49.600 point:(408, 221)
Al right! This might sound crazy.
I want to know if it is possible to see the code or something that a specific process is using. I have a scheduled task that is supposed to be ran by a process. The font end to this app says that the scheduled task executed successfully, but I don't see anything in the logs.
I want to know if there is a tool to see what a process is doing.
The best suite is Windows Sysinternals by Mark Russinovich (now bought by Microsoft).
Get it here.
Use Process Explorer.
Ever wondered which program has a
particular file or directory open? Now
you can find out. Process Explorer
shows you information about which
handles and DLLs processes have opened
or loaded.
The Process Explorer display consists
of two sub-windows. The top window
always shows a list of the currently
active processes, including the names
of their owning accounts, whereas the
information displayed in the bottom
window depends on the mode that
Process Explorer is in: if it is in
handle mode you'll see the handles
that the process selected in the top
window has opened; if Process Explorer
is in DLL mode you'll see the DLLs and
memory-mapped files that the process
has loaded. Process Explorer also has
a powerful search capability that will
quickly show you which processes have
particular handles opened or DLLs
loaded.
The unique capabilities of Process
Explorer make it useful for tracking
down DLL-version problems or handle
leaks, and provide insight into the
way Windows and applications work.
I'm not sure which technology you are using, but you could try to remotely debug the application the application, for example using Visual Studio. But to be able to do this you'll need the correct debug information files (.pdb).
If this is your own application you are trying to monitor, inserting some extra logging of some sort will be the easiest solution.
Another option you have is to make a memory dump at the moment you think the application is running. A memory dump is basically a snapshot of the entire memory. YOu can then use a tool like WinDbg to examine the contents of the dump.
Maybe you can give us a little bit more information?
I am new to programming.
I know only Start debug before. Maybe start debug suit for some small application develop better.
I found Visual studio IDE provide another method of attach to process for using.
When & Why must I use the attach debugging?
Such as multi-threading application debugging. Client/Service application debugging. etc. Thank you.
Sometimes you need to debug a process started by another program.
For example you need a reliable solution and in order to protect against access violations, memory leaks and other barely recoverable stuff, you have a master program and several worker programs. The master program starts the worker program and passes parameters to it. How do you debug a worker program which is not intended to be started by anything except the master program?
You use "attach to process for that".
Typically you do it this way: insert a statement that blocks the worker program for some time - for example, call Sleep() for 15 seconds. Then you kindly ask the master program to start the worker program. When the worker program is started it blocks and you now have 15 seconds to attach to it.
This way you can debug almost any issues - problems at early startup stages, wrong parameters, etc, which you wouldn't reliably reproduce with "run with debugging".
Attaching to a process is useful if you don't want to debug right from starting the process. For example, debugging usually slows down execution, so it can be quicker to start the app, get it to a state where a bug appears, and then attach a debugger.
It's also useful if you already have an external means of launching the process that you don't want or can't to import into the IDE.
Start debugging from VS launches an instance of the VS webserver and attaches the debugger to it.
Attach to process allows you to attach to any process and debug it, usually you'd do this to your instance of w3wp.exe running your code in IIS
Attach to process is mostly used when you can't run the application from Visual Studio.
For example, if it's a service or if it is a process that has run for a long time and now you want to start debugging it.
Sometimes you also want to debug a remote process, not on your machine - and you can do that using attach to process.
An application is hanging occasionally, and I would like to see the dump at the time to figure it out. I had written an application that the user can run to automatically create a dump that I can look at. However I can't seem to get the users to remember to run it when it hangs, no matter what I try. They always end up closing the program, which invokes Windows Error Reporting.
WER will create dumps in the temp directory, but unfortunately they are deleted as soon as the dialog for sending the info to Microsoft or not is closed.
Becoming an ISV and getting this info from Microsoft's error reporting servers is one solution.. but not one that is realistic at the moment.
I can't imagine that I am the only one faced with this issue. The software is used concurrently by dozens upon dozens of staff, so reaching them all and getting them to run an application or not click close on that dialog until running some other application or etc has not been working out.
The app is running on Windows Server 2003. Too bad, since I know Server 2008 has some LocalDumps options that will let me retain them.
Any ideas for somehow keeping these dumps around so I can analyze them? The obstacle is the user, in the sense that I've accepted to their stubbornness and do not expect them to run any other application or do anything special.
Thanks for any advice!
You could opt for an automatic solution. I believe there're multiple options at your disposal for detecting if you're hung.
One would be the use of SendMessageTimeout (also pay attention to SMTO_ABORTIFHUNG as one of the fuFlags values) from a separate thread in your app. Once you have determined the main thread is not responding you can save a dump file wherever you want.
There's also a IsHungAppWindow() (user32.dll) available since w2k.