I receive letting_statedump events when I enable these events and start the tracing. I noticed that I receive this only once for the life of the trace session. How would I initiate the generation of statedump events without stopping the trace and restarting?
As far as I know, there is no way at the moment to trigger a statedump manually.
However I would ask, why would you need to do so? The goal of the statedump is for trace viewers or analysers to be able to have an idea of the initial state of the system. Afterwards, they can use the actual trace events to update their state trackers.
If you do not wish to write your own state tracker, another option that may help are the event contexts. If you enable a context like pid or procname, that information will be dumped as part of every trace event. This can be used to get up-to-date state information more easily. You can use
lttng add-context --help
to list the available contexts.
Related
I have a .NET application which spawns multiple child 'worker processes'. I am using the Windows Job Object API and the JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE setting to ensure the child processes always get killed if the parent process is terminated.
However, I have observed a number of orphaned processes still running on the machine after the parent has been closed. Using Process Explorer, I can see they are correctly still assigned to the Job, and that the Job has the correct 'Kill on Job Close' setting configured.
The documentation for JOB_OBJECT_LIMIT_KILL_ON_JOB_CLOSE states:
"Causes all processes associated with the job to terminate when the last handle to the job is closed."
This would seem to imply that a handle to the Job was still open somewhere... I did a search for handles to my Job object, and found instances of WmiPrvSE.exe in the results. If I kill the relevant WmiPrvSE.exe process, the outstanding handle to Job is apparently closed, and all the orphaned application processes get terminated as expected.
How come WmiPrvSE.exe has a handle to my Job?
You may find this blog in sorting out what WmiPrvSE is doing.
WmiPrvSE is the WMI Provider host. That means it hosts WMI providers, which are DLLs. So it's almost surely the case that WmiPrvSE doesn't have a handle to your job, but one of the providers it hosts does. In order to figure out which provider is the culprit, one way is to follow the process here and then see which of the separate processes holds the handle.
Once you have determined which provider is holding the handle you can either try to deduce, based on what system components the provider manages, what kind of query would have a handle to your Job. Or you can just disable the provider, if you don't care about losing access to the management of the components the provider provides.
If you can determine what kind of query would be holding a handle, you may be able to deduce what program is issuing the query. Or maybe the eventlog can tell you that (first link above).
To get more help please provide additional details in the OP, such as which providers are running in WmiPrvSE, any relevant eventlog events, and any other diagnostics info you obtain.
EDIT 1/27/16
An approach to find out what happened that caused WMIPrvSE to obtain your job's handle is to use Windbg's !htrace extension. You need to run !htrace -enable after you load you .EXE but before you execute it in Windbg. Then you can break in later and execute !htrace <handle> to see stack traces when the handle was manipulated. You may want to start with this article on handle implementation.
I am using Lttng in an application. I have enabled heavy traces and I have found that there is a loss in traces. Is there any way of knowing if there is any trace loss or any information regarding about it . Are there any API calls to know about them.
Thanks & Regards.,
K.V.Ranganadh.
Both viewers for LTTng traces should both be able to report if a trace has lost events in them.
Babeltrace, the command-line trace reading tool, prints lost events on stderr. So a quick way to locate these is to reroute stdout elsewhere, so you only see the lost events in the console, using a command like:
babeltrace /path/to/trace > /dev/null
Alternatively, the graphical viewer Trace Compass displays lost events in its Statistics View.
In general, lost events happen when the machine is too loaded and the tracer cannot keep up with the events coming in. To reduce the chance of losing events, you can look at increasing the subbuffer sizes and number (see the 'lttng' man page), or enabling less events in your tracing session (instead of doing 'lttng enable-event ... -a', only enable the events you need).
We have a system where there are typically two processes running on the same system. One process handles the GUI and the other runs like a service (although for historical reasons, it's not a service, just an exe with no visible window).
The two processes undertake IPC mainly via registered messages asynchronously - i.e. we use RegisterWindowMessage() in both processes to define a large'ish set of messages that effectively form the API to the server process.
I have written a "hands-free" monitoring application that uses SetWindowsHookEx() to monitor and display the message queues of both processes and provide some level of decoding of the way the API is being utilised and how notifications are being propagated to the GUI process (each individual window can subscribe to notifications from the server directly).
So, there are a large number of messages in both directions so I have filtering and summary counts etc. so I can focus on particular activity. All this can be done without affecting the live code, which is good.
This all works well, but it now would be very useful to be able to "tag" a message originating in the GUI so I can trace the same message when it's processed by the server. This would be enormously useful for debugging and diagnosing system issues, but I can't find a clean way (actually I can't find any way!) of doing this without adding such support to our registered message API, which would be a lot of work and involves more risk than I'm comfortable with at the moment. It gets further complicated by the fact that the server pre-processes some messages and then does a PostMessage() back to itself to perform the action, so the originating message can get "lost".
Has anyone here tackled this type of problem? If so, can you give me some pointers? If not, then are there any documented or undocumented ways of adding a small block of data to a Windows message and retrieving it later? I've looked at SetMessageExtraInfo() but that seems to be per-queue rather than per-message.
FindWindow or FindWindowEx will give you the details of the GUI Window. Compare the details with message intercepted
I have an app which uses a keyboard hook procedure in a library. The wParam in the hook for one message is 255 which we think is "(reserved / OEMClear)". I'd like to work out the source of this message as it causes my application to crash in the library, and given it shouldn't be happening it would be good to identify it. The message comes in repeatedly on only one PC we have - other computers don't see the message at all.
So, is there a way to trace the source of a message sent to a window please, or all those on the system?
There is no built-in way to find out who sent the window message, not even win32k keeps track of this; you might be able to find it out with a kernel debugger and a conditional breakpoint.
However, I would argue that you don't really need this information; you need to make your app properly handle any message sent to it.
I came up with a technique for determining who is sending a win32 window message across threads/processes during one-off debugging/troubleshooting sessions. It requires making a couple of assumptions so it's not 100% reliable, but so far I haven't found a case where it didn't work.
The basic idea is to exploit the fact that, when the message arrives, the recipient window thread is typically blocked waiting in its message loop (specifically, GetMessage()). When the message is delivered, the sending thread readies the receiving thread, pulling it out of its wait state.
It turns out that Windows provides ways to precisely trace which threads are readying which other threads, using Event Tracing for Windows. Using this feature, it is often possible to determine which thread sent the message - it's the thread that readied the receiving thread. It's even possible to see what the call stack of the sending thread was at the time it sent the message, and even the kernel side (win32k) part of the stack!
The basic procedure goes like this:
Use the Windows Performance Recorder to start a trace. Make sure to include the "CPU usage" profile.
Trigger the sending of the message you are interested in.
Stop the trace.
Open the trace in the Windows performance Analyzer.
In the "CPU Usage (Precise)" graph, "Stacks" graph preset, zoom in on the time the message was received.
One way is to locate the receiving thread and determine when it woke up.
If correlation is difficult, it might be worth instrumenting the receiving thread using e.g. TraceLogging to produce a clear reference time point.
You should be able to find a context switch event where the receiving thread is readied in GetMessage.
The "Readying Process", "Readying Thread Id" and "Readying Thread Stack" columns will then show the details of the readying thread, which is likely to be the sender of the message.
For example, in the below screenshot, TID 7640 receives a shell hook message originating from WindowsTerminal.exe, TID 1104:
(I originally suggested using Spy++ or winspector, but they do not hook into the sending of messages. That doesn't even make sense! A window receives messages but they don't send them, a thread does that. I'll leave my suggestion about using a debugger.)
Sometimes debugging can help. Try downloading the windows PDB files and setting a breakpoint that hits only when one of these messages occur. Looking at the call stack at that point can often shed some light on why things are happening. Posted messages and messages send from other processes will foil this approach.
Im not sure if this does what you want it to but have a look at Process Monitor by sysinternals.
http:// technet.microsoft.com/en-us/sysinternals/bb896645.aspx
It shows everything that happens to a process so i assume it catches messages as well. The site was down at time of writing so i couldnt check.
we're shipping a shell extension dll (registered with regsvr32).
is there an easy way to get debug output from this dll from another application (so we can send these traces home when something is broken)?
any ideas? what's the easiest way to get logdata from the dll to another process?
If it's a shell extension DLL, then doesn't it run as the logged-in user, and can't it therefore write to a log file in some suitable directory on disk? If so why then would you want it to write to another process?
You can use Event Tracing for Windows (ETW) to trace your extension DLL execution. ETW has almost no overhead when no listener is active, so in normal conditions your DLL will incur no perf penalty; at the same time it allows for detailed output at various levels of details.
The way ETW works is when the APIs are called, they check if there is a listener subscribed to the traces from particular publisher and if no, nothing is generated. If there is a listener, only the traces to which the listener is subscribed are written to a memory-mapped file. Thus, only as much traces data is generated as requested.
ETW listeners can be activated at any time and the publisher does not have to be restarted. Also, ETW is not flavor bound and can be used in both debug and retail. Thus, if a customer of yours has a problem, you have to only send them the listener with instructions on how to run it and collect the info; you don't have to sent them an instrumented binary version. You can either write your own app that acts as a listener, or you can use the standard tracelog.exe and tracefmt.exe tools to get the traces written to a file.
To generate the necessary ETW code in your DLL, you can use the WPP preprocesor instead of directly using the ETW APIs.
Note: While all the links I post here are to the Windows Driver Kit documentation, ETW and WPP can be (and are heavily) used for regular user mode programs.