Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 6 years ago.
Improve this question
I have an application which has a loop, part of a "Scheduler", which runs at all time and is the heart of the application. Pretty much like a game loop, just that my application is a WPF application and it's not a game. Naturally the application does logging at many points, but the Scheduler does some sensitive monitoring, and sometimes it's impossible just from the logs to tell what may have gotten wrong (and by wrong I don't mean exceptions) or the current status.
Because Scheduler's inner loop runs at short intervals, you can't do file I/O-based logging (or using the Event Viewer) in there. First, you need to watch it in real-time, and secondly the log file would grow in size very fast. So I was thinking of ways to show this data to the user in the realtime, some things I considered:
Display the data in realtime in the UI
Use AllocConsole/WriteConsole to display this information in a console
Use a different console application which would display this information, communicate between the Scheduler and the console app using pipes or other IPC techniques
Use Windows' Performance Monitor and somehow feed it with this information
ETW
Displaying in the UI would have its issues. First it doesn't integrate with the UI I had in mind for my application, and I don't want to complicate the UI just for this. This diagnostics would only happen rarely. Secondly, there is going to be some non-trivial data protection, as the Scheduler has it's own thread.
A separate console window would work probably, but I'm still worried if it's not too much threshold. Allocating my own console, as this is a windows app, would probably be better than a different console application (3), as I don't need to worry about IPC communication, and non-blocking communication. However a user could close the console I allocated, and it would be problematic in that case. With a separate process you don't have to worry about it.
Assuming there is an API for Performance Monitor, it wouldn't be integrated too well with my app or apparent to the users. Using ETW also doesn't solve anything, just a random idea, I still need to display this information somehow.
What others think, would there be other ways I missed?
Respectfully - both Adrian K and Dima's answers are not correct. The right answer is to use Event Tracing For Windows (ETW). This is what we use for all logging in Windows. Its extremely robust and very well performing. For example W7 logs an ETW event on many OS events - all the time - including processor context switch. Ever use the performance monitor in W7? It is consuming ETW events from the kernel.
I recommend you do all your logging with ETW. Why? Several reasons:
Its ubiquitous
You can enable disable logging in your running process. No process restarts required. (yes, other loggers do this, but some do not).
Its designed for including in shipping code.
Logging an event is guaranteed to be non-blocking: it will not cause a 'wait'.
We provide lots of tools for ETW trace processing. most notably the XPERF tools (link, link, link)
A big benefit of instrumenting your performance paths with ETW events is that your events can be seen integral with the kernel events using the XPERF tools.
Its also pretty easy to write a 'watch' application that watches ETW events from your components. I have one for one of our components that simply displays the events to the console.
I highly recommended to not try and write your own high performance logging system. This is challenging to do well, but in terms of performance and reliability. The Windows ETW system is super-robust and very well performing.
Back to basics - seperate concerns.
My usual solution would be use the Microsoft Enterprise Libraries to handle the actual logging; I'd use a database as the repository, you can then query it at will, from any application (your existing one or something completely stand-alone).
The thing I like about the MS Ent Libs is you can configure them to log to a wide variety of repository types. You could extend them with if need be; I'm not sure if you want to work asynchronously for performance / execution constraints.
I prefer logging to a DB as it gives a good level of control: it's easy to query and reasonably easy to mamage the data. Having sia dthat the Ent Libs do allow rolling file based logging - that would help you manage files sizes - but using a Db would be quicker than reading a file.
I guess it comes down to what you mean by "realtime" as to whether logging to a db would be fast enough. - real-time to a computer is very different from real-time to a person.
You could log to memory, and then asynchronously iterate through those logs entries and commit them to log term storage (DB). For reporting you could use the in-memory copy to show the 'current' state, and refer to the DB for longer periods of time / stuff in the more distant past.
Related
I am running Windows 7 on which I want to do kernel debugging and I do not want to mess with boot loader. So I've downloaded LiveKd as suggested here and make it run and seems it is working. If I understand correct it is some kind of read only debugging. Here is mentioned that it is very limited and even breakpoint cannot be used. I would like to ask if is possible in this mode to periodically dump all the instructions that are being executed or basically all events which are happening on current OS? I would like to have some system wide strace (Linux users know) and to do some statistical analysis on this. I suppose it depends on more factors like installed debug symbols to begin able resolve addresses etc.
I'm not sure if debugger is the best tool you can use for tracing live system calls. As you've mentioned LiveKd session is quite limited and you are not allowed to place breakpoints in it (otherwise you would hang your own system). However, you still can create memory dumps using the .dump command (check windbg help: .hh .dump). Keep in mind though that getting a full dump (/f) of a running system might take a lot of time.
Moving back to the subject of your question, by using the "dump approach" you will miss many system calls as you will have only snapshots of a system at given points in time. So if you are looking for something similar to Linux strace I would recommend checking those tools:
Process Monitor (procmon) - it's a tool which will show you all I/O requests in the system, as well as operations performed on the registry or process activity events
Windows Performance Toolkit - it contains tools for collecting (WPR) and analysing (WPA) system and application tracing events. It might be a lot of events and it's really important to filter them accordingly to your needs. ETW (Event Tracing for Windows) is a huge subject and you probably will need to read some tutorials or books before you will be able to use it effectively (but it's really worth it!).
API Monitor - it's one of many (I consider it as one of the best) tracing applications - this tool will allow you to trace method calls in any of the running processes. It has a nice interface and even allows you to place breakpoints on methods you'd like to intercept.
There are many other tools which might be used for tracing on Windows, but I would start with the ones I listed above. You may also check a great book on this subject: Inside Windows Debugging. Good luck! :)
I'm interesting in gathering usage metrics for an application that I did not write and have no control over. This is a applicaiton running on Windows.
My plan for this is to register a global windows hook for mouse and keyboards events, and record those events for windows that have pre-determined titles or other identifable attributes.
Using this data, I hope to be able to determine how a user uses the application in question. What buttons they click and when, as well as common workflows. Etc.
Any thoughts on this idea? Are there 3rd party products or libraries that would facilitate this solutions that do not require modification of the existing application?
I assume (hope) this is for something like usability experimentation and not nefarious purposes.
The approach you outlined seems reasonable. The drawbacks of global hooking are:
It's a burden on every process, since your hook code will be injected into every process.
It can run into security barriers. For example, if you're hooking from a medium integrity level app, I don't think you'll be able to hook a high integrity level app. Also, you're essentially creating a keylogger, so don't be surprised if some anti-malware app flags you as possibly spyware.
You may need a 64-bit version and a 32-bit version.
One way to mitigate the impact you have on the machine is to use a more targeted hook: find the particular process you care about, enumerate its threads, and hook only those.
Spy++, a developer tool for tracking Windows messages, does much of what you want, but I'm not sure if you could leverage it for your purpose. I'm not aware of any other tools or frameworks for doing this kind of instrumentation.
I am working on building a system that can monitor how users react to security alerts on their systems (software updates, warnings etc.). It also needs to monitor the web traffic and the processes running on the system and I am looking to the community to help me design this system. We intend to provide users with test laptops and monitor their behavior over a period of time to see how they react to security alerts thrown by various applications and the OS(windows in this case).
Following are my questions
Can I use windows hooks to solve the first problem i.e finding how users reacted to the alerts thrown by various applications. Specifically, can global hooks be used to solve this?
(How this information should be collected (XML?) and relayed back to a server(how frequently?) is another problem)
Can I do this in C# or it has to be done only in c++ or VB?
Do you know any alternate approach to solve the problem? Is there any software that has these capabilities.
I have many more questions but getting these answered would be a good first step. Really hoping for some good insights from the knowledgeable people on this community
Thank you in advance
Edit:
Example scenario is when adobe prompts you to update the flash player or the antivirus prompts you to update definitions or any application displays a notification(security related having keywords like update, warning, install etc.) needing the user to take some action. Windows system updates is another example. I want to know how the user reacted to these alerts/notifications/updates (which are typically a pop-up window). So i was wondering if i placed a global hook that can monitor the content of the windows displayed on screen and notify me(server) when certain words like update, alert, warning etc. appear in the content/title of the windows and what the user did with the message(dismissed it, Oked it etc). Unfortunately, i do not have any more specifications than this. I can use anything I want to achieve this and I am not clear on what my choices are.
Edit 2:
After having reviewed my requirements and having read about hooks, I feel like I could achieve this by a combination of hooks and the following textGrab SDK, http://www.renovation-software.com/en/text-grab-sdk/textgrab-sdk.html. I want some guidance to know if I am on the right track. I am thinking if I can install hooks then it gives me handles to all possible windows on the screen and I can use the textGRAB SDK to look for certain keywords in those windows. Although this may capture some interesting text, I am still not sure how I will know what action the user had taken on the window. Anybody having any experience with either hooks or textGRAB, please let me know if this looks like a reasonable thing to do. If the community has some other Ideas on how I could possibly monitor security related messages thrown by any application in the system, please suggest. I am looking forward to some useful advice for completing a challenging project.
First of all, you need to define, how you will "see" security alerts in code. "Security alert" is quite a vague term. Will it be some window with some caption and some message to the user or ... ?
Next, about web and processes: Windows hooks won't help you with your task. They are more low-level and not as advanced as you'd need. You can't hook network traffic (you need either network filter driver for pre-Vista or Microsoft Filtering Platform for Vista and later). See this question for some information about checking the process list with C# (there seems to be no easy way to catch process startup either).
It honestly sounds like you need a more solid direction. I commend you for trying to provide details, but It appears that you still need more information about your problem(s)..
I will attempt to answer some of your questions, but like I said - it sounds like you need to know more about your problems before we can provide you with optimal answer(s).
-Alerts is too vague a term, you will need to define this better. Are these 'alerts' applications that YOU have control over or are they third party applications? Not every application will show an 'Alert' in the same fashion, and even if they did - I think using a System Level Hook would probably be too problematic to implement your solution with. I'm not saying it's necessarily impossible, but you're talking about possibly implementing a different set of logic(to determine the data for a given application's Alert(s)) for each application that you want to monitor.
-It's impossible for any of us to determine the optimal storage mechanism for your particular needs, that is something that you will either need to provide more details about or decide on your own.
-How often you collect data is also something that you will have to either provide more details for or decide for on your own.
-C/C++ Would probably provide you with the most portable solution, although there is nothing preventing you from using c# to call Win32 API. (Not everyone has the .NET framework installed - believe it or not)
-The problem that you mentioned appears to be a somewhat specialized problem... I don't know of any existing software that will do everything that you want to do.
Another possible issue that you haven't touched on:
You haven't specified your target audience for this 'service', but I want you to know that if I found an application monitoring as many events as what you're talking about doing, I would promptly remove it and write a nasty letter to the company that wrote it.
In summary, Read this Article on hooks to get a better understanding of how they work.
An answer to this question has led me to look into using "Event Tracing for Windows" for our tracing needs. I have come across NTrace, which seems to be a good way to produce ETW events from C# code (using the XP-compatible "classic provider" model).
However, I am unable to find an easy way to consume these events - to see them in real-time and/or log them to a file. The only way I have found is that described in the NTrace documentation: using a tool which is only available as part of the Windows DDK.
In the case of a complex problem in the field, we may need to ask the user to produce a file containing a trace. We can't ask users to download the DDK or carry out a number of complex operations in order to do this.
Is there a straightforward, user-friendly way to log ETW events to a file?
Also, is it possible for someone to consume ETW events on Windows Vista/7 if they are not running as administrator?
TraceView is the easiest out-of-the-box solution, but it is possible to write your own ETW viewer that is specific to your provider. This would give you full control over the presentation and make it much easier on the end user as TraceView is really more of a debugging tool than something you can ask end users to run.
As far as real-time tracing goes, according to the documentation:
Only users with administrative privileges, users in the Performance Log Users group, and services running as LocalSystem, LocalService, NetworkService can consume events in real time. To grant a restricted user the ability to consume events in real time, add them to the Performance Log Users group.
Windows XP and Windows 2000: Anyone can consume real time events.
If you're interested in writing your own ETW viewer (real-time or log file), here is the relevant documentation.
Windows Event Log reads the ETW. In fact I'd say this is the correct way for a consumer (non program) to view and export the ETW traces.
See here for an example. http://blogs.microsoft.co.il/blogs/applisec/archive/2009/10/12/reading-etw-tracing-using-event-viewer.aspx
This question on msdn Discuses what to do when the logs don't appear. Does anything here help?
ETW tracing was designed to run only by administrators because trace may contain personal identifiable information. And it would pose security threat if a non-admin can capture the trace.
Here is a warning Example from xperf
The trace you have just captured "C:\Windows\system32\kernel.etl" may contain
personally identifiable information, including but not necessarily limited to paths to files accessed, paths to registry accessed and process names. Exact information depends on the events that were logged. Please be aware of this when sharing out this trace with other people.
Hope this answers your question
Here is how you can get custom ETW traces from your own custom provider and how ETW can be used within managed code
Hope this helps.
IMO Perfview is one of the best tools available to control and view ETW traces. It can also provide managed call-stacks. The best part of it is you could xcopy on to any server and collect traces.
Perfview uses TraceEvent library and here the samples how you could use it using an API and C#
When working in a big project that requires debugging (like every project) you realize how much people love "printf" before the IDE's built-in debugger. By this I mean
Sometimes you need to render variable values to screen (specially for interactive debugging).
Sometimes to log them in a file
Sometimes you have to change the visibility (make them public) just to another class to access it (a logger or a renderer for example).
Some other times you need to save the previous value in a member just to contrast it with the new during debugging
...
When a project gets huge with a lot of people working on it, all this debugging-specific code can get messy and hard to differentiate from normal code. This can be crazy for those who have to update/change someone else's code or to prepare it for a release.
How do you solve this?
It is always good to have naming standards and I guess that debug-coding standards should be quite useful (like marking every debug-variable with a _DBG sufix). But I also guess naming is just not enough. Maybe centralizing it into a friendly tracker class, or creating a robust base of macros in order to erase it all for the release. I don't know.
What design techniques, patterns and standards would you embrace if you are asked to write a debug-coding document for all others in the project to follow?
I am not talking about tools, libraries or IDE-specific commands, but for OO design decisions.
Thanks.
Don't commit debugging code, just debuggin tools.
Loggin OTOH has a natural place in execption handling routines and such. Also a few well placed logging statments in a few commonly used APIs can be good for debugging.
Like one log statment to log all SQL executed from the system.
My vote would be with what you described as a friendly tracker class. This class would keep all of that centralized, and potentially even allow you to change debug/logging strategies dynamically.
I would avoid things like Macros simply because that's a compiler trick, and not true OO. By abstracting the concept of debug/logging, you have the opportunity to do lots of things with it including making it a no-op if needed.
Logging or debugging? I believe that well-designed and properly unit-tested application should not need to be permanently instrumented for debugging. Logging on the other hand can be very useful, both in finding bugs and auditing program actions. Rather than cover a lot of information that you can get elsewhere, I would point you at logging.apache.org for either concrete implementations that you can use or a template for a reasonable design of a logging infrastructure.
I think it's particularly important to avoid using System.outs / printfs directly and instead use (even a custom) logging class. That at least gives you a centralized kill-switch for all the loggings (minus the call costs in Java).
It is also useful to have that log class have info/warn/error/caveat, etc.
I would be careful about error levels, user ids, metadata, etc. since people don't always add them.
Also, one of the most common problems that I've seen is that people put temporary printfs in the code as they debug something, and then forget where they put them. I use a tool that tracks everything that I do so I can quickly identify all my recent edits since an abstract checkpoint and delete them. In most cases, however, you may want to pose special rules on debug code that can be checked into your source control.
In VB6 you've got
Debug.Print
which sends output to a window in the IDE. It's bearable for small projects. VB6 also has
#If <some var set in the project properties>
'debugging code
#End If
I have a logging class which I declare at the top with
Dim Trc as Std.Traces
and use in various places (often inside #If/#End If blocks)
Trc.Tracing = True
Trc.Tracefile = "c:\temp\app.log"
Trc.Trace 'with no argument stores date stamp
Trc.Trace "Var=" & var
Yes it does get messy, and yes I wish there was a better way.
We routinely are beginning to use a static class that we write trace messages to. It is very basic and still requires a call from the executing method, but it serves our purpose.
In the .NET world, there is already a fair amount of built in trace information available, so we do not need to worry about which methods are called or what the execution time is. These are more for specific events which occur in the execution of the code.
If your language does not support, through its tracing constructs, categorization of messages, it should be something that you add to your tracing code. Something to the effect that will identify different levels of importance and/or functional areas is a great start.
Just avoid instrumenting your code by modifying it. Learn to use a debugger. Make logging and error handling easy. Have a look at Aspect Oriented Programming
Debugging/Logging code can indeed be intrusive. In our C++ projects, we wrap common debug/log code in macros - very much like asserts. I often find that logging is most usefull in the lower level components so it doesn't have to go everywhere.
There is a lot in the other answers to both agree and disagree with :) Having debug/logging code can be a very valuable tool for troubleshooting problems. In Windows, there are many techniques - the two major ones are:
Extensive use of checked (DBG) build asserts and lots of testing on DBG builds.
the use of ETW in what we call 'fre' or 'retail' builds.
Checked builds (what most ohter call DEBUG builds) are very helpfull for us as well. We run all our tests on both 'fre' and 'chk' builds (on x86 and AMD64 as well, all serever stuff runs on Itanium too...). Some people even self host (dogfood) on checked builds. This does two things
Finds lots of bugs that woldn't be found otherwise.
Quickly elimintes noisy or unnessary asserts.
In Windows, we use Event Tracing for Windows (ETW) extensively. ETW is an efficient static logging mechanism. The NT kernel and many components are very well instrumented. ETW has a lot of advantages:
Any ETW event provider can be dynamically enabled/disabled at run time - no rebooting or process restarts required. Most ETW providers provide granular control over individual events, or groups of events.
Events from any provider (most importantly the kernel) can be merged into a single trace so all events can be correlated.
A merged trace can be copied off the box and fully processed - with symbols.
The NT kernel sample pofile interrupt can generate an ETW event - this yeilds a very light weight sample profiler that can be used any time
On Vista and Windows Server 2008, logging an event is lock free and fully multi-core aware - a thread on each processor can independently log events with no synchronization needed between them.
This is hugly valuable for us, and can be for your Windows code as well - ETW is usuable by any component - including user mode, drivers and other kernel components.
One thing we often do is write a streaming ETW consumer. Instead of putting printfs in the code, I just put ETW events at intersting places. When my componetn is running, I can then just run my ETW watcher at any time - the watcher receivs the events and displays them, conts them, or does other interesting things with them.
I very much respectfully disagree with tvanfosson. Even the best code can benefit from well implemented logging. Well implimented static run-time logging can make finding many problems straight forward - without it, you have zero visiblilty into what is happening in your component. You can look at inputs, outputs and guess -that's about it.
They key here is the term 'well implimented'. Instrumentation must be in the right place. Like any thing else, this requries some thought and planning. If it is not in helpfull/intersting places, then it will not help you find problems in a a development, testing, or deployed scenario. You can also have too much instrumeation causing perf problems when it is on - or even off!
Of course, different software products or componetns will have different needs. Some things may need very little instrumenation. But a widely depolyed or critical component can greatly benefit from weill egineered instrumeantion.
Here is a scenario for you (note, this very well may not apply to you...:) ). Lets say you have a line-of-business app deployed on every desktop in your company - hundreds or thousands of users. What do you do when someone has a pbolem? Do yo stop by their office and hookup a debugger? If so, how do you know what version they have? Where do you get the right symbols? How do you get the debuger on their system? What if it only happens once every few hours or days? Are you going to let the system run with the debugger connected all that time?
As you can imagine - hooking up debugger in this scenario is disruptive.
If your component is instrumented with ETW, then you could ask your user to simply turn on tracing; continue to do his/her work; then hit the "WTF" button when the problem happens. Even better: your app may have be able to self log - detecting problems at run time and turning on logging auto-magicaly. It could even send you ETW files when problems occured.
These are just simple exmples - logging can be handled many different ways. My key recomendation here is to think about how loging might be able to help you find, debug, and fix problems in your componetns at dev time, test time, and after they are deployed.
I was burnt by the same issue in about every project I've been involved with, so now I have this habit that involves extensive use of logging libraries (whatever the language/platform provides) from the start. Any Log4X port is fine for me.
Building yourself some proper debug tools can be extremely valuable. For example in a 3D environment, you might have an option to display the octree, or to render planned AI paths, or to draw waypoints that are normally invisible. You'd probably also want some on-screen display to aid with profiling too: the current framerate, count of polygons on screen, texture memory usage, and so on.
Although this takes some time and effort to do, in the long run it can save you a lot of time and frustration.