An answer to this question has led me to look into using "Event Tracing for Windows" for our tracing needs. I have come across NTrace, which seems to be a good way to produce ETW events from C# code (using the XP-compatible "classic provider" model).
However, I am unable to find an easy way to consume these events - to see them in real-time and/or log them to a file. The only way I have found is that described in the NTrace documentation: using a tool which is only available as part of the Windows DDK.
In the case of a complex problem in the field, we may need to ask the user to produce a file containing a trace. We can't ask users to download the DDK or carry out a number of complex operations in order to do this.
Is there a straightforward, user-friendly way to log ETW events to a file?
Also, is it possible for someone to consume ETW events on Windows Vista/7 if they are not running as administrator?
TraceView is the easiest out-of-the-box solution, but it is possible to write your own ETW viewer that is specific to your provider. This would give you full control over the presentation and make it much easier on the end user as TraceView is really more of a debugging tool than something you can ask end users to run.
As far as real-time tracing goes, according to the documentation:
Only users with administrative privileges, users in the Performance Log Users group, and services running as LocalSystem, LocalService, NetworkService can consume events in real time. To grant a restricted user the ability to consume events in real time, add them to the Performance Log Users group.
Windows XP and Windows 2000: Anyone can consume real time events.
If you're interested in writing your own ETW viewer (real-time or log file), here is the relevant documentation.
Windows Event Log reads the ETW. In fact I'd say this is the correct way for a consumer (non program) to view and export the ETW traces.
See here for an example. http://blogs.microsoft.co.il/blogs/applisec/archive/2009/10/12/reading-etw-tracing-using-event-viewer.aspx
This question on msdn Discuses what to do when the logs don't appear. Does anything here help?
ETW tracing was designed to run only by administrators because trace may contain personal identifiable information. And it would pose security threat if a non-admin can capture the trace.
Here is a warning Example from xperf
The trace you have just captured "C:\Windows\system32\kernel.etl" may contain
personally identifiable information, including but not necessarily limited to paths to files accessed, paths to registry accessed and process names. Exact information depends on the events that were logged. Please be aware of this when sharing out this trace with other people.
Hope this answers your question
Here is how you can get custom ETW traces from your own custom provider and how ETW can be used within managed code
Hope this helps.
IMO Perfview is one of the best tools available to control and view ETW traces. It can also provide managed call-stacks. The best part of it is you could xcopy on to any server and collect traces.
Perfview uses TraceEvent library and here the samples how you could use it using an API and C#
Related
I am running Windows 7 on which I want to do kernel debugging and I do not want to mess with boot loader. So I've downloaded LiveKd as suggested here and make it run and seems it is working. If I understand correct it is some kind of read only debugging. Here is mentioned that it is very limited and even breakpoint cannot be used. I would like to ask if is possible in this mode to periodically dump all the instructions that are being executed or basically all events which are happening on current OS? I would like to have some system wide strace (Linux users know) and to do some statistical analysis on this. I suppose it depends on more factors like installed debug symbols to begin able resolve addresses etc.
I'm not sure if debugger is the best tool you can use for tracing live system calls. As you've mentioned LiveKd session is quite limited and you are not allowed to place breakpoints in it (otherwise you would hang your own system). However, you still can create memory dumps using the .dump command (check windbg help: .hh .dump). Keep in mind though that getting a full dump (/f) of a running system might take a lot of time.
Moving back to the subject of your question, by using the "dump approach" you will miss many system calls as you will have only snapshots of a system at given points in time. So if you are looking for something similar to Linux strace I would recommend checking those tools:
Process Monitor (procmon) - it's a tool which will show you all I/O requests in the system, as well as operations performed on the registry or process activity events
Windows Performance Toolkit - it contains tools for collecting (WPR) and analysing (WPA) system and application tracing events. It might be a lot of events and it's really important to filter them accordingly to your needs. ETW (Event Tracing for Windows) is a huge subject and you probably will need to read some tutorials or books before you will be able to use it effectively (but it's really worth it!).
API Monitor - it's one of many (I consider it as one of the best) tracing applications - this tool will allow you to trace method calls in any of the running processes. It has a nice interface and even allows you to place breakpoints on methods you'd like to intercept.
There are many other tools which might be used for tracing on Windows, but I would start with the ones I listed above. You may also check a great book on this subject: Inside Windows Debugging. Good luck! :)
I am working on building a system that can monitor how users react to security alerts on their systems (software updates, warnings etc.). It also needs to monitor the web traffic and the processes running on the system and I am looking to the community to help me design this system. We intend to provide users with test laptops and monitor their behavior over a period of time to see how they react to security alerts thrown by various applications and the OS(windows in this case).
Following are my questions
Can I use windows hooks to solve the first problem i.e finding how users reacted to the alerts thrown by various applications. Specifically, can global hooks be used to solve this?
(How this information should be collected (XML?) and relayed back to a server(how frequently?) is another problem)
Can I do this in C# or it has to be done only in c++ or VB?
Do you know any alternate approach to solve the problem? Is there any software that has these capabilities.
I have many more questions but getting these answered would be a good first step. Really hoping for some good insights from the knowledgeable people on this community
Thank you in advance
Edit:
Example scenario is when adobe prompts you to update the flash player or the antivirus prompts you to update definitions or any application displays a notification(security related having keywords like update, warning, install etc.) needing the user to take some action. Windows system updates is another example. I want to know how the user reacted to these alerts/notifications/updates (which are typically a pop-up window). So i was wondering if i placed a global hook that can monitor the content of the windows displayed on screen and notify me(server) when certain words like update, alert, warning etc. appear in the content/title of the windows and what the user did with the message(dismissed it, Oked it etc). Unfortunately, i do not have any more specifications than this. I can use anything I want to achieve this and I am not clear on what my choices are.
Edit 2:
After having reviewed my requirements and having read about hooks, I feel like I could achieve this by a combination of hooks and the following textGrab SDK, http://www.renovation-software.com/en/text-grab-sdk/textgrab-sdk.html. I want some guidance to know if I am on the right track. I am thinking if I can install hooks then it gives me handles to all possible windows on the screen and I can use the textGRAB SDK to look for certain keywords in those windows. Although this may capture some interesting text, I am still not sure how I will know what action the user had taken on the window. Anybody having any experience with either hooks or textGRAB, please let me know if this looks like a reasonable thing to do. If the community has some other Ideas on how I could possibly monitor security related messages thrown by any application in the system, please suggest. I am looking forward to some useful advice for completing a challenging project.
First of all, you need to define, how you will "see" security alerts in code. "Security alert" is quite a vague term. Will it be some window with some caption and some message to the user or ... ?
Next, about web and processes: Windows hooks won't help you with your task. They are more low-level and not as advanced as you'd need. You can't hook network traffic (you need either network filter driver for pre-Vista or Microsoft Filtering Platform for Vista and later). See this question for some information about checking the process list with C# (there seems to be no easy way to catch process startup either).
It honestly sounds like you need a more solid direction. I commend you for trying to provide details, but It appears that you still need more information about your problem(s)..
I will attempt to answer some of your questions, but like I said - it sounds like you need to know more about your problems before we can provide you with optimal answer(s).
-Alerts is too vague a term, you will need to define this better. Are these 'alerts' applications that YOU have control over or are they third party applications? Not every application will show an 'Alert' in the same fashion, and even if they did - I think using a System Level Hook would probably be too problematic to implement your solution with. I'm not saying it's necessarily impossible, but you're talking about possibly implementing a different set of logic(to determine the data for a given application's Alert(s)) for each application that you want to monitor.
-It's impossible for any of us to determine the optimal storage mechanism for your particular needs, that is something that you will either need to provide more details about or decide on your own.
-How often you collect data is also something that you will have to either provide more details for or decide for on your own.
-C/C++ Would probably provide you with the most portable solution, although there is nothing preventing you from using c# to call Win32 API. (Not everyone has the .NET framework installed - believe it or not)
-The problem that you mentioned appears to be a somewhat specialized problem... I don't know of any existing software that will do everything that you want to do.
Another possible issue that you haven't touched on:
You haven't specified your target audience for this 'service', but I want you to know that if I found an application monitoring as many events as what you're talking about doing, I would promptly remove it and write a nasty letter to the company that wrote it.
In summary, Read this Article on hooks to get a better understanding of how they work.
I have a .net application which I don't have source too. I'm interested in determining what files this application is accessing. I thought that a debugger would be a good way to do this. I work only in the linux world and know nothing about windows debuggers. I feel like there should be a way to do this with windbg, I know how I would go about it with gdb but not with windbg. Any help or advice is much appreciated.
If you just want to monitor file system activity for a given process then you won't find anything better than Process Monitor. It does lots more too and it's free from Sysinternals (now Microsoft).
Process Monitor is an advanced
monitoring tool for Windows that shows
real-time file system, Registry and
process/thread activity. It combines
the features of two legacy
Sysinternals utilities, Filemon and
Regmon, and adds an extensive list of
enhancements including rich and
non-destructive filtering,
comprehensive event properties such
session IDs and user names, reliable
process information, full thread
stacks with integrated symbol support
for each operation, simultaneous
logging to a file, and much more. Its
uniquely powerful features will make
Process Monitor a core utility in your
system troubleshooting and malware
hunting toolkit.
Stu is right if you just want to know what files are being accessed.
But if you want to know more about debugging .net apps, you can place break points on .net code when you have SOS.dll loaded into windbg. It's more involved than I can describe here, but you can Google windbg and SOS.DLL to find more info. Tess Ferrandez has a great blog that you may find useful, too: http://blogs.msdn.com/b/tess/
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 6 years ago.
Improve this question
I have an application which has a loop, part of a "Scheduler", which runs at all time and is the heart of the application. Pretty much like a game loop, just that my application is a WPF application and it's not a game. Naturally the application does logging at many points, but the Scheduler does some sensitive monitoring, and sometimes it's impossible just from the logs to tell what may have gotten wrong (and by wrong I don't mean exceptions) or the current status.
Because Scheduler's inner loop runs at short intervals, you can't do file I/O-based logging (or using the Event Viewer) in there. First, you need to watch it in real-time, and secondly the log file would grow in size very fast. So I was thinking of ways to show this data to the user in the realtime, some things I considered:
Display the data in realtime in the UI
Use AllocConsole/WriteConsole to display this information in a console
Use a different console application which would display this information, communicate between the Scheduler and the console app using pipes or other IPC techniques
Use Windows' Performance Monitor and somehow feed it with this information
ETW
Displaying in the UI would have its issues. First it doesn't integrate with the UI I had in mind for my application, and I don't want to complicate the UI just for this. This diagnostics would only happen rarely. Secondly, there is going to be some non-trivial data protection, as the Scheduler has it's own thread.
A separate console window would work probably, but I'm still worried if it's not too much threshold. Allocating my own console, as this is a windows app, would probably be better than a different console application (3), as I don't need to worry about IPC communication, and non-blocking communication. However a user could close the console I allocated, and it would be problematic in that case. With a separate process you don't have to worry about it.
Assuming there is an API for Performance Monitor, it wouldn't be integrated too well with my app or apparent to the users. Using ETW also doesn't solve anything, just a random idea, I still need to display this information somehow.
What others think, would there be other ways I missed?
Respectfully - both Adrian K and Dima's answers are not correct. The right answer is to use Event Tracing For Windows (ETW). This is what we use for all logging in Windows. Its extremely robust and very well performing. For example W7 logs an ETW event on many OS events - all the time - including processor context switch. Ever use the performance monitor in W7? It is consuming ETW events from the kernel.
I recommend you do all your logging with ETW. Why? Several reasons:
Its ubiquitous
You can enable disable logging in your running process. No process restarts required. (yes, other loggers do this, but some do not).
Its designed for including in shipping code.
Logging an event is guaranteed to be non-blocking: it will not cause a 'wait'.
We provide lots of tools for ETW trace processing. most notably the XPERF tools (link, link, link)
A big benefit of instrumenting your performance paths with ETW events is that your events can be seen integral with the kernel events using the XPERF tools.
Its also pretty easy to write a 'watch' application that watches ETW events from your components. I have one for one of our components that simply displays the events to the console.
I highly recommended to not try and write your own high performance logging system. This is challenging to do well, but in terms of performance and reliability. The Windows ETW system is super-robust and very well performing.
Back to basics - seperate concerns.
My usual solution would be use the Microsoft Enterprise Libraries to handle the actual logging; I'd use a database as the repository, you can then query it at will, from any application (your existing one or something completely stand-alone).
The thing I like about the MS Ent Libs is you can configure them to log to a wide variety of repository types. You could extend them with if need be; I'm not sure if you want to work asynchronously for performance / execution constraints.
I prefer logging to a DB as it gives a good level of control: it's easy to query and reasonably easy to mamage the data. Having sia dthat the Ent Libs do allow rolling file based logging - that would help you manage files sizes - but using a Db would be quicker than reading a file.
I guess it comes down to what you mean by "realtime" as to whether logging to a db would be fast enough. - real-time to a computer is very different from real-time to a person.
You could log to memory, and then asynchronously iterate through those logs entries and commit them to log term storage (DB). For reporting you could use the in-memory copy to show the 'current' state, and refer to the DB for longer periods of time / stuff in the more distant past.
I am facing a severe problem with my program, which gets reproduced only in the customer place. Putting logs, are not helping as I doubt the failure is happening in a third party dll. For some reasons, I couldn't get help from the library provider. I am thinking of producing a dump at the point of failure, so that to analyze it offline. Is this a recommended practice? Or any alternatives?
Yes, this is something that every program should have and utilize as often as possible.
I suggest that you don't use third party libraries. Create your own dumps instead. It's very simple and straight forward. You basically need to do the following:
Your program needs to access dbghelp.dll. It's a windows dll that allows you to create human readable call stacks etc. The debugger uses this dll to display data in your process. It also handles post mortem debugging, i.e. dumps of some sort. This dll can safely be distributed with your software. I suggest that you download and install Debugging Tools for Windows. This will give you access to all sorts of tools and the best tool WinDbg.exe and the latest dbghelp.dll is also in that distribution.
In dbghelp.dll you call e.g. MiniDumpWriteDump(), which will create the dump file and that's more or less it. You're done. As soon as you have that file in your hands, you can start using it. Either in the Visual Studio Debugger, which probably even might be associated with the .dmp file extension, or in WinDbg.
Now, there are a few things to think of while you're at it. When checking dump files like this, you need to generate .pdb files when you compile and link your executable. Otherwise there's no chance of mapping the dump data to human readable data, e.g. to get good callstacks and values of variables etc. This also means that you have to save these .pdb files. You need to be able to match them exactly against that very release. Since the dump files are date stamped with the date stamp of the executable, the debugger needs the exact pdb files. It doesn't matter if your code hasn't changed a single bit, if the .pdb files belong to another compilation session, you're toast.
I encourage every windows win32 developer to check out Oleg Starodumov's site DebugInfo.com. It contains a lot of samples and tutorials and how you can configure and tune your dump file generation. There are of course a myriad of ways to exclude certain data, create your custom debug message to attach to the dump etc.
Keep in mind that minidumps will contain very limited information about the application state at exception time. The trade off is a small file (around 50-100 kB depending on your settings). But if you want, you can create a full dump, which will contain the state of the whole application, i.e. globals and even kernel objects. These files can be HUGE and should only be used at extreme cases.
If there are legal aspects, just make sure your customers are aware of what you're doing. I bet you already have some contract where you aren't supposed to reveal business secrets or other legal aspects. If customers complain, convince them how important it is to find bugs and that this will improve the quality of the software drastically. More or less higher quality at the cost of nothing. If it doesn't cost them anything, that's also a good argument :)
Finally, here's another great site if you want to read up more on crash dump analysis: dumpanalysis.org
Hope this helps. Please comment if you want me to explain more.
Cheers !
Edit:
Just wanted to add that MiniDumpWriteDump() requires that you have a pointer to a MINIDUMP-EXCEPTION-INFORMATION (with underscores) struct. But the GetExceptionInformation() macro provides this for you at time of exception in your exception handler (structured exception handling or SEH):
__try {
}
__except (YourHandlerFunction(GetExceptionInformation())) {
}
YourHandlerFunction() will be the one taking care of generating the minidump (or some other function down the call chain). Also, if you have custom errors in your program, e.g. something happens that should not happen but technically is not an exception, you can use RaiseException() to create your own.
GetExceptionInformation() can only be used in this context and nowhere else during program execution.
Crash dumps are a pretty common troubleshooting method and can be very effective, especially for problems that only reproduce at the customer's site.
Just make sure the customer/client understands what you're doing and that you have permission. It's possible that a crash dump can have sensitive information that a customer may not want (or be permitted) to let walk out the door or over the wire.
Better than that there are libraries that will upload crash data back you.
BugDump and BugSplat
And there's the Microsoft way:
http://msdn.microsoft.com/en-us/library/aa936273.aspx
Disclaimer: I am not a lawyer, nor do I pretend to be one, this is not legal advice.
The data you can include in logs and crash dumps also depend on what domain you are working in. For example, medical equipment and patient information systems often contain sensitive data about patients that should not be visible to unauthorized persons.
The HIPAA Privacy Rule regulates
the use and disclosure of certain
information held by "covered entities"
(...) It establishes regulations for
the use and disclosure of Protected
Health Information (PHI). PHI is any
information held by a covered entity
which concerns health status,
provision of health care, or payment
for health care that can be linked to
an individual.[10] This is interpreted
rather broadly and includes any part
of an individual's medical record or
payment history. --Wikipedia
It should not be possible to link health information to an individual. The crash dumps and logs should be anonymized and stripped of any sensitive information, or not sent at all.
Maybe this does not apply to your specific case, so this is more of a general note. I think it applies to other domains that handle sensitive information, such as military and financial, and so on.
Basically the easiest way to produce a dumpfile is by using adplus. You don't need to change your code.
Adplus is part of the debugging tools for windows, as mentioned in the article above.
Adplus is basically a huge vbscript automation of windbg.
What you have to do to use adplus:
Download and install Debugging tools for windows to c:\debuggers
start your application
open a commandline and navigate to c:\debuggers
run this line "adplus -crash your_exe.exe"
reproduce the crash
you'll get a minidump with all the information you need.
you can open the crash dump in your favorite debugger.
within windbg, the command "analyze -v" helped me in at least 40% of all the crashes that only happened at customer site and were not reproducible in house.