I have weird problem when tracing my Media Foundation application using mftrace: when running in mftrace, the application crashes with an access violation. It is running from the Visual Studio debugger, but I need to get mftrace running to troubleshoot a Media Foundation problem. After investigating the problem for quite some time, I have added my own tracing code and discovered the following:
2952 22320,5CE8 13:05:14.78185 CMFStreamSinkDetours::GetEvent #000001CFB1E85BD0 - enter
2953 22320,5CE8 13:05:14.78455 CKernel32ExportDetours::OutputDebugStringA # [23784 ENTER]RtspStreamSinkBase<class RtspVideoStreamSink>::IsMediaTypeSupported
2954 __M_F_T_R_A_C_E___LOG__
The method RtspStreamSinkBase<class RtspVideoStreamSink>::IsMediaTypeSupported is the one actually crashing due to an invalid (but non-null) pointer passed in as media type. I know that from attaching the debugger to the crashed process.
For me, it looks like as if mftrace thinks it is calling GetEvent, but the method actually invoked is IsMediaTypeSupported.
Has anyone encountered such an issue before? What was the solution? What could cause such an issue?
Best regards,
Christoph
Update: I did some further Research and found two possible areas where it might go wrong: first, I am using ATL for my implementation (not only ATL::CComPtr, but also the other base classes like ATL::CCoClass for using the registrar etc.). Although I have worked with detours myself, I do not know how mftrace uses it. I assume that they are overwriting the vtable pointers.
Could it be that using ATL prevents the detours from being installed correctly? Is it, in principle, possible to use ATL for implementing MFT classes? Has anyone done this successfully (most samples I see in the internet do all the COM stuff manually)?
The second observation is that only the debug build crashes in mftrace. It therefore might be that some unintended side effect in debug code (eg assert(ptr = nullptr)) is causing the issue. I have not found one in my own code so far, though.
Could it be that the reference tracking in ATL (#define _ATL_DEBUG_INTERFACES) interferes with detours/mftrace?
I finally found the problem: as it turns out, enabling #define _ATL_DEBUG_INTERFACES is a very bad idea with ATL-based MF classes. After I removed this define, the internal error disappeared and the trace does not crash any more.
Related
I'm currently in the process of learning different techniques malware use to prevent debugging, and I came across an issue I can't seem to really understand. This piece of malware that I am currently trying to figure out and debug uses some kind of anti-debugging technique that doesn't allow me to attach a debugger in the first place.
I have tried using things such as Scylla-hide, I've also tried manually going in and hooking common anti-debugging API calls to see what is going on, but no luck so far. I also tried with WinDbg hoping it would give me some error message I can go off of, and it gave me this:
WinDbg output when trying to debug program
So this got me thinking that it was doing something to prevent a thread being started and that is what is causing any debugger from being able to attach. I also tried injecting a DLL with the typical LoadLibrary CreateRemoteThreadEx technique and seems it doesn't allow me to create a thread that way either. I've looked at as much as possible on anti-debugging and can't seem to find an answer to this. Any nudge in the right direction would be greatly appreciated.
I'm hoping to be able to circumvent this in user-mode, as I'm trying to understand what exactly it is doing to stop me from debugging rather than simply bypassing it. This is my first time asking a question so sorry if anything was hard to understand.
This is the defacto antidebug reference by Peter Ferrie, it's an amazing resource: https://anti-reversing.com/Downloads/Anti-Reversing/The_Ultimate_Anti-Reversing_Reference.pdf
It contains almost every known anti-debug method, you can use this reference and tick each method off your list until you find the ones that are implemented, the book should also help you easily mitigate each method as well.
Per our conversation in the comments, the answer to your question was that the process was hooking both DbgBreakPoint and DbgUiRemoteBreakig. Patching them enabled you to then attach a debugger.
I have an MFC application developed with VS2003
It is working fine in XP vista etc.
But when i have executed it in windows 8, and we use it for some time,
then no window is displayed. Instead of that the a MessageBox with a message 'Out of Memory' is displayed. And the Message box is Having the caption of my application.
This issue is rarely occurred in windows 7 too.
I have tried watching the handles using tools like processexplorer and it is not increasing.
Also many forums says that it is because of increase in unclosed handles or resources.
Can any one suggest how can i find where the issue is. Or any one provide possible reason for this.
I cant setup the devenv in the machine causing the issue. I am confused how to diagnose by executing a test build in that.
Please provide your findings.
Thanks in advance.
You clearly have a memory leak somewhere. It's hard to be any more specific without seeing the code.
A debugger is really the best way to solve this problem. If you can reproduce the problem on your development machine, that would be the easiest case. If not, you can attach a debugger to the running process on another machine, either locally or remotely.
The MFC libraries also support some basic memory leak detection, turned on by default for Debug builds and controllable for other builds using the AfxEnableMemoryTracking function. You can use this feature to obtain information about which blocks of memory were allocated but not properly deallocated (i.e. were leaked).
Like you mentioned, Process Explorer is another good way to track down resource leaks. Are you sure that the handle counts are remaining constant rather than trending upwards over time? If the values in the columns are never changing like the question suggests, then you are surely doing something wrong. Your application has to be creating objects in order to do its job. The point is to make sure that it disposes of them when it is finished.
If you can't reproduce the problem with the running application and have only the source code available, you'll need to go through the code and make sure that every use of new has a corresponding use of delete (and that new[] matches up with delete[]). And in general in C++, you should avoid explicit dynamic memory allocation wherever possible. Instead, use the container classes that are provided either by MFC or the standard library. For example, don't allocate arrays manually, use std::vector to do it for you. These container classes ensure that the memory is automatically deallocated in the destructor when the object goes out of scope.
I have a little problem with visual basic 6 project. Everything works fine in the IDE
but the executable crashes every time, when I run the application. The application uses callbacks to communicate with a C++ dll. Even code as simple as showing a message box fails when the callback starts.
I changed the compilation mode to P-Code and still the problem persist.
Any help would be appreciate.
Thank you all
This sounds like the callbacks may be occurring on a different thread than your application is executing on. [EDIT: As I see Jim has already suggested.] If that's the case, yeah, kaboom just as soon as you "touch" anything OLE related or call into the runtime. Same story as with multimedia timer callbacks, fwiw, and I'd suspect you'll have to take the same precautions as one would with those if this is the case.
The short story with different thread callbacks is that you'll need to post a message to yourself, using PostMessage declared in a typelib so that the Err object isn't set by VB, then let the callback return. You do your own processing on receipt of the posted message. Here's the typelib I used for this with the CCRP Timers library:
http://vb.mvps.org/tools/files/postmessage.zip
Hope that helps...
Who's calling back to whom? Show us a little code.
The IDE can mask real problems, so just being able to run there is no guarantee that what you're doing is supported.
One common problem with callbacks is that VB6's runtime is not thread-safe, so if another thread is calling back into your VB code, you can't do anything that will invoke the runtime -- like access strings or objects.
There are ways around some of these issues, but I think we need to know more first.
Is the code being run from the same location as the IDE? Likely it is a reference problem, and you need to re-register the DLL.
A deployment package should make sure you have everything installed.
A few questions:
Is the executable on the same PC as it was developed, or a different one?
Does the file use a manifest file? If so, does mainfest call XP themes?
Also, if using manifest, does manifest use SXS for OCX files?
I am trying to help a client with a problem, but I am running out of ideas. They have a custom, written in house application that runs on a schedule, but it crashes. I don't know how long it has been like this, so I don't think I can trace the crashes back to any particular software updates. The most unfortunate part is there is no longer any source code for the VB6 DLL which contains the meat of the logic.
This VB6 DLL is kicked off by 2-3 function calls from a VB Script. Obviously, I can modify the VB Script to add error logging, but I'm not having much luck getting quality information to pinpoint the source of the crash. I have put logging messages on either side of all of the function calls and determined which of the calls is causing the crash. However, nothing is ever returned in the err object because the call is crashing wscript.exe.
I'm not sure if there is anything else I can do. Any ideas?
Edit: The main reason I care, even though I don't have the source code is that there may be some external factor causing the crash (insufficient credentials, locked file, etc). I have checked the log file that is created in drwtsn32.log as a result of wscript.exe crashing, and the only information I get is an "Access Violation".
I first tend to think this is something to do with security permissions, but couldn't this also be a memory access violation?
You may consider using one of the Sysinternals tools if you truly think this is a problem with the environment such as file permissions. I once used Filemon to figure out all the files my application was touching and discovered a problem that way.
You may also want to do a quick sanity check with Dependency Walker to make sure you are actually loading the DLL files you think you are. I have seen the wrong version of the C runtime being loaded and causing a mysterious crash.
Depending on the scope of the application, your client might want to consider a rewrite. Without source code, they will eventually be forced to do so anyway when something else changes.
It's always possible to use a debugger - either directly on the PC that's running the crashing app or on a memory dump - to determine what's happening to a greater or lesser extent. In this case, where the code is VB6, that may not be very helpful because you'll only get useful information at the Win32 level.
Ultimately, if you don't have the source code then will finding out where the bug is really help? You won't be able to fix it anyway unless you can avoid that code path for ever in the calling script.
You could use the debugging tools for windows. Which might help you pinpoint the error, but without the source to fix it, won't do you much good.
A lazier way would be to call the dll from code (not a script) so you can at least see what is causing the issue and inspect the err object. You still won't be able to fix it, unless the problem is that it is being called incorrectly.
The guy of Coding The Wheel has a pretty interesting series about building an online poker bot which is full of serious technical info, a lot of which is concerned with how to get into existing applications and mess with them, which is, in some way, what you want to do.
Specifically, he has an article on using WinDbg to get at important info, one on how to bend function calls to your own code and one on injecting DLLs in other processes. These techniques might help to find and maybe work around or fix the crash, although I guess it's still a tough call.
There are a couple of tools that may be helpful. First, you can use dependency walker to do a runtime profile of your app:
http://www.dependencywalker.com/
There is a profile menu and you probably want to make sure that the follow child processes option is checked. This will do two things. First, it will allow you to see all of the lib versions that get pulled in. This can be helpful for some problems. Second, the runtime profile uses the debug memory manager when it runs the child processes. So, you will be able to see if buffers are getting overrun and a little bit of information about that.
Another useful tool is process monitor from Mark Russinovich:
http://technet.microsoft.com/en-us/sysinternals/bb896645.aspx
This tool will report all file, registry and thread operations. This will help you determine if any you are bumping into file or registry credential issues.
Process explorer gives you a lot of the same information:
http://technet.microsoft.com/en-us/sysinternals/bb896653.aspx
This is also a Russinovich tool. I find that it is a bit easier to look at some data through this tool.
Finally, using debugging tools for windows or dev studio can give you some insight into where the errors are occurring.
Access violation is almost always a memory error - all the more likely in this case because its random crashing (permissions would likely be more obviously reproducible). In the case of a dll it could be either
There's an error in the code in the dll itself - this could be something like a memory allocation error or even a simple loop boundary condition error.
There's an error when the dll tries to link out to another dll on the system. This will generally be caused by a mismatch between dll versions on the machine.
Your first step should be to try and get a reproducible crash condition. If you don't have a set of circumstances that will crash the system then you cannot know when you have fixed it.
I would then install the system on a clean machine and attempt to reproduce the error on that. Run a monitor and check precisely what other files (dlls etc) are open when the program crashes. I have seen code that crashes on a hyperthreaded Pentium but not on an earlier one - so restoring an old machine as a testbed may be a good option to cover that one. Varying the amount of ram in the machine is also worthwhile.
Hopefully these steps might give you a clue. Hopefully it will be an environment problem and so can be avoided by using the right version of windows, dlls etc. However if you're still stuck with the crash at this point with no good clues then your options are either to rewrite or attempt to hunt down the problem further by debugging the dll at assembler lever or dissassembling it. If you are not familiar with assembly code then both of these are long-shots and it's difficult to see what you will gain - and either option is likely to be a massive time-sink. Myself I have in the past, when faced with a particularly low-level high intensity problem like this advertised on one of the 'coder for hire' websites and looked for someone with specialist knowledge. Again you will need a reproducible error to be able to do this.
In the long run a dll without source code will have to be replaced. Paying a specialist with assembly skills to analyse the functions and provide you with flowcharts may well be worthwhile considering. It is good business practice to do this sooner in a controlled manner than later - like after the machine it is running on has crashed and that version of windows is no longer easily available.
You may want to try using Resource Hacker you may have luck de-compiling the in house application. it may not give you the full source code but at least maybe some more info about what the app is doing, which also may help you determine your culrpit.
Add the maximum possible RAM to the machine
This simple and cheap hack has work for me in the past. Of course YMMV.
Reverse engineering is one possibility, although a tough one.
In theory you can decompile and even debug/trace a compiled VB6 application - this is the easy part, modifying it without source, in all but the most simple cases, is the hard part.
Free compilers/decompilers:
VB decompilers
VB debuggers
Rewrite would be, in most cases, a more successful and faster way to solve the problem.
I'm using the VBO extension for storing Vertex, normal and color buffers (glBindBufferARB)
For some reason when changing buffers or doing some operation the application crashes with an access violation. When attaching The debugger I see that the crash is in some thread that is not my main thread which performs the opengl call with the execution in some dll which is related to the nvidia graphics driver.
What probably happened is that I gave some buffer call a bad buffer or with a wrong size. So my question is, how do I debug this situation? The crash seem to happen some time after the actual call and in a different thread.
Assuming this is about Windows, NVIDIA has a GLExpert tool. It can print various OpenGL warnings/errors.
In some other cases, using GLIntercept OpenGL call interceptor with error checking turned on can be useful.
If the tools do not help, well, then it's good old debugging. Try to narrow down the problem and locate what exactly causes a crash. If it's a NVIDIA specific problem, try installing different drivers and/or asking on NVIDIA developer forums.
I think you may just have to brute force that one.
I.e. comment out any vbo using lines a few at a time till your program doesn't crash anymore. Then you'll have an idea of which lines to focus your attention on and really scrutinize the parameters you're passing.
Also try sprinkling glError() calls liberally around your program. Often if you pass a bogus parameter glError will tell you something is wrong before it gets to the point of crashing.
One of the best OpenGl/D3D debugging tools is nVidia's NvPerfHUD. It won't help you find your exact problem, but it does provide another view of what you are sending into the rendering pipeline.
However, I will say that I've only used it with D3D applications so I don't know if it helps as much with OpenGL programs.
EDIT:
I'm not sure why this got voted down. I have debugged VB and IB problems with NvPerfHUD before. Simple things such as bad primitive counts and be diagnosed by looking at each individual draw call.