I'm having a strange issue with unit testing in Visual Studio 2010.
I have a test that passes when I use "Debug test" (without any breakpoints), but failes when I use "Run test".
This test uses external dll's, so I can't debug it properly.
Do you know of any reason why a situation like this is possible?
Why "Debug test" is different than "Run test", when all other parameters are the same?
Switch you solution to Release mode instead of Debug, run a full build, switch back to debug and retry and let me know the outcome, I think the tests will pass....
There can be several reasons, but to pin-point one, you'll have to give us some code to work with.
It could be code exclusion:
#ifdef _DEBUG
//do something
#endif
This will only be executed in debug mode.
It could be optimizations. Although they shouldn't normally affect the behavior, you shouldn't rely on destructors or copy constructors being called.
If you're doing any hacking inside the code, it might also only be valid for debug.
I'd suspect that you are accessing memory out of bounds. One major difference between debug and release is that debug adds kits if padding and special markers to memory allocations to catch certain kinds of errors. This means the memory layouts are often very different between debug and release builds. I would suspect that you are accessing memory out of bounds in a way that the debugger isn't catching, where you find an accidentally valid value in the debug version, but an invalid one in the release version.
Related
I'm having a weird problem with Visual Studio 2010 Ultimate:
One of my Nunit(2.6.2) test cases is failing in the debug mode but passing in the run mode, as if we had completely different code paths for the two scenarios.
Is this a known bug? or is there some option in vs I'm missing?
Please enlighten me!
Thanks a lot.
EDIT - MORE INFORMATION
My application submits some requests to a dll written by a group of people within the organization. The dll does some computation and returns the results back to me.
In a test case exploring the dll's behavior (e.g. submit a request having certain parameter, check the dll's output), running the Nunit test works fine, but debugging the test case gives me an error - an exception was thrown from within that dll.
IMPORTANT: Running/debugging another test case gives me consistent results.
So, for the weird test case:
1, either the dll is good, and something under the debug mode breaks the dll
2, or, the dll has a bug, which is triggered by something under the debug mode.
To my knowledge, the ONLY difference between running and debugging a piece of code in visual studio is that, when debugging, a pdb file is loaded while when running it's not. Essentially, a symbol table is loaded to identify code execution.
Then the issue doesn't make sense in the first place - why would loading a symbol table affect the dll's behavior? (It's unfair to ask anyone to give an explanation without seeing any code; however, since it's corporate prop. code, I can't show it here. Please, if you've ever encountered such things in your career, do share with me what happened in your case - let's hope my problem has the same cause so that I can actually know what went wrong. Thanks)
Are you using code coverage?
If so, try disabling it and run. It will probably work.
For more details, check: http://social.msdn.microsoft.com/Forums/en-US/aba3d58f-f19f-4742-b960-8ac2be29bb88/unit-test-passes-when-in-debug-but-fails-when-run
You may have run into a situation where you're taking the same code path, but the results are subtly different in debug vs non-debug due to optimizations. There are a couple of distinct possibilities here:
Your code has a subtle bug, e.g. a race condition
Your test is being overly specific, e.g. for floating point comparisons where you should use a tolerance
It's a pain not being able to debug, but I suggest you add logging throughout the method and the test so you can see what's going on. (And hope that the logging itself doesn't change the test result, which is also possible...)
Thank you for your response.
I've identified the reason: it's due to a false parameter which drives the dll nuts. My bad.
It still doesn't answer the question why the behavior at debug time goes crazy, but good while running the test case.
However, I guess, since the parameter is wrong in the first place, I can't really blame the dll for going nuts. Anyway, when I passed in the correct param, all went good.
Thanks a lot guys.
I'm trying to debug my unit tests (xunit) under Visual Studio 2012 (via Test Explorer -> Debug Selected Test).
But I can't step into my code of target library. In Modules window I see, what my library symbols are loaded, and the symbol file is found. But in "User Code" column it marked as "N/A" (like all other libraries, include running test library).
Project in Debug configuration isn't optimized, just xml documentation file generation is defined.
How to make VS determine my libraries as "user code" in debug mode?
Solved! Problem was in code. I'm using method which return IEnumerable<> via yield return, and I just call this method (without any iterations on results). So because of yield returns specifics actually this method wasn't called.
So, be careful with yield returns! It may lead to "strange" behaviour like this one :)
I was having this same issue but noticed that my build was targeting "Release" instead of "Debug".
Interesting, since I click on "debug selected tests" I hope it will get debugged, not simply ran on selected build.
Compilation of my code in both modes debug and release is successful. Debug mode execute and works fine, but when i execute release mode, it says "the application was unable to start correctly 0x80000003".
What is this error and why debug mode works fine but not release.
DLLs for debug and release are present in the same directory name "bin". "lib" for both modes also placed in the same directory.
I tried to solve it many ways but not succeeded ? Guide me how to solve this issue? Thanks.
why debug mode works fine but not release
Thre can be many reasons why one build works while another doesn't. The shape and size of data structures can be different because of #ifdef or because the compiler emits different code, the code being executed can be different - again because of constructs like #ifdef or because of the code the compiler is emitting.
That can matter when you've got code with a bug in it. Let's say you've got a bug that miscalculates the length of an array (or the size of a structure or whatever). You do some pointer arithmetic and write some data into memory using that pointer - only you're writing to the wrong place. Whether that mistake breaks your program may depend on what was in the memory you overwrote.
If you're lucky your program crashes almost immediately because what you overwrote was important to some code executed immediately after your bug. If you're only a little bit lucky your program crashes some time later in a completely different part of your program because what you overwrote was important to code far away from your bug. If you're really unlucky your program doesn't crash at all until several years later when a completely unrelated change moves things around in memory so now what you are overwriting suddenly is important.
There are lots of other possible causes of what are sometimes called Heisenbugs
What is this error
One place to look for errors like 0x80000003 is the file WinError.h which you'll find in the SDK you are using (either the one that came with Visual Studio 2010 or one you installed later). Look in WinError.h and you'll find that that E_INVALIDARG is defined to be _HRESULT_TYPEDEF_(0x80000003L).
That doesn't necessarily help though because we we don't know enough about what is returning that error or why it is returning that error or even that your 0x80000003 is in fact an E_INVALIDARG - it could be some other error with the same value, or some piece of code mis-using E_INVALIDARG or something else.
Another possibility is that 0x80000003 is a hard-coded breakpoint exception being thrown - most likely because your program has got to one of those "this should never happen" places where the only thing that makes sense is to throw an exception and crash. If you look in NtStatus.h (in the same place as you found WinError.h) you'll find that STATUS_BREAKPOINT is defined to be ((NTSTATUS)0x80000003L)
how to solve this issue
The trick is to figure what is causing the 0x800000003 (and where in your code it is happening) so you can narrow down why it is happening. Most likely it's an exception but why jump to conclusions?
You can run a release build inside the debugger just as you would run a debug build - that is, build the code using the Release target and then press F5 or Debug | Start Debugging. Look in the Output Window and you may see some information that will help you interpret the error.
You can also use the Debug | Exceptions menu to add a new Win32 Exception with the value 80000003 and set that up to break when thrown rather than to break when unhandled. That way you should stop in the debugger when that exception is thrown (if it is in fact thrown).
Of course it could be that even running your program within a debugger is enough to change things so your problem doesn't occur.
I've got a little problem. my application runs without problems in Debug mode, but crashes in release mode. I can't track down the problem, because in release mode all the Debuginfo appears to be nonsense. However - sometimes in other projects the Debug output is also valid in release mode. What projectsettings do I have to change such that the Debug output is valid in release?
thanks!
Even in Release mode "Generate Debug Info" should be set to "Yes" per default. The problem is that when you're running in Release Mode the compiler optimizes the code which makes it hard for the debugger to display the correct values of variables (it may for instance choose to keep some variables in registers etc.).
There's not much to be done about this, you could always turn off optimization either globally or around a specific function using #pragma optimize ("", off) / #pragma optimize ("", on) around it but this essentially means you're running in Debug Mode again and the crash will probably go away...
If you're comfortable with reading assembly code, you can switch over to disassembly mode and through a little investigation find the correct values of your variables.
Likely, you're making use of uninitialized variables.
In your project settings, set
Configuration Properties > C/C++ > General > Debug Information Format
to Program Database
Then, set
Configuration Properties > Linker > Debugging > Generate Debug Info
to Yes
The good old "debugging with traces" approach may help you having a rough idea of where the problem is. Then read this portion of the code again and chase uninitialized variables.
I have a workspace for running an H.263 Video Encoder in a loop for 31 times i.e. the main is executed 31 times to generate 31 different encoded bit streams. This MS Visual Studio 2005 Workspace has all C source files. When i create a "DEBUG" configuration for the workspace and build and execute it, it runs fine, i.e. it generates all the 31 output files as expected.
But when I set the configuration of the workspace to "RELEASE" mdoe, and repeat the process, the encoder crashes at some test case run.
Now to debug this is verified following:
Analyzed the code to see if there was any variable initialization being missed out in every run of the encoder
Checked the various Workspace(Solution) options in both the modes (DEBUG and RELEASE).
There are some obvious differences, but i turned the optimization related options explicitly same in both modes.
But still could not nail the problem and find a fix for that. Any pointers?
-Ajit.
It's hard to say what the problem might be without carefully inspecting the code. However...
One of the differences between debug and release builds is how the function call stack frame is set up. There are certain classes of bad things you can do (like calling a function with the wrong number of arguments) that are not fatal in a debug build but crash horribly in a release build. Perhaps you could try changing the stack frame related options (I forget what they're called, sorry) in the release build to the same as the debug build and see whether that helps.
Another thing might be to enable all the warnings you possibly can, and fix them all.
Could be a concurrency problem of two threads. The DEBUG configuration slows the execution down, so the problem does not occur. But, only a guess.
Interesting problem.. Are you sure you have no conditional compilation code lurking around that is not being compiled in release mode? i.e:
#if (DEBUG)
// Debug Code here
#else
// Release Code here
#endif
Thats the only thing I can really think of.. Never experienced anything like this myself..
Can you add the debug symbols to the release build and run it in the debugger to see where and why it crashed?
Yeah, those bastard crashes are the hardest to fix. Fortunatly, there are some steps you can do that will give you clues before you resort to manually looking at the code and hope to find the needle.
When does it crash? At every test? At a specific test? What does that test does that the others don't?
What's the error? If it's an access violation, is there a pattern to where it happens? If the addresses are low, it might mean there is an uninitialised pointer somewhere.
Is the program crashing with Debug configuration but without the debugger attached? If so, it's most likely a thread synchronisation problem as John Smithers pointed out.
Have you tried running the code through an analyser such as Purify? It's slow but it's usually worth the wait.
Try to debug the release configuration anyway. It will only dump assemblies but it can still give you an indication of what happens such as if the code pointer jumps in the middle of garbage or hits a breakpoint in an external library.
Are you on an Intel architecture? If not, watch for memory alignement errors, they hard crash without warning on some architectures and those codec algorithm tend to create those situations a lot since they are overly optimized.
Are you sure there are no precompile directives that, say, ignores some really important code in Release mode but allows them in Debug?
Also, have you implemented any logging that might point out to the precise assembly that's throwing the error?
I would look at the crash in more detail - if it's crashing in a test case, then it sounds pretty easily reproducible, which is usually most of the challenge.
Another thing to consider: in debug mode, the variables are initialized with 0xCCCCCCCC instead of zero. That might have some nasty side effects.