How to configure WinDbg to be one of the optional debuggers when application hit a user defined break point? - windows

when an application on my machine encounter a user defined break point it windows opens a dialog that let me choose a debugger to attach to the process.
Currently it offers me to choose one of the two versions of visual studio that installed on my machine (2008 or 2010) I want to add to the options WinDbg.
Is it possible?

There used to be (in VS2005, I think) an undocument way to do this, but it doesn't seem to work in VS2008 or 2010. You used to be able to run vs7jit.exe /RegisterOld "C:\debuggers\windbg.exe" "Windbg". I can't find how to add it to the list in VS2008 or VS2010.
However, you can set the default debugger to be WinDbg. This would replace the VSJitDebugger.exe that is the default (which is what gives you the two Visual Studio versions to choose from).
For the .Net Framework 4, you can set this in the HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\AeDebug key under the Debugger value. Just put in the path to WinDbg. For versions of the Framework previous to 4, the key is HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework and the value name is DbgManagedDebugger.
Here's some additional information on launching debuggers: http://blogs.msdn.com/shawnfa/archive/2004/07/15/184490.aspx

Related

Visual Studio MSI Installer - Not making registry changes

I created a Visual Studio MSI Project that runs an custom EXE (which is good). In this exe, there is some code to add additional registry changes to add entries into Internet Options (mainly Local Intranet and Trusted Sites under Security). When the exe is run by itself (and in the debugger), it executes as expected (and puts the correct registry entries in their correct location). However, when the MSI executes the exe file, this part gets ignored even though it runs (I've added Console.Writelines everywhere).
I've been beating my head over this for 2 days and if anyone has had experience with this, a correct direction would be great.
I am using Visual Studio 2013.
Shelling out to an EXE from an MSI is not considered "good". As I recall, one of the many limitation of Visual Studio Deployment Projects (.VDPROJ) is that custom actions are scheduled with impersonation. If the MSI is being invoked from a standard user context UAC elevation won't help because the standard user context is impersonated.
If you want a reliable, best practices based installer, you should use the Registry table. Of course the problem there is another one of the annoying limitations of VDPROJ is that every registry value is a key file of it's own component and you quickly get into self repair hell.
I'd really suggest starting over with another toolset. I can think of several good ones.
I didn't realize that there is a "Registry On Target Machine" option inside of the MSI builder for Visual Studio. All I had to do is right click the project > view> Registry.
There I just had to map out the values in the right places.
Thank you for everyone who answered.

Using windbg from Visual Studio

Is it possible to use windbg commands like !locks and .loadby sos mscorwks from the Visual Studio command window (in a debug session)? I've noticed I can use eg k to print a stack trace, so I was wondering if there is some way to get access to the other commands.
Yes you can do this.
Attach Visual studio 2008 to your process for debugging.
Break at some code.
Go to immediate window
type !load sos
Now you are free to use any of the windbg commands like !EEHeap,!DumpHeap
There are some conditions. You need to enable unmanaged code debugging in your project properties. Normally in 64 bit version studio doesn't support this.
Have fun.
Visual Studio 2012 has the "Windows User Mode Debugger" transport in the "Attach to Process" dialog. This allows you to use windbg commands in the immediate window.
Source: Use the WinDbg Engine in Visual Studio User-Mode Debugging
Starting with Windows Driver Development Kit (WDK) 8.0 the Windows debugger is integrated into Visual Studio (see Debugging Environments). Installing the WDK on top of a Visual Studio installation will enable a developer to pick a number of debuggers from the Attach to Process dialog, or set the debugging engine in a project's Debugger settings. This has been verified for Visual Studio 2012 and 2013. The way I read the MSDN link above, this should also work for any other IDE that is supported by the WDK 8.0 and above.
While WinDbg offers a lot of powerful commands, there are some drawbacks as well. This list is not exhaustive, and merely contains those issues I did encounter:
Cannot attach to more than one process at a time. While not generally an issue, this is quite unfortunate, if you do need to debug more than one process at a time. The Visual Studio debugger can attach to any number of processes simultaneously.
WinDbg will not use the symbol settings from your Visual Studio IDE (tested on VS 2012 Update 4). While you can configure the IDE where to look for debugging symbols, WinDbg will not honor those settings. WinDbg will use the _NT_SYMBOL_PATH environment variable, and/or the configuration set through .sympath.
Different syntax for function breakpoints. This is a minor annoyance, since you cannot use Visual Studio's syntax to specify function breakpoints. This also means, that function breakpoints are ignored that have been set using the Visual Studio syntax. What's more annoying, though, you do get a warning dialog for each when launching a debuggee.
No support for Tracepoints. I've been using tracepoints extensively, to unintrusively insert tracing into a debuggee. If you are using tracepoints as well, this may prove to be more than just an annoyance.

Most reliable way to find the visual studio executable?

Assuming I want the newest flavour of visualstudio that is able to open C# projects, and if there are multiple (pro/express), I'd like to open pro.
edit: I want my own program to open visual studio on the computer of my program's user.
Is this the sort of thing that would be changing with great frequency? If not, I'd just do a search on your PC for "devenv.exe" (the VS executable) and then associate *.cs, *.csproj, *.sln, etc. files to open automatically with the "newest flavor" found that you want to use.
Edit: You could try having your program check the HKEY_CURRENT_USER\Software\Microsoft\VisualStudio registry entry and retrieve the latest version from there, then have your program open the file using the Visual Studio executable at the appropriate location.
There isn't one API available that will simply open the latest version of Visual Studio. This is especially true when you start to consider the impact of expression versions of Visual Studio (and the ones which can open C# projects).
I think you're best bet is to simply create a list of Visual Studio versions that you care about in priority order. Then enumerate them in your process checking for the first one that is available and launching that process.
The most reliable way to find the installed location of Visual Studio is to examine the InstallDir key inside of
32 Bit: HKLM:\Software\Microsoft\VisualStudio{VersionNumber}
64 Bit: HKLM:\Softawre\Wow6432Node\Microsoft\VisualStudio{VersionNumber}

Visual Studio 2005 - can not open form on designer

My team developed a GUI application on Visual Studio 2005, managed C++. Since some deliveries it is not possible to open the form in the designer, even if the source code and the project settings have not been changed. The designer reports this error:
Exception of type 'System.OutOfMemoryException' was thrown.
at Microsoft.VisualStudio.Design.VSDynamicTypeService.ShadowCopyAssembly(String fileName)
at Microsoft.VisualStudio.Design.VSDynamicTypeService.CreateDynamicAssembly(String codeBase)
at Microsoft.VisualStudio.Design.VSTypeResolutionService.AssemblyEntry.get_Assembly()
at Microsoft.VisualStudio.Design.VSTypeResolutionService.AssemblyEntry.Search(String fullName, String typeName, Boolean ignoreTypeCase, Assembly& assembly, String description)
...
We successfully recompiled the project but we still encounter this problem.
Any idea?
This is how I used to debug these issues, Start a second instance of visual studio, load your project and attach to the first instance which also has the project loaded. Now set a breakpoint in the constructor and Page Load events and also any custom paint events that you may have in the form in the second instance and try to open the designer in the first instance, the breakpoints should get hit and you should be able to see what's going on.
I suspect that you have a Design Mode error where an infinite loop (or recursive control creation) occurs on the concerned Form.
One thing that helped me in these kinds of error on Windows Forms would be the following:
Open your Visual Studio 2005 solution for your GUI application. Don't open your form yet
Open another instance of Visual Studio 2005
In the second instance, Attach (Debug -> Attach to Process) the first instance of devenv.exe to the debugger. Make sure exceptions (Debug -> Exceptions) have all exceptions checkboxes under "Thrown" checked.
Now go to your first VS2005 instance and open the form. The second VS2005 instance will stop at the line where the error occurs.
This is a long shot, but try closing and opening the designer several times in a row. I have had the same kinds of problems with the C# Windows Forms designer (VS2005) : the form usually ended up opening correctly (after 5 tries, quite consistently).
I've run into the same issue intermittently when working with a large multi-project solution, or a project with an exceedingly large and complicated windows form.
I was able to solve the problem by enabling Visual Studio to use more than 2GB of memory. Here's the process...
(note: this assumes XP and Visual Studio 2005 - Vista and/or VS2008 will require slight changes)
Edit Boot.ini
Right-click My Computer, properties, Advanced tab. Under Startup and Recovery click Settings. Click the Edit button, and add the /3GB switch to the end of the [operating systems] line:
multi(0)disk(0)rdisk(0)partition(2)\WINDOWS="Microsoft Windows XP Professional" /noexecute=optin /fastdetect /3GB
Make Visual Studio "Large Address Aware"
Run a Visual Studio Command Prompt, and change to the IDE directory:
cd %ProgramFiles%\Microsoft Visual Studio 8\Common7\IDE
Use the microsoft tool editbin to modify devenv.exe:
editbin /LARGEADDRESSAWARE devenv.exe
Now reboot, and you're done!

How to stop the Visual Studio debugger starting my process in a job object?

When I start my process from Visual Studio, it is always created inside a job object. I would like to know how to turn this behaviour off. Any ideas?
I expect that it is created in a job object to be debugged. I want to place my program in a different job object.
It's not the hosting process. I'm talking about a Job Object. This is an unmanaged C++ application.
This happens when devenv.exe or VSLauncher.exe run in compatibility mode. The Program Compatibility Assistant (PCA) attaches a job object to the Visual Studio process, and every child process inherits it. Check if the job name (as reported by Process Explorer) starts with PCA. If so, PCA can be disabled as described in the link.
You can globally disable PCA using Run -> gpedit.msc -> Administrative Templates\Windows Components\Application Compatibility -> Turn off Program Compatibility Assistant -> Enable.
You can disable PCA for specific executables by adding a registry entry. For Windows 7, the appropriate registry key is HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\AppCompatFlags\Compatibility Assistant. In regedit, right-click that key, select New -> Multi-String Value, name it ExecutablesToExclude. Set the value to the full path of denenv.exe and VSLauncher.exe, on separate lines and without quotes. For me, these were:
C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\devenv.exe
C:\Program Files (x86)\Common Files\microsoft shared\MSEnv\VSLauncher.exe
A related issue, on Windows 7, is that executables you build in Visual Studio and run from Explorer (not Visual Studio or the command line) may run in compatibility mode, and again get job objects wrapped around them. To prevent this, your executable needs a manifest that declares compatibility with Windows 7, using the new Application Manifest Compability section. The link gives an example of a Windows 7 compatible manifest. The default manifest provided by Visual Studio 2010 does not include this Compatibility section.
I'm not aware of any ways to control this aspect of processes spawned for debugging by VS.NET. But there's a workaround, which is applicable to any situation in which VS.NET can't or doesn't start your process in the exact way you want:
Start your process (possibly using a wrapper EXE that runs as part of the post-build event), then attach to the newly started process using Tools/Attach to Process. If you break into the debugger as part of your startup code, this won't even be required (and you can also debug startup issues...).
I can't reproduce what you're seeing. I've created an unmanaged C++ application in both VS 2005 and VS 2008 and I have no problems associating that process to a new job object when starting the process in VS.
Are you sure the debugger is doing this?

Resources