I'm running tests in Visual Studio using "Test Explorer" with NUnit and a .runsettings file (specified by choosing the option in the GUI "Select Settings File")
My settings file (called mytests.runsettings) is:
<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
<RunConfiguration>
<DisableAppDomain>True</DisableAppDomain>
</RunConfiguration>
<ForceListContent>true</ForceListContent>
<NUnit>
<DomainUsage>None</DomainUsage>
</NUnit>
</RunSettings>
I have verified that it is loading this file (verified by adding the framework node and setting it to a fake version, which then results in an error).
But no matter what I do, it doesn't run without an AppDomain!
Running from the command line does work:
nunit3-console.exe --domain=None --inprocess MyTests.dll
What do i need to do to get it to use that setting in NUnit?
I don't believe you can do that.
There is no recognized DomainUsage element under NUnit in the runsettings file. DomainUsage is an internally used property, which will be honored if set. But you can't set it that way. Your DomainUsage element is simply being ignored.
If the adapter received your DisableAppDomain setting, then it would set DomainUsage to None. However, I don't believe it is actually receiving it.
Point 2 requires some explanation. Note that I haven't worked on the adapter for a few years and I'm going from memory but here it is...
The DisableAppDomain setting was added to allow Visual Studio to force NUnit to try to run without using an AppDomain. The Test Explorer is supposed to set things up so that it's possible to run that way, i.e. by making sure everything is already available in the current domain.
In order to prevent misuse of the feature, I believe that Test Explorer always overrides any user-provided setting. Again, this is from memory of work that was done a few years ago, but it seems as if the results you are seeing validate it.
The rationale for this past decision was that Test Explorer is completely responsible for setting up the Process and AppDomain used to run the tests. The user has no way to impact that and neither does NUnit. Of course, when using the console runner, that's not the case - control is in the hands of the user.
Something else to investigate is why you feel the need to run without a test AppDomain being created. But that's probably another question. :-)
I'll ask some other folks who may have a better memory than I to look at this as well.
UPDATE:
#Terje, who maintains the adapter now, replied and confirmed that there is no way to set DomainUsage in the runsettings file or any other way we know of when running under the test adapter. The docs have been corrected to avoid the implicit suggestion that it's possible.
We believe, but have not confirmed experimentally, that TestExplorer creates it's own AppDomain whenever it uses this setting to suppress its creation by the test adapter.
This answer is a follow up to the answer from #charlie above, just need some more space here.
I've checked on the domains that are being created, based on whether the DisableAppDomain is set or not.
When the disable app domain is not set, the appdomain is created by NUnit with applicationbase and friendly name as shown below:
The NUnitCheckDomain is the test dll.
When the disable app domain is set, NUnit no longer sets its own domain, and you then see it runs under the testhost appdomain, which is the process that runs all tests:
So this seems to work the way it should and can.
Do you need it to run under some other app domain than one of these?
BTW: If I run the NUnitCheckDomain test assembly using the NUnit3 console, then it a) works when run directly b) crashes when used with the parameters you give above (domain=None and --inprocess) with not able to load NUnit.Framework. #charlie - Any reason this should not work the same using NUnit3-Console ?
Related
I'm writing a unified project for 3 smart TVs. I have also 3 configurations created in Visual Studio. Now I want to execute some CLI scripts depending on selected configuration.
The problem is in new ASP.NET 5 project I don't have an editor for post build events.
I know I have to do this in project.json. What I found is:
"scripts": {
"postbuild": ""
}
But using this one I can't create different CLI scripts for different configurations.
I found also:
"configurations": {
},
And I guess this is probably what I want, but... How to use it? Intellisense has no power here and also I wasn't lucky searching the Web...
[edit]
Maybe I should try with .xproj?
You'll need to build a master script which uses the available context and environment variables to switch and run the other scripts of your choice.
In addition to the list of variables Here for compile, you also get these for publish related scripts and then these are available everywhere, as are environment variables returned by Environment.GetEnvironmentVariable, which can be seen here.
The image below shows the intellisense from the VS2015 Update 3 RTM, but it's misleading, since you get others depending on the script block you're using:
So, your full list of context variables that you can use to control flow in your scripts is:
Every script block:
%project:Directory%
%project:Name%
%project:Version%
Compile specific:
%compile:TargetFramework%
%compile:FullTargetFramework%
%compile:Configuration%
%compile:OutputFile%
%compile:OutputDir%
%compile:ResponseFile%
%compile:RuntimeOutputDir% (only available if there is runtime output)
%compile:RuntimeIdentifier% (only availabe if there is runtime output)
%comiple:CompilerExitCode% (only available in the postcompile script block)
Publish specific:
%publish:ProjectPath%
%publish:Configuration%
%publish:OutputPath%
%publish:TargetFramework%
%publish:FullTargetFramework%
%publish:Runtime%
I investigated on this a bit but did not really get to any good result.
There are some project variables that are exposed in scripts. Unfortunately, those are very limited:
%project:Name% gives you the project name
%project:Directory% gives you the project directory
%project:Version% gives you the project version
So there is no way to access the build configuration or the environment here.
The configurations option in the project.json is also limited to build configurations and only allows declaring compilation options there, so that also doesn’t work.
Unfortunately, there also doesn’t seem to be another way to solve this. At least not right now. I would consider myself sending a pull request to DNX to add some additional project variables which one could use but at the moment, it doesn’t really make any sense to invest time into DNX: After all it’s being replaced by the dotnet CLI. We’ll see if that one will come with functionality to access the environment—and if not, I might end up submitting a pull request to add this functionality. But until we get there, I’m afraid there is no solution for this.
So our TFS 2013 builds do everything correctly except report code coverage. I've seen similar questions here, e.g. TFS 2013 - No Code Coverage Results, but we've already tried the suggested fixes with no results.
UPDATE 1 — we've taken more steps to try to resolve this; here's the complete list:
Ensured Visual Studio Ultimate was installed on the build server
Tried setting the build definition's "CodeCoverageEnabled" to true as well as the "Code coverage is enabled" setting
Ensured the build was using the Debug configuration and that PDBs were being generated for the DLLs to be tested
Added a .runsettings file with the correct ModulePath included under <CodeCoverage> (verified in build log that the module path was being interpreted correctly; it would produce errors if we intentionally malformed it)
Checked in .runsettings file
Set build definition to "Custom" and pointed to .runsettings file
The build process itself works fine. We can get code coverage results when we build the project locally in the IDE. On the build server, both MSTest and NUnit test projects run fine, and we see pass/fail results as expected. The "No Code Coverage Results" message still plagues us though.
Update 2 -
Here is what we see in the run log:
Somebody suggested homegrown code-coverage calculator in https://stackoverflow.com/a/16198120/141508, but it'd be a crime to spend $150 bazillion-thousand dollars on TFS 2013 & VS Ultimate 2013 with MSDN and still not have this one basic function working.
Add a run settings file to source control. Set the tests to custom and point to the run settings file. More info on using the .runsettings file can be found on msdn: http://msdn.microsoft.com/en-us/library/jj159530.aspx
I was experiencing the same problem. My issue was with the ModulePath. The MSDN examples suggest you can just use the name of a target binary. That was not working for me. However, when I made the name a regular expression, it worked. I am also dumping build output into one folder so that pdb and other reference files are found. Hope that helps.
<ModulePath>.*Administration\.dll.*</ModulePath>
I am using local build server with Visual Studio online with a .runsettings file and I had exactly the same issue.
None of the trickery above helped, so I tested the build script on the hosted build controller and it worked fine, so I decided the problem must be the build server itself.
I changed the Build Service account from "Network Service" to a regular windows user account in the TFS Configuration Tool and now code coverage is collected. Note that this user will need access to the TFS build directories.
I found this question because I saw something peculiar on this article. (Look for the "Delay" setting that defaults to 60).
d. Add a new argument ‘Delay’, enter details as mentioned below
Name – Delay, Direction – In, ArgumentType-Int32, Default Value – 60
This argument is required to delay coverage check, so that required build details are filled up by the build agent, this delay varies from
system to system, in some cases this might not be required at all.
http://www.prowareness.com/blog/failing-build-on-insufficient-code-coverageblock-coverage-part-3/
Maybe try putting in a "delay" work-flow item in the template you are using.......
I have web config transformations set up that work fine when I publish to something like Azure. But selecting between Release and Debug locally seems worthless if the web.config transformations don't actually get applied when running locally.
Is there any way to get these transformations to apply when I click "run"? If not, what's the point of having that dropdown? I'm genuinely curious, not asking rhetorically.
The idea for web.config transforms is that you have some settings that change between environments. For example, you might have a local development connection string, and you'll always use that when running/debugging locally. When you publish to your production server, though, you want to use your production DB. Debug vs. Release configurations at build time just change some settings in how your project is built (most commonly, how debug symbols are generated or what compiler optimizations are enabled), whereas web.config transforms are changing the content of what gets deployed (in web.config).
If you do want to apply the web.config transforms locally on F5, there are extensions that allow you to do so. Slow Cheetah is one such. However, in many cases, you probably don't need this as your web.config will stay mostly constant whenever you run locally.
Almost all IDEs have release and debug modes, in the debug mode, the code is compiled using debug flags and not so much optimization, i.e., some relation between the built program and the source code is kept, in this way you can run your code using a debugger, it is a tool that allows you to track and control the flow of execution of your application, for example the value stored in some variable in some situation in your code, this in order to expose semantic problems that your application may have, i.e., when your code is not actually doing what you expect it to be doing (most of the people does this using print statements everywhere at least once).
The release mode is aimed to produce an optimized version of your application, without debugging/profiling flags.
Notice that this have nothing to do with debug mode of web applications, where we set automated views to show more information when a run time error occurs, it just have to do with optimization or debugging of the actual program running underneath. So setting debug or release mode may not change whether a web application shows tracebacks or not, but it certainly changes the performance of the application running locally in your server.
Regards.
I have a problem which has received plenty of attention which, despite much googling, I have been unable to resolve. I have a Test Project attached to my Visual Studio 2010 MVC 2 Application. When I attempt to run my tests, I get:
"no tests were run because no tests are loaded or the selected tests are disabled."
Following this, I followed the instructions in these posts, to no avail:
MSTest: No tests are run because no tests are loaded or the selected tests are disabled = my GUIDs are right, and the assembly IS being built - I can see the file in explorer.
MSTest Not Finding New Tests = reopening the solution & rebuilding didn't result in any change
http://richallen.blogspot.com/2008/05/ms-test-re-enabling-ignored-tests.html = the tests do not even appear in the test list browser under "loaded tests"
Furthermore, the output console reports:
"Could not load file or assembly 'file://\shared\shared\IT\Development\TPS\TPS.Tests\bin\Debug\TPS.Tests.dll' or one of its dependencies. Operation is not supported. (Exception from HRESULT: 0x80131515)"
I have confirmed the build settings on the Test Project are the same as those for the main project (Any CPU targeted).
Therefore, is it a problem accessing the resource over network share? Otherwise, does anyone have a suggestion?
If you want to continue using a network share to host assemblies in .NET 4, you can change a Visual Studio configuration option to grant those assemblies full trust. You need to edit C:\Program Files (x86)\Microsoft Visual Studio 10.0\Common7\IDE\devenv.exe.config and add the following line:
<loadFromRemoteSources enabled="true"/>
To the configuration/runtime element. This is described in (slightly) more detail at http://msdn.microsoft.com/en-us/library/dd409252%28VS.100%29.aspx. I would not recommend making this change without understanding the security implications of doing so, some of which are outlined in that MSDN article.
In general, though, I agree with the previous answer. Hosting Visual Studio projects on a network share is going to create a large number of problems for very little benefit.
Having your source code on a share is plain wrong (period) and will lead to all sorts of 'gremlins'.
Do yourself a favour, use Source Code Control and have a local copy of the source code. You will waste a lot less time, and as a bonus you will be able to track who changed what.
If you go with TFS, the Visual Studio TFS Branching Guide 2010 is a valuable resource.
If you use SubVersion, then the Red Bean Book is excellent.
I was running all local but still had the problem. I found that what caused it was removing an xml tag during manual editing.
add:
CodedUITest() inside of tag brackets
on the line immediately above your class where all your test methods are.
add:
TestMethod() inside of tag brackets
on the line immediately above your test methods you want to run.
build project and run.
I tried the following steps while encountering this problem and luckily the issue got resolved...
Close the solution and open the empty VS editor and Check out the testrunconfig file
Open the solution and under CodeCOverage column, uncheck and check the dll's available( these dll's would be having a warning symbol)
Rebuild the solution and now run the test cases.
Hope this resolves the issue... :)
I struggled with this for days, and didn't find the answer (for my situation) anywhere, so though I'd jot down my experience...
So I had the same problem, doing what I thought was local testing on a test project created locally.. (I'm a novice...) but returning the same error mentioned above : /
Anyway it seems that VS2010 had by default placed my project dir within the library folder, which was classified as network, subsequently all files within were 'unavailable offline'.
By moving my project dir to c:// my project files became index-able. (much to my relief!)
I set a breakpoint in my code in Visual-C++, but when I run, I see the error mentioned in the title.
I know this question has been asked before on Stack Overflow (Breakpoints cannot be set and have been disabled), but none of the answers there fully explained the problem I'm seeing. The closest I can see is something about the linker, but I don't understand that - so if someone could explain in more detail that would be great.
In my case, I have 2 projects in Visual C++ - the production dsw, and the test code dsw. I have loaded and rebuilt both dsws in debug mode. I want a breakpoint in the production code, which is run via the test scripts. My issue is I get the error message when I run the test code, because the break point is in the production code, which isn't loaded up when the test starts.
Near the beginning of the test script there is a mytest_initialize() command. I imagine this goes off and loads up the production dll. Once this line has executed, I can put the breakpoint in my production code and run until I hit it. But it's quite annoying to have to run to this line, set the breakpoint and continue every time I want to run the test.
So I think the problem is Visual C++ doesn't realise the two projects are related. Is this a linker issue? What does the linker do and what settings should I change to make this work?
Thanks in advance. Apologies if instead I should be appending this question to the existing one, this is my first post so not quite sure how this should work.
[Update 1] I think Chris O has identified the problem. I'll add a further update if I'm able to work out how to use the project settings to make this work.
It sounds like you are using VC6, since you mention dsw files. I think that is as good as it gets in VC6, you have to manually add the breakpoint after your module is loaded from LoadLibrary. Actually, there might be a project debug setting, so you can specify which DLLs to load when debugging your project, that will keep your breakpoints enabled when hitting F5.
You can also try attaching the debugger after you know the mytest_initialize() has been called, that might keep your breakpoints enabled.
I had this issue sometimes, but always pass this with some code replacement actions.
Here is some guy post, how he had fixed it.
Hope it helps.
In my case i solved this by setting the DLL project containing the breakpoint as Active Project and changed Debug settings for this project (right-click project>>settings>>Debug tab) to point to the project that actually runs and accesses the DLL. "Executable for debug session:" and "Working directory:" should be set to the executable that you actually want to run and its corresponding directory.
Hope this is of any help.
right-click project>>settings>>Link tab
check on Generate debug info
check on Link incrementally