I am running the Xunit Test cases as mentioned in my previous question.
How to get passed and fail test case count in xunit using cake(c# make) script
While running the test cases, most of my test cases are failed while trying to access a file from my AppDomain.
The test cases are succeeded in Visual Studio.
From the Error log, I can see that it tries to read the file from Xunit console runner location instead of application location.
Note:
I am using NoAppDomain of Xunit2Settings as false.
When working with unit tests and files, I would recommend first to see if you can avoid using the actual filesystem by using data in memory or assembly resources. That way parallel execution, filesystem locks or similar environment related things won't be an issue.
So if you have the opportunity to refactor the filesystem out of the tests (unless that is what your testing) I would go that route first.
One way to work around the issue you're having is to use absolute paths, you could achieve this by in your tests utilizing the test assembly location and then Path.Combine the relative path to that. There's a good StackOverflow answer explaining on how to get the path of your assembly:
https://stackoverflow.com/a/52956/5883153
A quick fix you could try is using the Xunit2Settings WorkingDirectory to set same current dirrectory as VisualStudio, but that isn't something I've testest or recomend.
Related
We noticed that on a certain dev machine a Visual Studio (2015 update 3) debug build of a C# solution was generating a $RANDOM_SEED$ file alongside every built DLL.
The content of the file is just a single number e.g.
1443972318
Deleting the file(s) then rebuilding resulted in the file being regenerated, with a different number.
This behaviour was also observed when rebuilding a single project in the solution (one which has only the standard C# project refs/dependencies + System.Management).
Note that running a command line build e.g.
msbuild <sln-file>
did not regenerate the file (for build of complete solution or single project).
After a restart of VS, the file is no longer regenerated.
As far as we know this file name is not used in any of our source code, post build steps or internal dependencies.
There are quite a few dependencies on .NET framework classes, including Random and RNGCryptoServiceProvider, and also external dependencies. We don't have complete source code for all these so it's not possible to check exhaustively which if any of the dependencies are responsible.
This is a bit of a shot in the dark but the question is has anyone seen anything similar to this?
EDIT
I'm not surprised this has been downvoted - I appreciate it is pretty open ended, but as I'm currently not able to reproduce this and as it could have potentially serious consequences (random number generator attack?) I have posted it anyway. If I am able to repro I will of course update here.
I have the same file.
After a short investigation I found guilty:
this file is created by NUnit 3.x test adapter.
(You can check it in AdapterSettings.cs from NUnit adapter source code).
The file is used by NUnit to ensure that we use the same random seed value for generating random test cases in both the discovery and execution processes. This is required because the IDE uses two different processes to execute the adapter. It's not actually required (or created) when running the adapter under vstest.console.exe.
I'm writing tests using MSTest. I need to run executable from another project (in solution) before tests starts.
Is there other option beside invoking exe directly under [ClassInitialize]? In this case I'm loosing debugging capabilities.
Something like Multiple startup projects in solution properties.
If it is in the same solution, can't you add a reference to it in your test project and call it directly from [ClassInitialize]. This shouldn't cause you to lose debugging capabilities. Do you not have access to the source code of the executable?
I hear that accessing to database is wrong in testing.
But what about file manipulation? Things like, cp, mv, rm and touch methods in FileUtils.
If I write a test and actually run the commands (moving files, renaming files, making directories and so on), I can test them. But I need to "undo" every command I ran before running a test again. So I decided to write all the code to "undo", but it seems a waste of time because I don't really need to "undo".
I really want to see how others do. How would you test, for example, when you generate a lot of static files?
In your case accessing the files is totally legit, if you are writing file manipulation code it should be tested on files. The one thing you have to be careful about is that a failed test means that you code is wrong and not that somebody deleted a file that is needed for the test or something like that. I would put the directory and the files you need for the tests in a separate folder that is only used for the test. Then in the build up of the test copy the whole folder to a temporary place do all the testing and then after the test delete the temporary files. In that way each test has a clean copy of the files that are saved for the test.
"Pure" unit testing shall not access "expensive" resources such as filesystem, DB ...
Now you may want to run those "integration" tests (or whatever you call them) at the same time as your unit-tests, and use the same framework it's convenient.
You can have a set of files for unit testing that you copy into temporary location as suggested in Janusz' answer, or generate them in your unit tests, or you can use a mock of the FileUtils instead of the real FileUtils when unit testing.
Accessing a database is not "wrong in testing". How else will you test the integration of your code with the database?
The key to repeatable testing is a consistent environment. So long as you start from the same file system or database contents for your tests, you should are not wrong. This is usually handled via a cleanup process at the start of the test suite.
Accessing resources like the database, file system, smtp server, etc. are bad ideas for unit testing. At some point obviously you have to do have to try it out with real files, that's a different kind of test, an integration test. Integration tests are more painful, you do have to take care to make sure your test is starting from a well-defined state, also they will run slower since you're accessing the real file system. On the other hand you shouldn't have to run them as frequently as you would with unit tests.
For unit tests you should be able to take advantage of duck typing to create objects that react to the same methods that the file objects you're working with have. Plus there's nothing to undo with this approach, and the tests will run a lot faster.
If your operating system supports RAM-based filesystems, you could go with one of these. This has even the advantage that an occasional `unix command` in your code keeps working.
Maybe you could create a directory inside your test package named "test_*". Then, the files that you change will be put on this directory (for example: if you create a directory, you will create the directory inside the test directory). At the end of the test you could delete this directory (with only one command). This is the unique UNDO operation that you will execute.
You will put all files that you need to the test on your test directory inside the test package.
I am forcing myself to learn test-driven development, and so far I'm enjoying myself. There's a few quirks that Visual Studio Unit Testing has that is driving me batty though. A bit of background information, my project folder looks like this:
[Root] BitFlex
BitFlex\Code
BitFlex\Debug
BitFlex\Documents
BitFlex\Release
Now of course all the source code is stored in the code folder, and on a build the project output either goes to the debug or release folders depending on the current configuration. Now for my unit testing, I have it setup so the test project is output to either:
BitFlex\Debug\Unit Tests\
BitFlex\Release\Unit Tests\
1) At this point, everything is fine and dandy. There are two problems with this, the first being that when I run a test it cannot find the assembly, as it gives me this error:
Error AssignDefaultProgramTest BitFlex.UnitTests The test assembly 'D:\src\DCOM Productions\BitFlex\Code\TestResults\David Anderson_DCOMPRODUCTIONS 2009-07-31 23_21_00\Out\BitFlex.UnitTests.dll' cannot be loaded. Error details: Could not find file 'D:\src\DCOM Productions\BitFlex\Code\TestResults\David Anderson_DCOMPRODUCTIONS 2009-07-31 23_21_00\Out\BitFlex.UnitTests.dll'.
I cannot seem to find information on this error, or how to resolve it so I suppose that's where everyone's experise around here comes in to play.
2) My other beef is that Visual Studio generates the "Test Results" folder in my code directory, I would prefer to move that to my Unit tests folder in either output configuration. Is there a way to do this, or a better practice to setting up a well organized Unit Test using my folder hierarchy?
By default MSTesting framework runs all tests in an 'isolated' location and not from the binaries directory.
To fix this you can do one of these two:
1. go to the test configuration file and under deployment uncheck deploy the tests.
2. don't use path when looking for external files instead use deploy attribute or the test config to deploy the needed files along with your tests.
For doing TDD with MSTest, turn off deployment. You shouldn't need it for "unit testing."
Also, never ever EVER have VS automatically generate tests for you. What's generated may be fine for some types of functional testing, but are usually very poor unit tests.
I have data files used as input to my unit tests. These files are quite big and I don't want to copy them each time unit tests are executed. Tests are executed without deployment. So I can just put them into folder under my solution, and... how to obtain path to my solution (or test project source code) when unit test is executing?
Because you can run a test project in different ways (TD.NET, Visual Studio, R# etc.), the path used to reference the tests can change.
For this reason, I embed test needed files in my test assembly and draw them out from there.
You can use:
Assembly.GetExecutingAssembly().Location
in your tests to get the path of the assembly containing the unit tests.
Simple, make the location of the files configurable (and testable).
Then either, set it in the unit testing code or set it thru some config file.