When I run tests in Xcode, as long as I run all the tests collectively they execute normally. However, if I run only one individual test class or one individual test function within a test class, it doesn't execute any tests at all and prints:
Executed 0 tests, with 0 failures (0 unexpected) in 0.000 (0.001) seconds
Upon debugging I've found that it does execute individual tests if I choose a particular other scheme I have, but my other two schemes don't execute the individual tests. I also noticed that if I edit one of the two problematic schemes to use the same Build Configuration as the working scheme, the individual tests do work (each of my schemes uses a different build configuration).
How do I make the problematic build configurations work to let me execute individual tests?
I was able to figure it out!
If I click on my project in the Project navigator, then click on my project in the left sidebar of the page that appears, and then go to the Info tab, I can see at the bottom of the Configurations section it says:
"Use ConfigurationName for command-line builds"
If I click the dropdown and change it to a different build configuration, it makes the scheme that uses that build configuration able to execute individual tests! And, as one might expect, it also makes the scheme previously able to execute individual tests no longer able to.
It would be nice if I could dynamically change the build configuration for command-line builds based on which scheme is active in Xcode, but as of yet I have not figured out how to do so.
So for now, just know that you can manually change this build configuration value whenever you want to run individual tests for a scheme.
Related
I am using TeamCity to run around 15 different test suites consisting of hundreds of test cases whenever any new changes come in the existing project. It is configured in a way that it checks for any new changes in Git every half an hour and if it finds any changes, it starts building the project even in the middle of running the test suites. As a result, the database is getting outdated and the test cases are failing for the wrong reasons. How can I configure it so that if TeamCity is in the middle of running the test suites and if any changes come before it can finish running all the available test suites it will not start building the project. It will wait until all the test suites are run and then start building the project with the latest changes. Please help me out.
You may use a Resource Locker with a quota of 1.
Create the Locker at project level (a project both your builds are a part of). For the example I named it ServiceLocker.
Then, in both builds (Tests and Build project), add the Resource Locker you created.
As a result, when coming to the top of the build queue, both the builds will :
check that the Locker is used by less than 1 build (meaning 0 build) ;
if another build is using this Locker, then the build does not start. It will stay in the queue until the Locker is free ;
if/when this Locker is free, the build takes the Locker and starts.
In other words, it prevents the builds having this Locker to run simultaneously.
Is there a way to show different reports inside TeamCity build page for different types of tests?
We have the only one Gradle build for all checks (and we don't want to have separate builds for every check type). But we want to show separate test reports for every type of tests (UI, unit, etc).
We have the following project structure:
Workspace:
- app project
- cocoapods project
App scheme:
- app target (run)
- ui test target (test)
- unit test target (test)
After upgrading to Xcode 10, Xcode insists on building the entire project, including pods and ui tests, every time I run a single unit test, which is a quite heavy and slow process.
If i create a seperate scheme which only include the unit and/or ui tests, it only rebuilds the tests when I run them. Just as I want.
However, as they are no longer member of the main app scheme, I can no longer press Test on the main scheme, as it no longer contains test targets.
My question is then: Is it normal or recommended to have seperate schemes for tests, or can I prevent the entire project from being rebuild in another way, when running a single unit test?
(note: I have set the Host Application setting to None on the unit tests, so I don't get why it always builds the entire project anyway?)
Why not both? In your "Test All the Things" scheme, add each test target. This scheme should be shared.
But when I'm working in one target, I make a scheme for it alone. (More accurately, AppCode creates one for me.) Such schemes are not shared.
Is there a way to run the unit tests in a debug build and performance tests in a release build without manually selecting and running individual schemes?
I have a unit test and a performance test scheme. In the test configuration for the unit test scheme, I selected debug build, and for the performance test scheme I selected release build. If I run each scheme individually, I get a debug build and a release build respectively.
If I create another scheme that runs both of these schemes, then that new scheme will have its own build configuration. If I set a build configuration of debug for this new scheme, then I will get a debug build for my performance tests as well.
You could use different test bundles and include/exclude what you want.
What I mean is creating your own custom test targets (bundles) and using the Xcode Test Navigator.
So for example creating a MyUnitBundleTests target and a MyPerformanceBundleTests target. They would be two separate test 'bundles' where you choose to included/exclude classes, methods etc.
The Test Navigator displays a hierarchical list of the test bundles and associated, classes, and methods etc included in a project.
You can enable and disable test bundles, classes, and methods selectively by Control-clicking the items in the test navigator list and choosing Enable or Disable from the shortcut menu, thereby enabling or disabling the items in the scheme.
Testing with Xcode - Quick Start
Testing with Xcode - Running Tests and Viewing Results
There are several additional interactive ways to run tests. Xcode runs tests based on what test targets are included and enabled in a scheme. The test navigator allows you to directly control which test targets, classes, and methods are included, enabled, or disabled in a scheme without having to use the scheme editor.
Not sure if this is what you're wanting but this is the only option I can think of for testing without going up to the Scheme Editor.
There might be a way to achieve the desired behaviour but it is more of a hack.
The main problem I see is the desired build configuration for the test action of the scheme. Since the build configuration applies to all targets in the scheme you need to find a way to trick Xcode into building your target with both Debug and Release configuration. That being said, here is the idea:
Export the release configuration for your target into an xcconfig file (xcodebuild -scheme "schemeName" -showBuildSettings >> release.xcconfig and make sure it only contains the release settings)
Create another target for your performance tests that builds the same source files
Set the debug configuration of this new target to the aforementioned xcconfig file of the release configuration (that is basically the part where we fool Xcode)
Add the original and the newly created target to the build action of your scheme and add the test bundles (assuming they reside in a separate test bundle)
Select the debug build configuration for the test action.
If you test your scheme it should now build the sources in both debug and release and should run your unit tests on the debug build configuration and the performance tests on the release configuration.
The setup is a bit fragile as you need to add new source files to both targets. You should be able to automate the process using a ruby script and the xcodeproj gem though.
Hope that helps.
I'm new to load testing in Visual Studio/MSTest, and I created a new Load Test recently to validate some high-traffic scenarios for a WCF service. I want to add this to the tests project for the service, but I don't want the test to be executed whenever I "Run All Tests in Solution" nor as part of our Continuous Integration build-verification process because a) it takes 5 minutes to run, and b) the service call that it is testing generates many thousands of email messages. Basically, I'd like to do the equivalent of adding the [Ignore] attribute to a unit test so that the load test is only executed when I explicitly choose to run it.
This MSDN Article ("How to: Disable and Enable Tests") suggests that the only to disable the test is to use Test Lists (.vsmdi files), but I don't have much experience with them, they seem like a hassle to manage, I don't want to have to modify our CI Build Definition, and this blog post says that Test Lists are deprecated in VS2012. Any other ideas?
Edit: I accepted Mauricio's answer, which was to put the load tests into a separate project and maintain separate solutions, one with the load tests and one without. This enables you to run the (faster-running) unit tests during development and also include the (slower-running) load tests during build verification without using test lists.
This should not be an issue for your CI Build Definition. Why?
To run unit tests as part of your build process you need to configure the build definition to point to a test container (usually a .dll file containint your test classes and methods). Load tests do not work this way, they are defined within .loadtest files (which are just xml files) that are consumed by the MSTest engine.
If you do not make any further changes to your CI Build definition the load test will be ignored.
If you want to run the test as part of a build, then you need to configure the build definition to use the .loadtest file.
Stay away from testlists. Like you said, they are being deprecated in VS11.
Edit: The simplest way to avoid running the load test as part of Visual Studio "Run All" tests is to create a different solution for your load tests.
Why don't you want to use Test Lists. I think is the best way to do that. Create different Test Lists for each test type (unit test, load test...) and then in your MSTest command run the Test List(s) you want:
MSTest \testmetadata:testlists.vsmdi \testlist:UnitTests (only UnitTests)
MSTest \testmetadata:testlists.vsmdi \testlist:LoadTests (only LoadTests)
MSTest \testmetadata:testlists.vsmdi \testlist:UnitTests \testlist:LoadTests (UnitTests & LoadTests)