Can I run an XCTest suite multiple times? - xcode

Question

Is it possible to have Xcode run your unit tests multiple times?
I had an issue in several unit tests that caused intermittent failures. Now that I think I've fixed it, my only option appears to mash ⌘ + U until I'm 95% confident the bug is gone.
I know other unit testing frameworks make it quite easy to run a single test, test case, or test suite multiple times. Do we have this luxury in XCTest yet?

Answer 1

For me it works in swift
override func invokeTest() {
for time in 0...15 {
print("this test is being invoked: \(time) times")
super.invokeTest()
}
}

Answer 2

Try overriding invoke test: https://developer.apple.com/documentation/xctest/xctestcase/1496282-invoketest?language=objc
- (void)invokeTest
{
for (int i=0; i<100; i++) {
[super invokeTest];
}
}

Answer 3

It might help you to use
func testMultiple() {
self.measureBlock() {
...
XCTAssert(errMessage == nil, "no error expected")
}
}
This runs the code inside self.measureBlock() multiple times to measure the average time.
It is work to change the code, but you might want to know the execution time anyways.
This answer might be close enough to what you want and it is easy to do.

Answer 4

One alternative is to do this via the command line. You can run a single test using the -only-testing argument, and avoid building using test-without-building i.e. (new lines added for clarity)
for i in {1..10}; \
do xcodebuild \
test-without-building \
-workspace MyApp.xcworkspace \
-scheme Debug \
-destination 'platform=iOS Simulator,OS=11.2,name=iPhone 8' \
-only-testing:MyApp.Tests/TestFile/myTest;
done

Answer 5

Try using a for loop:
func testMultiple() {
for _ in 0...100 {
...
XCTAssert(errMessage == nil, "no error expected")
}
}
Note this doesn't work within a self.measureBlock(). You'll get an NSInternalConsistencyException: Cannot measure metrics while already measuring metrics
However, you can CALL this within a measureBlock():
func testMultiple() {
for _ in 0...100 {
...
XCTAssert(errMessage == nil, "no error expected")
}
}
func testPerformance() {
self.measureBlock() {
self.testMultiple()
}
}
Xcode 8 runs the measureBlock code 10 times.

Answer 6

I had used the invokeTest() override in the past (Xcode 10) with great success. But now in Xcode 11 its not working (for me at least). What I ended up doing was:
func test99Loop() {
for i in 0..<LOOP_COUNT {
if i > 0 { tearDown(); sleep(1); setUp() }
test3NineUrls()
do { tearDown(); sleep(1) }
setUp()
test6NineCombine()
print("Finished Loop \(i)")
}
}
I obviously use setup/teardown, and this is the proper way to do those multiple times (since the first and last are called by Xcode).

Answer 7

You can now do this in Xcode
Edit the Test Plan. In the Test Navigator, at the top you should see "Test Plan: MyAppName (Default)". Tap on this and select "Edit Test Plan".
In the editor, select Configurations, Configuration 1
Under Test Execution you can set Test Repetition Mode and Maximum Test Repetitions.
Run the tests from the menu Product > Test

Answer 8

Xcode has a built-in way you can do this:
Right-click the test and select Run <test> Repeatedly...
That menu has a few configurations, one of them is Stop After where you can select Failure which will stop the execution if it failed during one of the repetitions.
You can also run a test class or the whole test suite repeatedly.