I've encountered a wierd behaviour of a groovy script when running the whole suite. I have a script which orders my testcases alphabetically before they run and it seems to be running forever even when the whole test suite finishes.
After I click on it to see details and immediately go back to test suite it shows that it's finished and no longer running.
Is there something wrong with my script please? I don't see any infinite loops or anything like that. Is it just a bug in the ReadyAPI? Thanks for advice.
My sorting script:
ArrayList<String> testCaseList = new ArrayList<String>();
for (testCase in testRunner.testCase.testSuite.getTestCaseList()) {
testCaseList.add(testCase.getName());
}
testCaseList.sort();
int i = 0;
for (tCase in testCaseList) {
def curCase = testRunner.testCase.testSuite.getTestCaseByName(tCase);
curIndex = testRunner.testCase.testSuite.getIndexOfTestCase(curCase);
testRunner.testCase.testSuite.moveTestCase(curIndex,i-curIndex);
i++;
}
Currently, looks you have separate test case for sorting. But actually, that is not a valid test case of yours.
So, the first change to be made is that the script should be moved from test case to Setup Script of test suite.
Here is the Test Suite's Setup Script which does it alphabetical order. Should be paid special attention in case if the test case names have numbers in it, have to do natural order. Otherwise, it should be ok.
Please follow the in-line comments.
//Get the sorted order of the test case which is expected order
def newList = testSuite.testCaseList.name.sort()
log.info "Expected order of test cases: ${newList}"
//Get the current index of the test case
def getTestCaseIndex = { name -> testSuite.getIndexOfTestCase(testSuite.getTestCaseByName(name))}
//Closure definition and this is being called recursively to make the desired order
def rearrange
rearrange = {
def testCaseNames = testSuite.testCaseList.name
if (testCaseNames != newList) {
log.info testCaseNames
newList.eachWithIndex { tc, index ->
def existingIndex = getTestCaseIndex(tc)
if (index != existingIndex) {
testSuite.moveTestCase(index, existingIndex-index)
rearrange()
}
}
} else {
log.info 'All cases sorted'
}
}
//Call the closure
rearrange()
With that Setup Script, when test suite is executed, automatically the test cases are moved alphabetically. Hence, no separate test case required for just ordering.
Now, the suite gets executed with desired test cases and the current issue mentioned in the question should not be there at all.
Related
I have a method where I tried to parallelise the calc using GPARS and calculate an aggregate boolean 'And' result across the calls. This method is wrapped as a #ActiveObject which will deliver the result as a dataflow - the code below has the original approach where I tried to store the aggregate using AtomicBoolean to protect it.
This didn't work (sometimes my tests would pass others they would fail) on the calculated 'end truth'. To fix this I changed from AtomicBoolean to a Agent(boolean) approach and I think it's 'fixed' it - at least my spock tests are continuously succeeding.
Where was my logic flawed trying to use AtomicBoolean to build the final result? It felt like it should work - but doesn't and I don't understand why.
Method below - I've put the original version, and the corrected version below
#ActiveMethod
def evaluateAllAsync () {
AtomicBoolean result = new AtomicBoolean(true)
GParsPool.withPool {
// do as parallel
conditions.eachParallel { condition ->
println "evalAllAsync-parallel intermediate result start value is ${result.get()} and condition with expression ${condition.expression} evaluated to ${condition.evaluate()}"
result.getAndSet(result.get() && condition.evaluate())
println "recalc bool value is now ${result.get()}"
}
}
println "evalAllAsync-parallel final result value is ${result.get()}"
result.get()
}
Fixed issue by using Agent form like this
#ActiveMethod
def evaluateAllAsync () {
def result = new Agent (true)
GParsPool.withPool {
// do as parallel
conditions.eachParallel { condition ->
println "evalAllAsync-parallel intermediate result start value is ${result.val} and condition with expression ${condition.expression} evaluated to ${condition.evaluate()}"
result << { def res = it && condition.evaluate(); println "start> $it and finish> $res"; updateValue(res)}
println "recalc bool value is now ${result.val}"
}
}
println "evalAllAsync-parallel final result value is ${result.val}"
result.val
}
I put debug println in here just so I could see what code was doing.
The version with the Agent to protect the bool aggregate value appears to be working.
Why doesn't the Atomic Boolean work?
Well, this is an issue in how you use AtomicBoolean. You always overwrite by force (getAndSet()) the value stored in it and ignore the possibility that other threads could have changed it while the current thread is busy 'evaluating'.
You perhaps wanted to use the compareAndSet() method instead:
def valueToUse = result.get()
result.compareAndSet(valueToUse, valueToUse && condition.evaluate())
I have following Extension Method which is just a negation of Linq.Any()
These two UnitTests do test it completely
[TestMethod]
public void EnumerableExtensions_None_WithMatch()
{
Assert.IsTrue(_animals.None(t => t.Name == "Pony"));
}
[TestMethod]
public void EnumerableExtensions_None()
{
var emtpyList = new List<Animal>(); { };
Assert.IsTrue(emtpyList.None());
}
As you can see in the picture, when I run a Code Coverage Analysis, the delegate body is not covered (white selection), because of the deferred execution.
This question comes close to the problem:
Code Coverage on Lambda Expressions
But does not quite solve it: Since the List must stay empty, it's impossible to actually step into that piece of code.
I am tempted to mark the segment with [ExcludeFromCodeCoverage] ...
How would you write the UnitTest?
You need to test that None() returns false when given a non-empty list of Animal. As it is, you never execute your default lambda expression.
You might even find a bug...
This is the correct way to write the Test. Even found a bug!
public void EnumerableExtensions_None()
{
// _animals HAS entries
Assert.IsFalse(_animals.None());
}
Code Coverage 100%!
I have the following Test:
[Test]
public void GrantResourceOwnerCredentials_NullClientID_ThrowsArgumentNullException()
{
Assert.Throws<ArgumentNullException>(() =>{ new ApplicationOAuthProvider(null,null); });
}
The test passed, but when i run code coverage, it highlights
new ApplicationOAuthProvider(null,null);
as "Coverage Partially Touched Area".
How do i fix that?
By the way, I am new to Unit Testing/Code Coverage so go easy on me.
Thanks
When you see a result of "Partially touched", what that means is that there are multiple code paths that can be traversed through a method (such as through an if block, a null check, etc.), and only one path was executed by your tests.
For example, if you had the following method:
public string IsThisEvenOrOdd(int number)
{
return (number % 2 == 0) ? "Even" : "Odd";
}
And you write the following unit test:
public string IsThisEvenOrOdd_PositiveEvenNumber_ReturnsEven()
{
var number = 2;
var expected = "Even";
var actual = IsThisEvenOrOdd(number);
Assert.AreEqual(expected, actual);
}
If you run that test, it should succeed. However, it only tests one path - the "Even" path. You'll need to write at least one other test to test the "Odd" result before that method will show up as "fully covered" by MSTest.
I'm trying to test my application using the XCTest framework.
I want my single test case to fail if some logical condition holds (using an assertion).
I don't want the rest of the code in the test case to run, because this might lead to problems (access to null pointers, for example)
I also want the rest of the test case to run normally, and just the failed test to be marked as failed.
I've noticed XCTestCase has a property called continueAfterFailure.
However, setting it to YES caused the failed test to continue executing lines after the assertion, and setting it to NO caused the rest of the tests not to run at all.
Is there a solution to this issue?
Pascal's answer gave me the idea to achieve this properly. XCTool now behaves like OCUnit when an assertion fails: the execution of the test case is aborted immediately, tearDown invoked and the next test case is run.
Simply override the method invokeTest in your base class (the one that inherits from the XCTestCase class):
- (void)invokeTest
{
self.continueAfterFailure = NO;
#try
{
[super invokeTest];
}
#finally
{
self.continueAfterFailure = YES;
}
}
That's it!
The easiest way is to add:
continueAfterFailure = false
into setUp() method. So it will look like this:
Swift
override func setUp() {
super.setUp()
continueAfterFailure = false
}
Objective-C
- (void)setUp {
[super setUp];
[self setContinueAfterFailure:NO];
}
One option would be to check the condition normally, then fail and return from the test if it is false.
Something like this:
if (!condition) {
XCFail(#"o noes");
return;
}
You could wrap this up in a helper macro to preserve readability.
BDD test libraries like Kiwi are more elegant for this sort of thing, as they make it easier to share setup between many tests which leads to fewer assertions per test.
I am able to use continueAfterFailure and let the other tests run by using this pattern:
self.continueAfterFailure = NO;
#try
{
// Perform test code here
}
#finally
{
self.continueAfterFailure = YES;
}
In Swift projects, I use a helper function (defined in a shared superclass of all my tests which itself extends XCTestCase):
/// Like `XCTFail(...)` but aborts the test.
func XCTAbortTest(_ message: String,
file: StaticString = #file, line: UInt = #line
) -> Never {
self.continueAfterFailure = false
XCTFail(message, file: file, line: line)
fatalError("never reached")
}
As the comment suggests, the call to fatalError is never actually executed; XCTFail aborts the test in an orderly fashion (tearDown is called, next test runs, etc.). The call is only there to trick the compiler into accepting Never as return type since XCTFail returns Void (it does return if continueAfterFailure == true).
Note that self.continueAfterFailure is reset to the default true for every test method. You can also make that explicit in setUp().
I have the following UnitTest:
[TestMethod]
public void NewGamesHaveDifferentSecretCodesTothePreviousGame()
{
var theGame = new BullsAndCows();
List<int> firstCode = new List<int>(theGame.SecretCode);
theGame.NewGame();
List<int> secondCode = new List<int>(theGame.SecretCode);
theGame.NewGame();
List<int> thirdCode = new List<int>(theGame.SecretCode);
CollectionAssert.AreNotEqual(firstCode, secondCode);
CollectionAssert.AreNotEqual(secondCode, thirdCode);
}
When I run it in Debug mode, my code passes the test, but when I run the test as normal (run mode) it does not pass. The exception thrown is:
CollectionAssert.AreNotEqual failed. (Both collection contain same elements).
Here is my code:
// constructor
public BullsAndCows()
{
Gueses = new List<Guess>();
SecretCode = generateRequiredSecretCode();
previousCodes = new Dictionary<int, List<int>>();
}
public void NewGame()
{
var theCode = generateRequiredSecretCode();
if (previousCodes.Count != 0)
{
if(!isPreviouslySeen(theCode))
{
SecretCode = theCode;
previousCodes.Add(previousCodes.Last().Key + 1, SecretCode);
}
}
else
{
SecretCode = theCode;
previousCodes.Add(0, theCode);
}
}
previousCodes is a property on the class, and its Data type is Dictionary key integer, value List of integers. SecretCode is also a property on the class, and its Data type is a List of integers
If I were to make a guess, I would say the reason is the NewGame() method is called again, whilst the first call hasn't really finished what it needs to do. As you can see, there are other methods being called from within the NewGame() method (e.g. generateRequiredSecretCode()).
When running in Debug mode, the slow pace of my pressing F10 gives sufficient time for processes to end.
But I am not really sure how to fix that, assuming I am right in my identification of the cause.
What happens to SecretCode when generateRequiredSecretCode generates a duplicate? It appears to be unhandled.
One possibility is that you are getting a duplicate, so SecretCode remain the same as its previous value. How does the generator work?
Also, you didn't show how the BullsAndCows constructor is initializing SecretCode? Is it calling NewGame?
I doubt the speed of keypresses has anything to do with it, since your test method calls the functions in turn without waiting for input. And unless generateReq... is spawning a thread, it will complete whatever it is doing before it returns.
--after update--
I see 2 bugs.
1) The very first SecretCode generated in the constructor is not added to the list of previousCodes. So the duplicate checking won't catch if the 2nd game has the same code.
2) after previousCodes is populated, you don't handle the case where you generate a duplicate. a duplicate is previouslySeen, so you don't add it to the previousCodes list, but you don't update SecretCode either, so it keeps the old value.
I'm not exactly sure why this is only showing up in release mode - but it could be a difference in the way debug mode handles the random number generator. See How to randomize in WPF. Release mode is faster, so it uses the same timestamp as seed, so it does in fact generate exactly the same sequence of digits.
If that's the case, you can fix it by making random a class property instead of creating a new one for each call to generator.