When using XCTExpectFailure and run the test cases (Cmd + U) From "Show the Test navigator" it show's grey cross (x) Other test cases shows in green Colour.
What is the Grey Indicator (x) In Test Cases ?
Do need to do anything here?
Xcode has four indicators for the result of a test.
Green Tick – The test passed
Red Cross – The test failed
Gray Cross – The test failed, but it was expected to
Xcode will show you this only if you already declared that a test was expected to fail using the XCTExpectFailure API.
It's Xcode way to tell you "hey this test failed but I'm not going to report it as a failure because you told me it was expected." This is useful to avoid blocking a build in CI if because when there's an expected failure.
I would encourage you to use XCTExpectFailure parsimoniously. It's useful to avoid being blocked by a test failure that is unrelated to your work, but you should always take the time to go back and fix the failure, don't leave the XCTExpectFailure call in your test suite for long.
Gray Arrow – The test was skipped
Xcode will show this when a test was skipped using XCTSkipIf or XCTSkipUnless.
Similarly to an expected failure, you don't need to do anything about this because either you or someone else in your team added the API call to skip the test, meaning the behavior is expected. It's useful to skip tests if they depend on certain runtime capabilities that might not be available on the device or Simulator executing a test run.
Grey sign appears when test is run successfully and u can click on it to check the performance results in left bar
Related
I am trying to make my Xamarin.UITest output clearer and easier to work with. Every so often when Xamarin Forms updates the tree changes in subtle ways that break our UITests. Also, when developing a test it isn't always necessarily clear what the query should look like to get to a view element we want our test to interact with.
To address these, when a test fails with an "Unable to find element" error, I want to capture the app's view tree and output it to the test results.
Currently in these cases we have to modify the test code by adding app.Repl(); (see Working With the REPL), re-run the test, wait for the REPL window to appear, type tree, look at the output, type exit to leave the REPL, make my code changes based on what I saw in the tree command's output, and rinse-repeat until I have a working test. Instead, if the test results contains the outputs of the REPL's tree command, I can start making changes to fix the test code immediately and greatly speed up my testing feedback loop.
How could I most easily achieve this?
app.Print.Tree();
I think this is what you searched for.
I am triggering scenarios by calling startcmd.sh OdiStartscen it executes the scenario, but I would like to know if this execution failed or successed without checking the ODI.
Is there a way to elevate return code from OdiStartScen to startcmd.sh
Thanks,
Isn't it as simple as in case of any other OS command?
Once successful any OS should internally return success code (0 in linux) back to the parent process. Otherwise it would be error code (non-zero).
That means you simply need to handle 'success' and 'failure' paths in your ODI package (green and red lines). Simply put some nonsense in the startcmd.sh OdiStartScen call for some quick test so see if it behaves as expected in case of failures. Alternatively you may use same step for both paths (red and green) and test using getPrevStep() function for errors (cf. STATUS property).
Two things worth mentioning:
This makes only sense for synchronous calls (so it waits for your Scenario to finish).
Why don't you use ODIStartScen() as an ODI Tool type of package/procedure step instead? Seems lot more elegant. Same answer applies.
I'm using MSTest with Visual Studio 2012, is it possible to see which test runs first, second third and so forth?
I prefer the tests to run in random orders but I just want to know when tests fail, I like to know the sequence they ran and helps me to figure out why they fail easier.
If you want to know the sequence in which your test ran you can do the following:
A bit of work to do:
Add a breakpoint at the start of your method decorated with attribute 'TestMethod'.
Right click your breakpoint and select 'When Hit...'.
Make sure the checkbox next to 'Print a message' is ticked.
Do this for all your test methods.
Let Visual Studio do the rest:
Debug all your tests by pressing ALT + S, D, A.
Wow! It worked:
Find the sequence in your debug output window (ALT + V, O).
There are two different concepts
Coded UI Test - Mostly automation will be in done in these test
MS Test - Mostly unit tests will be written here
Its not possible to run MSTest in order even if you define .playlist file these will never run in sequence.
However in CodedUI case you will use and create Ordred list.
So the answer is NO. The reasoning behind that is - these are unit test case and all the unit test cases are meant to be executed independently.
I was wondering about something, cucumber returns an exit code 0 (which means "ok" as far as i know) when a Background fails.
Now failing steps shouldn't probably be in the Background (at best in a 'before'-hook i guess??). But does anybody know with what philosophy it returns this exit code?
Is it a bug or a feature?
Appendix:
A more concrete example:
Lets say this code passes:
Feature: Figuring out how Cucumber works
As a developer
I want to find out why cuccies fail, but my build doesnt
In order to have more confidence in my build
Background: logging in into the system
Given I am logged in
Scenario: creating a new test set
When I do something
Then I should see "you've done something"
It returns with exit code 0.
Lets make it fail:
Background: logging in into the system
Given I am logged in
Scenario: creating a new test set
Then I should see "there's no way you see this"
When I do something
Then I should see "you've done something"
The output shows a failing step and it returns with exit code 1
When I move the failing step to the Background:
Background: logging in into the system
Given I am logged in
Then I should see "there's no way you see this"
Scenario: creating a new test set
When I do something
Then I should see "you've done something"
The output still shows it failed, but it returns with exit code 0
I've asked the Cucumber guys (their mailing list) and they agree its a bug
http://groups.google.com/group/cukes/browse_thread/thread/e56699f0fabfc75f
Background is really like Before, with the only difference being, that it is run after Before. And since you're using Background/Before (in general) to set prerequisites for your Scenario(s) it wouldn't be very helpful, if the testing-process fails, which an error code different from 0 would suggest, just because you made a mistake in your prerequisites.
So my guess is, although i can't be sure, that this is a 'feature' and very much intended so.
I'm trying to watch the execution of a VB6 app and I'm running into an issue because once I enter the debugger and then hit Continue, it no longer lets me step through the code until I hit another break point. I want to be able to execute a program without stepping through something until I hit a point where I want to watch it execute. Ideally this would be something to the effect of holding a key down while I pressed a button to 'step into' that function.
Thanks in advance!
[EDIT]: I'm aware that I can use break points to stop the execution. To be more clear, the problem is that I don't know where the execution is going to, so I can't set the break point there (because I don't know where there is). That's why I essentially want to be able to say, 'after this next thing that I do, break, no matter what'. It sounds like this functionality does not exist, but I'm still keeping my fingers crossed.
While the code is running, press ctrl+break (or the 'VCR pause' button in the IDE) then press F8 (or choose 'Step Into'from the Debug menu in the IDE) to continue running the app. The next action will cause execution to break.
Note that the which causes the break will not always be the one you hoped it would be. Particularly annoying is the _MouseOver event which prevents you from doing a mouse down or a timer firing quckier than you can perform your action. Some breaks may even be fatal as regards running your app e.g. where Windows messages have been hooked (subclassing). Also consider there may not be an event handler in code (yet) for your action where it can break. But usually this technique identifies where you should be setting your breakpoint.
There is a Stop statement available for use in VB6 that will drop to the debugger when the statement is executed from code running through the IDE. (Just be sure to remove the all of the Stop statements from the code when compiling a release build.)
There are several techniques you can use.
These two have been mentioned
Using F8 and Shift-F8 to step through the program
Adding Stops (and later removing)
Others
Use a global variable to create a collection. Use it as a stack and have the subroutines you are interested in push and and pop strings. Conversely don't pop anything and you will get a trace.
Use Watches to monitor and break at selection conditions. You can setup just about any condition to break.
Make a Global String and have your procedures set when you enter them. Monitor it through a Watch.
Use Debug.Print in your code. Also Unlike Stop you can leave these in without effecting the production code.
Use the File System Object to create a text file to act as a log.
Sometimes problem only occurs in the Complied version then you need to use MsgBox or log to a text file. MsgBox can alter the behavior of complex user interactions with forms.
These are all techniques I used in debugging an application. If I had to monitor an application I would use Debug.Print. If that doesn't do the trick compile then log to a text file.
If you have something really complex going on then I recommend moving all your code out of the events into classes implementing a Command Pattern. Your commands classes should interact with the form through and interface.
In the Execute method of the command classes you will something like
<save the current state>
<Do your original code>
<save the modified state>
<push the command onto a stack>
What will happen is that you wind up with a list of all the commands you have executed (even things like mouseover) with the state they encountered and the modified state. You can then examine each object in turn to see what is happening. This is nearly the equivalent of creating Undo/Redo
Note however things like MouseOver can push a lot of classes on the command stack so you will have to structure your tests carefully or be overloaded with information. Remember you can always skip pushing the command onto the stack.
The downside of using commands is that you gone beyond debugging into redesigning. You will to decide whether the problem is worth doing this.
You can press the F8 key to step through the code line by line. Alternatively, you can press SHIFT-F8 to step through line by line.
F8 will step you in to a function, where SHIFT-F8 will step you over the function. If you click on the DEBUG menu in the VB IDE, you will see other options too.
EDIT:
You can also put a permanent break point in your code by using:
Debug.Assert False
By doing it this way, the 'breakpoint' is saved in your code. When you compile the app, debug code is ignored.