When I run my XCTests, I'd like to automatically rerun, once, any integration (unit/ui) test that fails. Is this possible?
This would be done in the same test, without having to hit 'run' again on my tests, or any part of my tests. I'm running all my tests from the command line.
I am doing UI tests that use the network to make calls to the server. If there is a significant server problem, the test will/should fail and report an error. However, if it is only a temporary problem with the request, it would be nice to rerun the test automatically and see if it passes. Also, with the current state of Xcode UI testing there are some occasional problems where it will crash for an obscure reason, and it would be nice to rerun the test automatically to see if it passes the second time.
It would be especially nice if it could output what happened, i.e. "The test failed the first time, because of failure getting refreshed snapshot, but passed the second time"
You can now do this in Xcode without any scripts.
Press ⌘ + 6 or select the test assistant view.
Filter for failed tests
Select tests, right click and choose Run x test methods > Run in all classes
(optional) If you want to run the same tests again press ⌃ + ⌥ + ⌘ + G for a quick way to run them right away.
Task list to accomplish this without fastlane and with granular test reruns
create a script to rerun the test and capture the list of failed tests, then rerun the tests using the -only-testing:TEST-IDENTIFIER flag of xcodebuild.
parse the resultBundle or .xcresult to find the failed tests
write code to erase the device or simulator executing the tests
reboot the device if it is a real one to clear any possible dialogs
use XCAttachment to collect things you care about, like app logs
use xchtmlreport to generate a nice webpage from the results bundle with your snapshots and attachments.
Fastlane can make it possible here is the awesome blog post regarding same
Stabilizing the CI By Re-runing Flaky iOS XCUI Tests
Related
I have several cypress spec files running against a web app in CI/CD build pipelines.
For whatever reason, there is a gap in time between each spec file is run in the pipeline, so the more spec files we add, the slower the build runs. I can see in the logs it's about 30s to a full minute between each spec file is run (I turned off the video record option to ensure that was not somehow related). Recently, it has begun to stall out completely and the build step fails from timing out.
To verify it wasn't related to the number of tests, I did an experiment by combining all the different tests into a single spec file and running only that file. This worked perfectly - because there was only a single spec file to load, the build did not experience any of those long pauses in between running multiple spec files.
Of course, placing all our tests into a single file is not ideal. I know with the cypress test runner there is a way to accomplish running all tests across multiple spec files as if they were in a single file using "Run all specs" button. From the cypress docs:
"But when you click on "Run all specs" button after cypress open, the Test Runner bundles and concatenates all specs together..."
I want to accomplish the exact same thing through the command line. Does anyone know how to do this? Or accomplish the same thing in another way?
Using cypress run is not the equivalent. Although this command runs all tests, it still fires up each spec file separately (hence the delay issue in the pipeline).
Seems like they don't want to do it that way. Sorry for not having a better answer.
https://glebbahmutov.com/blog/run-all-specs/
it does sound strange, but sometimes it happens that the status of a test case that was run a couple of days ago is automatically reset to active.
Do you have any idea why, or where to start investigating?
I mention that:
we did press save and close after the run (the test case was shown as passed for a couple of days);
the test was not included in any other test suite;
the test is a manual one;
it did not happen for test belonging to suits that were marked as completed;
We are using MTM 2012.
Thanks!
I only know a similar behavior regarding automated tests.
You will observe this if
you associate your automated test run with a CI or nightly build AND
if you set up a *Retention Policy" for these build AND
if "Build Delete Options" are set like this:
In this case the test runs will be deleted along with the build.
But it only applies to automated tests to my knowledge...
The behaviour you are observing is kind of similar one: your most recent manual test run gets deleted for some reason; that is the only reason I can imagine for a test case being reset to active "automatically".
Try to investigate in this direction; means what in your environment can cause manual test run to be deleted?
I am quite curious to know if there is a way to achieve the same that you get, when in Xcode you press cmd-u (or select Test from the menu).
I am writing some UI tests for an application, and would love to automate the whole thing.
So far, any attempt to automate via applescript has been quite unsuccessful (I told myself how hard it is to simulate 2 button pressed...); so I thought that maybe there is a proper way to just run the command line commands and have the simulator to pop and run the tests, like it does in Xcode.
Altho if this function was known, I would find something online, but it seems that there is no way to simulate the test command via console.
Am I missing something?
Have you checked out the xcodebuild command?
The man page says:
buildaction ...
Specify a build action (or actions) to perform on the target. Available build actions are:
test Test a scheme from the build root (SYMROOT). This requires specifying a scheme.
So, perhaps that will do what you want.
Attempting to automate a test run via the command-line Microsoft Test Manager utility, tcm.exe.
I'm using the following command:
tcm run /create /title:"Automated UI Tests" /planid:27 /suiteid:721
/configid:10 /settingsname:"UI Test Settings 3"
/testenvironment:"SanityEnvironmentSlemish"
/collection:[CollectionUrl]
/teamproject:Main
(I haven't included the collection URL above).
However, I've been getting the following error: "A test run must be created with at least one test case".
This is a bit of a mystery to me as my test suite already has a test case associated with it, which is associated with an ordered test list.
The reason it wasn't working was not because the test run didn't have a test case associated with it, but because the test case in question had a state of "Error" (due to previous test run failures), instead of a state of "Active".
To resolve this, you can choose either to edit the state via the Microsoft Test Manager GUI by going to the 'Test' tab, right-click on the test case of the particular test suite and click "Reset test to active".
Alternatively, if you are attempting to run tests using the Microsoft Test Manager command-line interface (tcm.exe), you can simply use the '/include' command-line switch - which includes all test cases in a test run, regardless of state.
is there a way? do I have to wait for building every time I start the test? I want to build from visual studio not from test
thanks
Any time your code changes and you run your test it is going to do a build... so technically you can run your test over and over again and they will only build the first time, but once you run your test why would you run them again without making a code change?
Couple of things that I use that make your test run faster are:
Check the box for "Only build startup projects and dependencies on Run", located Options->Projects and Solution->Build and Run.
Learn the short cut keys
a. "Ctrl+R, T" Runs test in current context, so if your cursor is inside a test method it will only run that test, but when you do it inside of a non test class it will run all of your test.
b. "Crtl+R, Ctrl+T" Debug test same except debug.
c. Others can be found here, those are 2008 if you need to reference others you can find them via google.
Make sure your test are not calling the database or other time intensive resources, use mocking and stubbing.
Run only small sets of test, ie if I am working in a service class I run only the service class test.
Edit: Reading your question again if you want to build and not from a test you can just go to the menu and click Build->Build Solution or press F6. Also it would be helpful if you indicated which version of visual studio you are using because 2010 is different in the sense that you have to click refresh. Either way are you able to clarify?
This is an old question, but I keep seeing people ask it and the issue is still true in VS2017, and it's also true of other test frameworks (Xunit, etc) run from within VS.
I don't know how to make VS stop building all the time. But I do know how to circumvent the compile - run your tests from a console runner, not from within VS. If you're using ReSharper, it has one.
If you aren't using ReSharper, for MSTest, you can start here. https://msdn.microsoft.com/en-us/library/ms182489.aspx
If you aren't using ReSharper, for XUnit, you can start here. https://xunit.github.io/docs/getting-started-desktop.html#add-xunit-runner-ref
Any changes to source code cause compilation, because in order to run tests VS needs up to date DLL with tests.
If you have already compiled project then you can run test multiple times without compilation.
PS: I run MSTest using TestDriven.NET as for me it is faster.