Is it possible to run multiple test runs with different test suites at the same time with an account that permits device concurrency?
https://forums.xamarin.com/discussion/39831/run-ui-tests-on-multiple-devices-simultaneously
In this question the answer was this
When you create a test run in Xamarin Test cloud, the second page in the Test Run wizard has an option to run tests concurrently (the Parallelization drop down).
If you are submitting tests at the command line, you can run tests in parallel using one of the following two command line parameters:
--test-chunk to run tests in parallel by method
--fixture-chunk to run tests in parallel by fixture.
But can I test on different devices like in this example?
Device1 - test1, test2
Device2 - test1, test3
Device3 - test4, test5
It is possible to run multiple tests with different test suites at the same time using device concurrency in the Xamarin Test Cloud. This is true whether or not you are using parallelization, however, parallelization does complicate the matter somewhat, because parallelization runs on multiple copies of a single device, and those copies also count against your concurrent devices.
When you select to run on parallel devices, the Test Cloud will automatically run the devices on as many copies of that device that are available.
Example Scenario
Device Concurrency - 3
First Test Run - 1 device selected
Second Test Run - 2 devices selected
Without parallelization - Both tests can run as soon as devices are available, because the concurrency is the total maximum for all tests. You could similarly have three test runs each with a single device and all could start immediately. If you exceed your device concurrency, then your remaining tests will be queued up to wait for another device to be finished.
With parallelization - The first test run may use up 1, 2 or all 3 device concurrency slots; depending on how many devices are available. The slots that are used up by the first test run won't be available for the second test run until tests on them have finished.
Conclusion
Theoretically you can have multiple test runs all using parallelization at the same time; but in practice you might not have enough concurrency slots for them to actually progress concurrently.
You can think of it as a trade-off, for individual test runs on a single device, parallelization will let you get your test results much faster; but subsequent test runs will often have to wait, so it is a tradeoff. But whether you use it or not, you can still queue up more tests afterwards; so there's no "penalty" for adding extra tests beyond what your concurrency will allow to immediately run.
Related
SetUp & Description:
there are 2 test Classes: 1. HomepageTest & SignIntest
Methods under these test classes have Groups and Priorities
**Target is to run these test using groups
Scebario1: When I use Threadcount =1;
below are the behaviours:
1.1. On execution it launches Homepage > executes Priority 1 of Homepagetest > Then launches
Then executes priority1 of SignIntest
After this it executes priority2 of HomepageTest and the priority2 of SigninTest and so on..
1.2. Once Execution finishes, it closes 1 browser (Homepage)but fails to close another browser(SigninPage)
Scenario2: When I use threadcount = 2:
Behaviours:
2.1. It opens 2 Browsers simultaneously In one browser it launches URL(For URL wait for 10 seconds) but in another browser it doesnt launch any URL (Remains blank throughout test process)
2.2: Execution takes place just like 1.1 mentioned above
After execution of few tests then another 3rd broser launches..this 3rd browser closes once exections
finishes but remaining 2 browsers(BlankBrowser and Brwser With URL) remains in open state
2.3 After execution finishes it closes 1 browser but fails to close the browser on which no URL is launched
Quetion1: Why Execution is not on the basis of Group. It is running on the basis of priority as mentioned in 1.1 which looks incorrect.
Ideally it should execute Group1&2 of testClass1
then it should close that browser Then launch 2nd browser and run Group1&2 tests and then close that browser
Question2: Why It fails to close the 1 Browser. I counted the active window using window handle and count was returning 1 whereas it should be 2
Question3: When thread-count =2 then why it is launching a blank browser(As mentioned in 2.1)
I think you are mixing groups and priority, and you try to use the incorrect tool to achieve something. And you have extra code that maybe works in one threaded environment, but it can't work in parallel environment, because parallel environment is asynchronous by nature.
Answer 1:
Groups: Provide only grouping of test methods (by function, or similarity, or by feature).
Priority: Simply a priority, its a type of ordering. TestNG sees the priority and automatically sorts the tests by it, because you have 2 tests with top priority, it runs them first.
Grouping is used to manage multiple test case execution, but by group, let's say you need only feature A tests to run, and they all belong to group A, so you run group A.
There is no ordering implemented on group level, because it doesn't make sense. Grouping is grouping, not ordering. (Example: You see bunch of dogs, and you see it as a group of dogs, not some kind of order like dog A, dog B, dog C etc.)
Note: If you have some kind of dependency between tests in same class, running TestNG in parallel will run the groups in parallel, because it can't run the tests in parallel caused by their dependencies.
Answer 2:
If the browser stays open, it's because your code doesn't close it. Maybe you should check how many times you open a new instance, and how many times you close it.
I suspect by your first scenario that you are opening more than one instance of the browser, but you are using the same WebDriver to manage it, not closing the second instance created.
WebDriver doesn't just pop up a new browser instance by itself. You need to create an instance to use it. Similarly, WebDriver doesn't just close a browser instance by itself, you need to close it with your code.
Answer 3:
Look in answer 2.
For future reference: There is no such thing erratic behavior of TestNG or SeleniumWebDriver or event the combination of the two. This is the most common combination of testing tools used by all QA engineers. My company has parallel tests with thread count = 16 and they all work fine.
Keep in mind, test code should be optimized for parallelism, you can't just run regular old tests that were designed for synchronous environment.
This command will run UI test on Simulator for 2 test cases :
xcodebuild test -project MyLib.xcodeproj -scheme MyAppUITests -destination 'platform=iOS Simulator,OS=11.3,name=iPhone 5s' -only-testing:MyAppUITests/MyAppUITests/testExample -only-testing:MyAppUITests/MyAppUITests/testExample2|xcpretty --test --color
This cause testExample and testExample2 to be executed but the app under test is restarted between the 2 test cases executions, despite the fact that test setup is the same for both.
How could I keep the app started just once ?
If you want to do multiple actions in a row without a restart in between, they need to be done as part of a single test method.
The reason that the app is restarted between each test is to ensure that the outcome of each test is independent from the others. Repeating the setup for each test is essential if you want to ensure that when a test fails, it has failed because there is something wrong with that test case.
If you chain many tests together without resetting the state, and one part of the test fails, the other parts of the test will be affected and you will not have clear test results.
While it is true that the setUp() method runs multiple times as part of this framework, this is necessary to ensure test independence. The power of automated UI tests lies in the automation (no manual effort required for repeated execution of the same tests) and the data they can give you (a clear breakdown of which features are or are not working), not in their speed. Restarting the app between each test and running the setUp() for each test should not increase the time complexity of the execution of your test suite.
For example, say you have four components, A, B, C and D, and B breaks. A, C and D are all still working.
If you only have one large test method, testAAndBAndCAndD(), and it fails, you will be unable to tell whether A, B, C or D are broken, or if all of them are broken.
Whereas if you have four separate tests, testA(), testB(), testC() and testD(), only testB() will fail and you will instantly know from the test results that the problem lies with B.
I hope this illustrates why it is necessary to restart the app between each test.
I am setting up a load test with multiple web performance tests.
I want one user to run each test. I can't have multiple users run the same test.
I have 10 tests, and I want them all to be run at the same time (i.e. simulating peak load).
When I run my load test only 1 of the tests gets loaded.
Here's my configuration:
I have 10 web performance tests defined that I've added to a load
test.
I have set the "Test Mix Model" to be based on the "total number of
tests".
I have allocated each test to have 10% distribution.
I set the user count to 10 (constant load).
I'm not sure how to capture the testing result data related to unit tests each time a unit test is run. I use Bamboo as a continuous integration server. It works great, basically makes a build of your project every time you submit code, and send you an email if the build failed / you screwed up somewhere. I would like to begin having Bamboo running full unit tests as well as builds. I would also like to begin gathering data about said unit tests.
My question is, I know in a lot of dif programs you can track the number of lines of code changed, and the total lines of code period in the entire program. I also know that with unit testing, it gives you data such as the number of passes / failures. What I would like to do is automatically gather this data, among other data such as defect density, etc.
I use jasmine to test my server side code and i need to run tests in serial, not in parallel.
My tests need to make CRUD operation in database. If test are executed in parallel i can't ensure that the database is in a good context for my test
Unless you explicitly choose to create asynchronous tests in Jasmine, everything in Jasmine happens sequentially, in the sense that one test runs only after its preceding test has finished. And if you do write asynchronous tests, then parts of your single test may run in parallel, but you still have the constraint that one test runs only after its preceding test has finished.
However, there are a couple caveats to be aware of:
In an async test if your code exceeds Jasmine's timeout period, you might still have code running when Jasmine decides to give up on that test and proceed to the next. (Thanks to #Gregg for this tip; see this answer.)
"JavaScript is usually considered to have a single thread of execution... however, in reality this isn't quite true, in sneaky nasty ways." I am quoting #bobince from this answer.