xUnit documentation does not specify any limits on the number of parallel tests - https://xunit.github.io/docs/running-tests-in-parallel.html
See the "Changing Default Behavior" section on that page.
Default: number of virtual CPUs in the PC
Related
I have a lot of tests based on the Google Test framework.
All tests were built in Release mode, and when I manually run the separate test under the "Visual Studio" on "Diagnostic Tool Window" (Ctrl+Alt+F2) in the "Process Memory" chart I can see the maximum heap memory consumption.
Is there exist way, to log memory consumption automatically for each running test?
All I need to know is the maximum amount of memory that each test used during the execution.
Recently I run the load test of 5k users for 30 minutes using concurrency thread group. When I want to read file in the Graph Report listener, this out of memory error prompted on the cmd:
I tried to modify the heap size to 6GB in notepad++ (my laptop spec: 16GB DDR4, 512GB SSD, i5 7th gen, available ram 10GB):
But in the jmeter cmd screen, it does not shows the changes i made just now. how to exactly change the heap size? I tried to follow the steps already using notepad++ but seems not working for me. Anything that I missed out here? I followed the steps from 9 Easy Solutions for a JMeter Load Test “Out of Memory” Failure by Blazemeter
The easiest is setting HEAP environment variable like:
set HEAP="-Xms1g -Xmx1g -XX:MaxMetaspaceSize=256m" && jmeter -n -t test.jmx
In order to make the change permanent modify the line # 151 of jmeter.bat startup script
Don't pay attention to the console message when you start JMeter in GUI mode, it's hard-coded and doesn't pick up any changes
If you're following recommendation to increase the heap why don't you follow the recommendation to run the test in non-GUI mode?
More information: 9 Easy Solutions for a JMeter Load Test “Out of Memory” Failure
We have a JMeter test which stops in between giving the following error message
It runs fine when I run the tests with a load of 100,200 and 300 Users
I found one question jmeter test failed with out of memory error but that didn't help to solve the issue.
Have tried running this on machines with 12 GB RAM and also with a 16 GB RAM.
What could be the possible reason for it to behave so?
also tried to run on non-GUI mode even there the test stops.
Don't run your test in GUI mode, as per first line of your terminal output you should be using non-GUI mode for running your tests. GUI mode is for tests development or debugging only.
Make sure to use 64-bit Java JDK or Server JRE, you need to choose Windows x64 option
Increase your JVM Heap size by amending HEAP environment variable like:
set HEAP=4G && jmeter.bat -n -t test.jmx -l result.jtl
Make sure to follow JMeter Best Practices and recommendations from 9 Easy Solutions for a JMeter Load Test “Out of Memory” Failure article
I had to add the following piece of code in jmeter.bat ( Batch file )
set JVM_ARGS=-Xms512m -Xmx8000m
I added this in here
if not defined HEAP (
rem See the unix startup file for the rationale of the following parameters,
rem including some tuning recommendations
set HEAP=-Xms1g -Xmx1g -XX:MaxMetaspaceSize=8000m
)
set JVM_ARGS=-Xms512m -Xmx8000m
then it worked just fine.
As Dimitri's answer mentions, don't use the GUI, it should only be used for testing or debugging. But this doesn't seem to be the issue for the OP.
In my case I got the issue 2 different times, with 2 different solutions:
I had been using the GUI for testing for a quiet a while and also had some of its output files opened, which were around a GB in total. Closing the GUI and those files gave me back enough memory to run via command line.
I increased the Heap size to 4GB and got error The paging file is too small for this operation to complete. So I upped the system's page file size and was able to run Jmeter. See this link on how to up the page size of the system.
Running same Cucumber Features on different machines concurrently using parallel_tests gem
I'm trying to figure out how to use the parallel_tests gem to run the EXACT SAME Cucumber Features on a variety of different machines concurrently.
Currently I have them running on different machines but the SAME features do not execute, it just splits up the features and runs different features on different machines.
More clarification on what I want to do:
MACHINE 1 (Win 7) - EXECUTE SAME FEATURES CONCURRENTLY
MACHINE 2 (Win 8) - EXECUTE SAME FEATURES CONCURRENTLY
MACHINE 3 (Mac OS X) - EXECUTE SAME FEATURES CONCURRENTLY
...
Also my architecture is:
Cucumber -> Capybara -> Selenium Webdriver -> Selenium Grid
parallel_tests gem: https://github.com/grosser/parallel_tests
The parallel_tests gem is really purposed to parallellism, not executing multiple platforms.
You may find the following useful:
Sauce Gem 30 rake tasks
"The Sauce integration is currently targeted at RSpec and Cucumber, running on a local server you can spin up multiple copies of. It runs a copy of each test for each platform, and divides them up across all the concurrency available to your Sauce Labs account by default"
Hope it helps, I haven't used it myself yet.
Thanks for the clarification. To accomplish testing your app for all the supported environments (OSs/browsers) you can use something as Selenium Grid here is an example Selenium RC: Run tests in multiple browsers automatically.
TestingBot is a service that use this: http://blog.testingbot.com/2012/02/19/selenium-cucumber-capybara
Other approach is described here: http://altoros.github.io/2013/running-capybara-tests-in-remote-browsers/
http://paauspaani.blogspot.mx/2013/05/running-tests-remotely-using-selenium.html
There is also a presentation: http://www.slideshare.net/martinkleppmann/crossbrowser-testing-in-the-real-world
Also this could help you Has anyone figured out a way to run the same cucumber scenario on multiple browsers/web drivers?
Hopefully this will give you some clues.
I am confused by this statement of yours:
Currently I have them running on different machines but the SAME
features do not execute, it just splits up the features and runs
different features on different machines.
parallel_tests is used for splitting up tests and running them on multiple cores, not on a distributed environment of multiple machines. It can group tests to be run on a distributed environment using the --only-group flag, but it doesn't actually run tests on a distributed environment. So you have to do some lifting yourself to get the tests running on a distributed environment. You could run all of your tests on multiple machines like so:
Use net-ssh to ssh into each one of your boxes.
Use each ssh session to run your parallel_tests script (which should break the tests up and run them in parallel on the current box -- just don't use the --only-group option as described here)
I've followed the instructions on http://blogs.msdn.com/b/vstsqualitytools/archive/2009/12/01/executing-unit-tests-in-parallel-on-a-multi-cpu-core-machine.aspx to the letter and I can execute tests in parallel.
So far so good.
Now the problem: I can't set the parallelTestCount to zero (Auto configure) or to a value above 5. Running five parallel tests works fine but not six.
Yes I have more than five cores on my machine; I am running on an Intel i7 processor which has four cores with hyperthreading meaning I should be able to run 8 tests in parallel.
Now I'm sure there are some other logical or hard limitations and I am not expecting to be able to run on eight cores lightning-speeding eight tests through in parallel.
However it would be cool to know the reason for it not running more than five tests in parallel?
It seems that this is a bug in MSTest. I have stumbled upon the same problem, all tests are marked as aborted and there is a "Test run error" link in the Test Results window which if clicked yields this message: "Test run is aborting on '', number of hung tests exceeds maximum allowable '5'".
Update: It is fixed in the Visual Studio 2010 Service Pack.
It could just be me but hyperthreading isn't a real core so an i7 only really has 4 physical cores not 8. Maybe it is something to do with that.
Or perhaps there is some bios setting relating to how the hyperthreading can be used by the OS that is preventing it.