I have a set of Watin GUI tests that run on Teamcity. They each run in about 300 seconds locally. They previously ran in about the same time on a TeamCity agent (PC specs are about the same). For some reason they now run incredibly slow. The tests have a timeout of 1500 seconds and this is being triggered to cause them to fail.
I have checked the tests again locally and they run in the same time as expected (about 300 seconds). When I remote view into the Team City agent I can see that the tests are keying in text at a very slow pace. The actual speed at which tests operate is slower. It almost seems as if the tests are set to run at a quarter speed if such a thing existed.
These tests are only run about once every two weeks so its possible something on the agent has changed to effect their functionality.
I am at a loss as to what could be wrong. I've never encountered this before and on paper everything is correct but it is just going slow.
Does anyone have any ideas?
Related
I'm having serious issues with Cypress soaking up all my available memory (16Gb). I have 30+ tests and if I attempt to run them through the UI all at once then Cypress gradually uses all my available memory and then typically fails with a test timeout error. Closing the Cypress UI always recovers all the memory. I've seen https://github.com/cypress-io/cypress/issues/431 which suggests setting numTestsKeptInMemory to 0 but this makes no difference, also running in headless mode with cypress run makes no difference either: Ultimately all my memory get soaked up.
Also during development of the tests I've been using it.only but even when running only one test at a time the memory gradually gets soaked up until restarting Cypress is needed.
I'm using Cypress 1.4.1 on Ubuntu 16.04 (elementaryOS Loki)
Does anyone else have the same trouble?
I'm assuming this is happening while using cypress open?
cypress open is used for TDD, so you can get immediate feedback while you're developing. It's recommended in the docs to NOT run all your tests in the test runner, but with cypress run instead.
You won't get to do the snapshot history navigating, and instead will only get pictures and a video recording, but your tests will run in a headless browser and not soak up your memory.
PS: If you need to fix a broken test and want to use the test runner, you can isolate it using it.only('test case...)
I'm trying, but failing to setup a reliable continuous integration environment using Xcode server.
I have a git repository on a headless mac mini server running the Xcode server service, the server has a separate development user account with administrator privileges that is used by Xcode.
I have setup my schemes, with testing included and shared them to the repository.
The bots run, check out code, build, analyze and archive, but only seems to run tests when it feels like it, which is almost never. I've checked the schemes and they have not changed since Xcode ran the tests and when it didn't.
On first setting them up, tests wouldn't run at all, until I added administrator privileges to the development account, then the tests ran a couple of times, before Xcode server decided to stop running them again.
I don't seem to get any reason why the tests aren't run, sometimes the bots fail to run because of some crash during the setup, and an error is reported, but mostly the bot seems to run, they just don't execute the tests, and no error is reported.
I've logged in remotely to the server, and the simulator is running, but never seems to do anything.
Here's a screenshot of an example bot, you can see the tests used to run, it sees I've reduced my warnings and got rid of an analysis issue. You can also see where no tests run, and no kind of warning or error is given as to why.
I've tried restarting the server, nope.
I've tried restarting the client, nope.
It's really frustrating and can't find any recent issues that offer a proper solution to this. The server is in constant use running backups and other tasks, so I'd rather not have a solution that involves me logging in to the server and restarting something every time there's a problem, which is always, it makes the whole point of bots useless if I'm spending more time logging in to my server trying to get them to work than they are at actually running.
Anyone have similar issues and a solution?
Edit: Noticed that my memory usage was very high on the server, memory pressure was practically always amber, so went out and got some memory today, increased the mac mini's memory from 4GB to 16GB, and now the tests have started running again. Also, the whole process is much faster (less than surprising i guess).
Could it just be low memory causing problems with the simulator? I've only just installed the memory and restarted, so I'll give it a few test runs before I confirm this solution, it's stopped working before...
Seems that this may be a memory issue, I upgraded the servers memory from 4GB to 16GB as my Activity monitor was showing significant memory pressure.
Since doing this, the bots started running tests again, and the total running time for the bot is a quarter that it was.
As per my edit, I've been running the bots for a day now, including bots that run on multiple simulators, and everything seems to be fine.
It's not very good that no obvious indication is given in xcode as to why the tests didn't run.
For reference and to see if this might fix your problems, original server specs were :
Mac Mini Server edition (late 2012)
2.3 GHz Intel Core i7
4GB memory
2x1TB drives
Replaced the 2x2GB memory sticks with 2x8GB sticks (The maximum allowed for the model)
EDIT : After a month of running with no problems, increasing the memory has solved the problem permanently.
I've been having a persistent reliability issue when running Cucumber\Watir-Webdriver (Ruby) tests on a TeamCity build agent.
When running the 96-step suite against the CI web app server from a local machine, all tests run green, but running the same tests against the same environment using a Build Agent produces a rash of intermittent errors such as:
Selenium::WebDriver::Error::NoSuchElementError: Unable to locate element:
Using the WW and PageObject waiting methods just produces intermittent timeouts such as:
Watir::Wait::TimeoutError: timed out after 30 seconds
It's incredibly hard to debug this as it's an intermittent problem with a lead time of about 20 minutes for feedback. In order to improve it, we've upped the RAM on the build agent's host and increased the heap memory available to the Build Agent's process to 4G using the TEAMCITY_SERVER_MEM_OPTS environment variable. This doesn't seem to have helped.
Is it running at the same resolution?
My team ran into an issue where the build server was running at a different (smaller) resolution than our machines, so it would fail to find an element due to the smaller resolution. The element would be outside the browser window.
We solved this by setting the browser's size to a specific value every time the driver is created so if it would fail on the build machine it would fail on ours as well.
I have some tests in nUnit that call an external program that sometimes takes more than 10 seconds to finish. This works fine when I run them in nUnit's "Unit Test Sessions" panel. However, when I use the test adapter (which has the benefits of more through error output and automatic test discovery), I get the following error:
The request has taken more than 10 seconds to respond, aborting it.
Exception has been thrown by the target of an invocation.
Is there any way I can extend this time limit for my tests? Is this an issue with the adapter, or with Visual Studio itself?
Edit: To clarify, I know that mocking out time-consuming functionality is the proper thing to do. Right now, though, it's not worth the time (in my estimation) that it would take to refactor my unit tests.
It has nothing to do with VS, as the Resharper is able to run tests well exceeding 10 seconds. This is most likely a limitation of the adapter.
This issue (https://github.com/nunit/nunit-vs-adapter/issues/24) is fixed in version 1.2 of the adapter.
We are considering a move to jruby, and as part of this have been researching testing approaches and frameworks. I've been testing rspec on my local development machine and am finding that the total time taken to run 2 tests for a single class with 2 very simple methods is 7-8 seconds. By simple, I mean one setter and one that returns true.
The rspec output shows that the tests run in roughly 2 seconds, so 5-6 seconds of the total time is spent loading and initializing rspec. I'm running from the command line using
C:\rubycode\rspec_tutorial>rspec --profile user_spec.rb
..
Top 2 slowest examples:
User User should NOT be in any roles not assigned to it
0.023 seconds ./user_spec.rb:15
User User should be in any roles assigned to it
0.006 seconds ./user_spec.rb:10
Finished in 2 seconds
2 examples, 0 failures
I'm running jruby 1.6.5 and rspec 2.7.1
I've read this post, Faster RSpec with JRuby
but it's over 1.5 years old, and the answer relates to running suites of tests as opposed to short bursts of a small number of tests locally to aid TDD, which is how we want to develop. Down the line we'll incorporate a CI server.
My question is, is this the expected execution time? Is there any way to speed up the running of rspec tests on the local development machine uner JRUBY?
EDIT:
Biggest performance gains were switching from 64 bit "server" JVM to 32 bit "client" mode. I saw about a 40% reduction in the time taken to run a simple test. I got Nailgun up and running as well, but performance varied. The link provided below by banzaiman was most helpful
It is not loading RSpec, but it is JVM startup time you are feeling.
See https://github.com/jruby/jruby/wiki/Improving-startup-time for more information.