I'm having trouble wrapping my head around Karma. I'd like to:
Set up multiple hosts on my network, running Linux, Mac and Windows
Preferably also run on Android and iPhone
Have these be available for running end-to-end tests through Karma
Have them run tests on a remote location, not locally
The goal: being able to automate tests which ensures that our site works on all platforms and browsers, not only the ones available to me locally.
Is this possible? I'm struggling to find any good guides for setting this stuff up.
You can start a webdriver server on your remote servers and configure karma to use the karma-webdriver-launcher to run the tests on the browsers from your webdriver servers.
WebDriver : http://code.google.com/p/selenium/wiki/RemoteWebDriverServer
karma-webdriver-launcher : https://github.com/karma-runner/karma-webdriver-launcher
I've been using karma for a short while myself and I think I can answer some of your questions.
I am not sure what you mean with setting up multiple hosts, but I guess you mean that you want to run the tests on several different devices (maybe even on different browsers?).
All you have to do really is to have the tests and karma installed on some server that you can access remotely. Running Karma from that server should make it possible for your other devices to access it's instance of Karma simply by opening a browser and typing in serverURL:9876 in the URL-bar of the browser. This should cause all the tests found on the server to be run on the browser that opened the page.
If you want to see the output from Karma during the tests, you will either have to make karma spit out some HTML using a reporter (if you manage to do this, give me a call!), use the junit reporter and post process the xml that it generates, or simply SSH to the server and see what comes out in the console.
If you use some sort of regex in the karma config file that is able to find any new code and test files you push to the server, karma will automatically load these files when you push them to the server and re-run all tests.
I am actually in the process of doing this myself, but I would like to create HTML test-reports instead of having to post process some XML or having to SSH and look at the command-line output. I am also having some problems with Istanbul, the code coverage tool, in that if you run the tests on several browsers at once, only one of them will have code coverage generated.
Related
I can give the URL in cy.visit to point to deployed application, but I am not clear on the setup, as I don't have CI/CD yet. The goal is to be able to run this test on a button click, not having to checkout workspace and build the application.
I can give the URL in cy.visit to point to deployed application, but I am not clear on the setup, as I don't have CI/CD yet. The goal is to be able to run this test on a button click, not having to checkout workspace and build the application.
Yes, you could test your application directly on a staging/preproduction environment on AWS without A CI/CD.
However, I would not recommend using Cypress against a production environment because for E2E testing to be efficient, you will probably create/delete/edit many things, which may not be suited for a prod env.
Finally, it depends on what you try to achieve with your tests. In general, you will use Cypress to ensure your product works as expected after adding a new feature, for example. You always want to test against the latest version of your code.
On the setup, the E2E tests can be packaged in the project and run manually until you have a CI to execute them automatically.
https://docs.cypress.io/guides/guides/command-line#cypress-run
One of our E2E tests involve checking whether the sample apps for our SDKs work properly. To do that, we would just mimick the end user behaviour (download the sample app zip file, unzip it and run npm start) and the tests pass if the server starts in http://localhost:3000.
Now the problem is that while the test suite is running, we cannot run anything on port 3000, as the sample app instances keep getting started and killed for the E2E tests. This brought the following question into my mind.
Is there any way to configure Cypress to open this sample app servers inside a cypress supported sandbox environment, instead of opening in a new tab of one of my working browser windows?
Thanks in Advance.
Regarding the Cypress Dashboard and runs
We are running smoke and end-to-end tests post-deployment and the tests are run against multiple environments
Is it somehow possible to combine the results for all the environments into a single run in the Dashboard?
Current results as an example:
We run the smoke tests against 8 environments that have different configurations
7 of these environments are ok and are marked as a success
1 environment fails
If the run for the failed environment isn't the latest to be run the test in the Dashboard is green and we sometimes don't notice that something failed through the Dashboard
Technically they are all being run from the same commit id
Is there any command-line parameter that will combine this so that the Dashboard will look at this as the same run but just against different environments similar to how it does with browsers?
I've been going through the documentation and issues on GitHub but cant find anything related to this
Yes, you can construct a unique id and use the --ci-build-id passed to cypress run https://docs.cypress.io/guides/guides/command-line.html#cypress-run-ci-build-id-lt-id-gt
There is an example in the Cypress Real World App, a payment application to demonstrate real-world usage of Cypress testing methods, patterns, and workflows, in the Circle CI configuration. Note, this uses the Circle CI Orb syntax which maps to the --ci-build-id command line parameter.
I'm trying to get theIntern up and running on a project, I've managed to get Webdriver.io running but need to demo theIntern for web automation. But I can't seem to get the set up right.
I'm using Node and NPM
The Intern dependency is outlined in my package.JSON file
I've ran an Npm install - and all is ok there.
I have a test file
And selenium standalone is running but I can't seem to get it to work, any suggestions or steps I'm missing would be great thanks for your help.
Intern includes support for everything you need to do WebDriver tests, including a Selenium manager and its own WebDriver library. (Note that you will need Java installed to run Selenium, but it sounds like you've already taken care of that.)
To run WebDriver tests, you'll want to write functional tests. A functional test will call methods on "remote" (this.remote), which is an object implementing a WebDriver API, and return the result.
To actually run the functional tests, use node_modules/.bin/intern-runner, or intern run -w if you've installed intern-cli. In your Intern config, set the tunnel property to 'SeleniumTunnel' to let Intern download and manage Selenium, or to 'NullTunnel' to use your own running instance of Selenium.
I wrote several Rails 3 integration tests using Watir-Webdriver (0.9.1) that test whether a user can register. These tests run fine and pass. However, I've been running these tests with a rails server running so that Watir can access my site and work its way through the webpages. This presents a problem because that means I am running my tests on the development environment, and because of that I can't get accurate numbers for lines covered by my tests and all the information I input through Watir goes into a different database that my tests do not have access to.
If I try to run my rails server in the test environment, it gives me an error (SQLite3 ReadOnlyException) when I try to register a user. I've been led to believe that this is because SQLite3 does not want to be used by both the test server and my test code at the same time. I do not think it is actually a Read Only database because I have used chmod 777 on it and its folders with no success.
Is there a way to make it so that my integration tests do not actually need a rails server up in order to access my site? Or do I just need to use the development environment for my server and test to make sure the right data is being submitted somewhere else (i.e. a different type of test)?