i'm evaluating Cypress (Version 3.4.1), and running into inconsistencies between running the same tests from the cypress tool and running them from the terminal, i'm using the same browser in both cases (Electron 61). Anyone experienced this? (failing test from the terminal, but same test runs smoothly from the Cypress tool)
The interactive test runner can be flakey, but I do not see the same issues when I run without the interactive test runner. I wouldn't worry about the test failing in the interactive test runner if it passes when you refresh the page, or passes when run via command line. Use the browser refresh button not the interactive test runner refresh button.
Note: cypress is pretty heavy on your resources. I noticed I have intermittent failures when doing a screen share.
While running Cypress rest from command, I found that the test finishes with All specs passed, but yet it didn't complete the full iterations (last one is to save form in db, didn't happen).
I didn't know why? I tried to change the Cypress code but with no result.
I decided to check the recorded video after test finishes from terminal so I enabled video recording and TARAA!! Test finished correctly. Once I disable video recording It fails.
I'm having serious issues with Cypress soaking up all my available memory (16Gb). I have 30+ tests and if I attempt to run them through the UI all at once then Cypress gradually uses all my available memory and then typically fails with a test timeout error. Closing the Cypress UI always recovers all the memory. I've seen https://github.com/cypress-io/cypress/issues/431 which suggests setting numTestsKeptInMemory to 0 but this makes no difference, also running in headless mode with cypress run makes no difference either: Ultimately all my memory get soaked up.
Also during development of the tests I've been using it.only but even when running only one test at a time the memory gradually gets soaked up until restarting Cypress is needed.
I'm using Cypress 1.4.1 on Ubuntu 16.04 (elementaryOS Loki)
Does anyone else have the same trouble?
I'm assuming this is happening while using cypress open?
cypress open is used for TDD, so you can get immediate feedback while you're developing. It's recommended in the docs to NOT run all your tests in the test runner, but with cypress run instead.
You won't get to do the snapshot history navigating, and instead will only get pictures and a video recording, but your tests will run in a headless browser and not soak up your memory.
PS: If you need to fix a broken test and want to use the test runner, you can isolate it using it.only('test case...)
Last couple of days I have been trying to figure out the ways to handle a few issues with Selenium and IE Such as this
Today, I wanted to run all the Selenium tests against IE9 but in the middle of the test, due to a JavaScript error the debugger popped up and all the tests started failing. Just an extra bit of information I tried using
options.UnexpectedAlertBehavior = InternetExplorerUnexpectedAlertBehavior.Dismiss;
which was not able to dismiss that debugger window.
I know I can manually do it by unchecking the debugger as follows. But, this is not a feasible solution since these tests can be executed in a lot of different machines.
Is there a way to do these programatically with DesireCapability or Options? Or at least any commands for commandline?
Is there any possibility to run written CUIT tests without necessity of having active browser window?
Basically, what I would like to have is just to run a bunch of tests on local machine. And I need to wait until all the tests are executed (each test requires browser window to be active during test run). But during this execution I can't use my machine for any other work - otherwise tests fall due to loosing control on the window. So is there any ability to run tests locally and work on the machine without any limitations simultaneously?
Coded UI Test Needs Active Browser while Execution.
To match your necessity you should try running test cases into virtual machine. That will not affect your work in local machine.
You might consider using WatiN and run them by "normal" MSTest/NUnit/whatever instead of using CodedUiTests. It does open up a browser window too, but allows you to interact with your desktop nevertheless.
That of course means rewriting your tests but as these tests are much more readable IMHO this might be worth it.
I am having selenium troubles. I will do my best to explain the setup I have and what I'm trying to do.
The short version:
Running my automated tests locally using a ruby webdriver gem works fine. Running the exact same script through the selenium-server standalone jar (whether remotely or locally) does not work without strange alteration in the code.
Is there a way to get a version of the selenium server standalone jar that behaves the same way as the webdriver client libraries?
Or is there some remote ruby gem version of the selenium server so that however the gemified version of the selenium stuff can work remotely?
Basic summary: "I feel like I am in Hooters and asking for my meal to go. I am missing something very fundamental here."
What I'm trying to do:
I am trying to automate testing a web application. The idea is to be able to load the company's website and then interact with the page elements like a real user would (click a link, put text into a text box, select radio buttons and checkboxes, etc). I developed these tests using ruby 1.9.3 and the selenium-webdriver gem for ruby on a mac (10.8.2). I created a ruby wrapper library myself, which I have called "WebAutomation.rb" in which I created my own methods in order to click on elements. An example of one of the wrapper methods I made is as follows:
def WebAutomation.click_element_by_attribute(attribute_name, attribute_value, tag, contains=true)
element = WebAutomation.find_element_by_attribute(attribute_name, attribute_value, tag, contains)
#log.debug("Element returned: #{element}")
raise "Could not click element with #{attribute_name} attribute of #{attribute_value}" unless element.click
end
And the WebAutomation.find_element_by_attribute calls another methods that looks through all the elements that I give it. As another layer of abstraction, I am not running the ruby code directly, I am running it through cucumber scripts That's not my problem. This code all works locally - by that I mean when the browser being driven automatically is local to the code being run.
However, I want to be all fancy-like and not have to run the code locally because I'm on a mac, and let's say I want to do cross-browser testing like run it on IE. So I have both a remote mac and a remote windows laptop with the goal of running this through some system like Jenkins where the Jenkins box would be able to tell these remote machines to run browser tests. I'm not that far yet where I'm worried about Jenkins. I'm just trying to get the remote versions of the tests to pass.
My troubles:
Running my cucumber/ruby scripts locally work great. They're awesome, and I thought I was the man. Then I ran it to the remote mac using the same browser (chrome), and everything went to pot.
Here's what I'm doing:
On the remote mac laptop, I downloaded the selenium-server standalone jar and started it like so:
java -jar selenium-server-standalone-2.31.0.jar
It looks happy to me:
Mar 14, 2013 8:00:06 AM org.openqa.grid.selenium.GridLauncher main
INFO: Launching a standalone server
08:00:11.606 INFO - Java: Oracle Corporation 23.6-b04
08:00:11.608 INFO - OS: Mac OS X 10.8.2 x86_64
08:00:11.616 INFO - v2.31.0, with Core v2.31.0. Built from revision 1bd294d
08:00:11.728 INFO - RemoteWebDriver instances should connect to: http://127.0.0.1:4444/wd/hub
08:00:11.729 INFO - Version Jetty/5.1.x
08:00:11.730 INFO - Started HttpContext[/selenium-server/driver,/selenium-server/driver]
08:00:11.730 INFO - Started HttpContext[/selenium-server,/selenium-server]
08:00:11.731 INFO - Started HttpContext[/,/]
08:00:11.744 INFO - Started org.openqa.jetty.jetty.servlet.ServletHandler#4f8429d6
08:00:11.744 INFO - Started HttpContext[/wd,/wd]
08:00:11.747 INFO - Started SocketListener on 0.0.0.0:4444
08:00:11.747 INFO - Started org.openqa.jetty.jetty.Server#4dfbca86
So then I run my cucumber scripts, passing it some command line arguments that tell it to go to the IP of the remote mac laptop and what browser to run and what environment I want the browser to go to for our application (this isn't important).
It looks like this:
cucumber REMOTE_URL=http://10.110.10.233:4444/wd/hub BROWSER=chrome JJ_ENV=staging features/jabberjaw/contact_us.feature:3
I have code that when a REMOTE_URL parameter is passed that a remote browser is called instead of a local one. The code that gets executed is below:
#This is the code that runs for a local browser
def WebAutomation.set_browser(browser)
#log.debug("Starting browser: #{browser}")
#driver = Selenium::WebDriver.for(browser)
end
#If the remote url is passed in, then I make a remote browser
def WebAutomation.set_remote_browser(url, browser)
#log.debug("Starting remote browser: #{browser} at #{url}")
#driver = Selenium::WebDriver.for(:remote, :url => url, :desired_capabilities => browser)
end
The browser window pops up on the remote machine and it goes to the correct url and logs in. However when running remotely, the selenium-server seems to have issues in clicking elements that are not visibly on the screen. I fixed that (sort of) with a
#driver.action.move_to(element, 100, 100).perform
I had to add the 100, 100 offset because even moving seem to only get to the top left corner and the element still wasn't on the screen. The other thing is that you know that exception I raise unless element.click (code above)? Yeah, that triggers regardless of whether the click really happened or not because for some reason the remote version (selenium-server) returns nil on the element.click whether it succeeds or not. When I run the same code locally (where it uses the webdriver gem), it gives me an {} when successful and nil when it's not. So to get this code to work remotely on chrome I had to do the following:
def WebAutomation.click_element_by_attribute(attribute_name, attribute_value, tag, contains=true)
element = WebAutomation.find_element_by_attribute(attribute_name, attribute_value, tag, contains)
#driver.action.move_to(element, 100, 100).perform
#log.debug("Element returned: #{element}")
element.click
end
In essence, explicitly moving to the element found, and just trusting the click works. Yes, with cucumber I do have a thin layer of protection in that the next step in the script should be a Then step that checks whether whatever action the click was supposed to do succeeded, but it feels wrong that I have to take out that exception and potentially open myself up to false positives.
And even this altered code fails completely on a remote version of firefox (I had to move down to firefox 18, since anything above 18 seems to not work with selenium webdriver - even locally - it just opens the browser window and does nothing else). On firefox, the browser window comes up, navigates to the url, logs in (up to this point, it's like chrome), but then it just gives me the finger and says "MoveElementTargetOutOfBoundsException".
I have also tried taking out the "remoteness" and tried running the selenium-server jar locally and running my tests on a local browser but through the selenium-server jar. Like a so!
cucumber REMOTE_URL=http://localhost:4444/wd/hub BROWSER=chrome JJ_ENV=staging features/jabberjaw/contact_us.feature:3
And I get the same results, so I'm pretty convinced that my problem is that the selenium-server jar is interpreting my scripts utterly differently than my selenium webdriver gem. I cannot be the first person to have run into this, but I have googled until my eyes bled and cannot find a solution to this.
There has to be some way that the client code that runs locally is interpreted the same way remotely, yeah? This can't be a new problem, because if I have to create weird custom code for whether I am running locally or whether it's remote and whether it's chrome and whether it's firefox then this whole "automation is robust and awesome because you can do cross-browser testing and scale across environments" is some mayo-mustard packed cheesecake. That creme filling? Do not want.
So it looks like I was able to finagle my way to a partial solution.
Apparently just starting the selenium server stand alone jar was not enough. The following got me a little farther.
On the server system (the remote system that will be opening and interacting with the actual browser):
java -jar selenium-server-standalone-2.31.0.jar -role hub
Then in a new terminal window (still server system):
java -jar selenium-server-standalone-2.31.0.jar -role node http://localhost:4444/grid/register
Then on the client system (the one with the actual cucumber and webdriver scripts)
cucumber REMOTE_URL=http://10.110.10.233:4444/wd/hub BROWSER=firefox JJ_ENV=staging
**Note, if you're following along at home, this command will be different since I coded my scripts to accept the REMOTE_URL and BROWSER variables to map to a remote webdriver call.
In any case, this allowed the original code that didn't have the explicit move method to work. I still had to eliminate my raise condition because clicks are still giving me nil whether successful or not when going remotely. Firefox also no longer throws up the MoveElementTargetOutOfBoundsException.
The tests still seem much more fragile than when run locally, but it's a least some progress. If anyone has any information about why my clicks always give me "nil" when run remotely whether successful or not, I'd appreciate it. If anyone also has any other information about why adding these role paramemters (and perhaps the registration) seems to stabilize things compared to just the regular java -jar seleneium-server-standalone-2.31.0.jar command that almost all the tutorials I read told me to do, I'd be interested in this.
However, like a big bowl of fiber, these commands have at least gotten me unblocked. Nothing worse than major blockage. I hope if anyone else is having similar troubles that this helps you as well.
I have gotten myself unblocked on windows remote testing as well. Apparently for that, I needed to get the chromedriver.exe and ieserverdriver.exe and put those on the server system. On windows, you still need to start the hub, but when starting the node, you'll need to add the following parameters:
-Dwebdriver.chrome.driver=<path_to_chromedriver.exe> -Dwebdriver.ie.driver=<path_to_ieserverdriver.exe>
And then from my client machine, I had to use port 5555 instead of 4444.