Every time a Protractor element locator fails, it prints an error and continues down a horrible path of endless cascading failures in my spec and suite. Every test that follows depends on the element locator finding its element, and depends on the current spec passing.
I would like to keep the web page under test open while I use the console. The goal is to debug the current state of the page and investigate why the element locator may have failed to find its target.
I'm not too concerned about failing the entire suite and exiting on the first spec failure (I've seen other answers on --fail-fast and stopping on first spec failure.) This is not the approach I would like to take. I want to set a breakpoint, and inspect the environment while the page is running.
Maybe there's something like a Jasmine option for doThisOnFailure: () => { debugger }, which would work for me I think.
I really do not like the solution of using a spec reporter to execute during afterEach and check the failed spec count on the Jasmine environment for the entire spec function. I want to immediately know when an element locator has failed and immediately break as soon as it has failed.
Maybe something really gross would work $('element').click().catch(() => { debugger }).
EDIT: Please, note that I am asking about breaking in a spec, not breaking at the end of the spec.
it('should execute deadly code', function () {
p.navigation.openStorageConfigTab()
$$('.bad-selector').get(0).click() /* IMPORTANT: I want to break here */
p.volume.navigateTo()
})
it('should not execute this spec', function () {
$$('.bad-selector').get(0).click()
})
And the output
✗ should execute deadly code
- Failed: Index out of bound. Trying to access element at index: 0, but there are only 0 elements that match locator By(css selector, .bad-selector)
✗ should not execute this spec
- Failed: Index out of bound. Trying to access element at index: 0, but there are only 0 elements that match locator By(css selector, .bad-selector)
I can recommend you the approach I use, and I hope you can take it from here
Overall approach is to wait until until you type close/ command in browser url:
await browser.waitForAngularEnabled(false);
await browser.wait(
async () => {
let url = await browser.getCurrentUrl();
return url.includes('close/');
},
5 * 60 * 1000,
'Keep-alive timeout reached, closing the session...'
);
The question is when you want to call it. I use the advantage of onComplete callback function in config file. When it's called, the browser is still available. So once all tests are completed, it doesn't exit for 5 minutes unless I submit close/ to the url field. Obviously that can be conditional, by adding something like if (DEBUG === true)
A downside of this setup is it's called when all tests are completed, and it's possible your spec has navigated away from the page where there was error. So what you can also do is to use advantage of jasmine reporter (if you use jasmine). Roughly, you just need to add this to your onPrepare func:
jasmine.getEnv().addReporter({
jasmineStarted: function(suiteInfo) {},
suiteStarted: function(result) {},
specStarted: function(result) {},
specDone: async function(spec) {
if (spec.status === 'failed') {
await browser.waitForAngularEnabled(false);
await browser.wait(
async () => {
let url = await browser.getCurrentUrl();
return url.includes('close/');
},
5 * 60 * 1000,
'Keep-alive timeout reached, closing the session...'
);
await browser.close();
process.exit(35);
}
},
suiteDone: function(result) {},
jasmineDone: function(result) {},
});
So if any it block has failed status, then it'll stop. BUT, I have not tested it, I'll leave it up to you. And second, I didn't think about what will happen to the rest of queued specs since you're redirected to non existing url close/, but I believe it'll still work for you. Worst case scenario, you can play around and make it continue or close the browser instance, as long as you understood the concept
P.S.
I modified the code to close the browser when you type close/, by adding
await browser.close();
process.exit(35);
I tested this code with the following scenarios:
happy path: all 5 it are successful
first element finder of second it block fails
second element finder of second it block fails
All passed. The code works as expected
Related
I would like to test my "waiting for server reply" UI behavior. How do I do that reliably without pausing my test for a hardcoded delay?
Say, I have a button that triggers an http request and has to show certain animation / actions until the response arrives.
A dumb working method is:
cy.route({
delay: 1000,
response: "blah blah",
})
// triggger submission
cy.get('#my-submit-button').click()
// will disappear upon receiving response
cy.contains('Waiting for response...')
I am pretty much sure that the "waiting" text will appear within a second while the response paused, but then I commit the sin of pausing the test for a whole second.
If I start to shorten or remove the delay then I am running a risk of creating flaky tests since there's a chance the response would be processed before I check for the existence of the "Waiting..." text, which would've been removed by that moment.
Is there a way to ensure the response is produced only after the check for the "Waiting..." text without a hard delay?
I naïvely tried to do a cypress assertion from the route's onResponse, but cypress wasn't happy about that:
cy.route({
onResponse: xfr => {
cy.contains('Waiting for response...')
return xfr
},
response: "blah blah",
})
cy.get('#my-submit-button').click()
producing the https://on.cypress.io/returning-promise-and-commands-in-another-command error:
Error: Uncaught CypressError: Cypress detected that you returned a promise from a command while also invoking one or more cy commands in that promise.
The command that returned the promise was:
> cy.click()
The cy command you invoked inside the promise was:
> cy.contains()
Because Cypress commands are already promise-like, you don't need to wrap them or return your own promise.
Cypress will resolve your command with whatever the final Cypress command yields.
The reason this is an error instead of a warning is because Cypress internally queues commands serially whereas Promises execute as soon as they are invoked. Attempting to reconcile this would prevent Cypress from ever resolving.
Could this work?
cy.route({
delay: 1000,
response: "blah blah",
}).as('getResponse')
cy.wait('getResponse')
https://docs.cypress.io/guides/guides/network-requests.html#Waiting
You could record the time these things happen using cy.moment(), then compare it afterwards using .isBefore() or one of the other moment functions.
I'm using onResponse here to record the time the response comes in.
let timeDisappeared;
let timeResponded;
cy.route({
delay: 1000,
response: "blah blah",
onResponse: () => {
timeResponded = Cypress.moment()
}
})
// triggger submission
cy.get('#my-submit-button').click()
// waits for the text to appear
cy.contains('Waiting for response...').should('exist')
// waits for the text to disappear
cy.contains('Waiting for response...').should('not.exist').then(() => {
timeDisappeared = Cypress.moment()
})
expect(timeResponded.isBefore(timeDisappeared))
After saw your comment and check the other answers
I think Bredan's answer is the best solution to do this. You can
stub a response with delay: x seconds
check the text to appear. Record the time: timeAppear
check the text to disappear. Record the time: timeDisappear
Check if timeDisappear - timeAppear > x seconds
or Check timeDisappear - timeAppear - x seconds > certain seconds(You can define the tolerance)
This proves the text shows in the given response time.
You can extend the response Delay to longer value.
I modified Bredan's answer a bit to reflect the steps above,
let timeAppear;
let timeDisappear;
//delay the response to come back after 20 seconds
cy.route({
delay: 20000,
response: "blah blah"
})
// triggger submission
cy.get('#my-submit-button').click()
// waits for the text to appear and record the time starts.
cy.contains('Waiting for response...').should('exist').then(() => {
timeAppear = Cypress.moment()
})
// waits for the text to disappear and record the time ends
// add timeouts option here to make sure it is longer than delay
cy.contains('Waiting for response...',{timeout:30000}).should('not.exist').then(() => {
timeDisappear = Cypress.moment();
//check the time is above the delay time 2000 ms
expect(timeDisappear - timeAppear).to.be.above(20000);
//or check the tolerance value e.g.5000 ms
expect(timeDisappear - timeAppear - 20000).to.be.below(5000);
})
Here is my custom command:
exports.command = function (element, time, debug) {
let waitTime = time || 10000
if (debug) {
return this
.log('waiting ' + waitTime + 'ms for: ' + element)
.waitForElementVisible(element, waitTime)
}
return this
.waitForElementVisible(element, waitTime)
}
I have also set this variable in the globalModules: abortOnFailure: true.
When I call this in a pageObject though like this:
findElement() {
this.waitFor('#driversLicenseNumbers');
return this
}
The object isn't found (which is expected and intended since I'm upgrading to Nightwatch v1.0.14) and the error message is logged to the console, but the test doesn't fail.
× Timed out while waiting for element <#driversLicenseNumbers> to be
present for 10000 milliseconds. - expected "visible" but got: "not
found"
Does anyone know what I'm doing wrong here?
There is already an open issue on the Nightwatch issues board regarding this specific problem. Here it is!
This behavior affects custom_commands in nightwatch#1.0.15 & nightwatch#0.9.21 (according to the BUG report, yet I am running nightwatch#0.9.21 & this behavior is not reproducible for me).
Basically your test fails, but it does so silently, at the end of the test, where you get the timeout error.
Proposed fix: Install a different version (npm install --save-dev nightwatch#0.9.x), or a suitable version that hasn't introduced the defect yet.
Cheers!
I'd like to implement websocket reconnect in webapp if internet connection is lost. In order to detect that internet is lost I use ping-pong approach, which means that I send from client ping-message and server returns me pong-message.
When webapp loaded I send init ping message and start to listen a reply on socket some kind of this:
this.websocket.onmessage = (evt) => {
try {
const websocketPayload: any = JSON.parse(evt.data);
if (websocketPayload.pong !== undefined && websocketPayload.pong == 1) {
this.pingPong$.next('pong');
}
It means that internet connection looks ok and we can continue. Also I have the follow code:
Observable.race(
Observable.of('timeout').delay(5000).repeat(),
this.pingPong$
).subscribe((data) => {
console.log("[ping-pong]:", data);
if (data == 'pong') {
Observable.interval(5000).take(1).subscribe(() => {
console.log("[ping-pong]:sending ping")
this.send({ping:1})
});
} else if (data == 'timeout'){
// show reconnect screen and start reconnect
console.error("It looks like websocket connection lost");
}
});
But!
When this.pingPong$ subject stops to emit events - .next() doesn't happen because of we can't get response when I break connection manually - I considered that in Observable.race this observable will be emitted
Observable.of('timeout').delay(5000).repeat()
But my subscribe never happens if this.pingPong$ stop emitting.
Why ?
Thank you
race picks and keeps subscribed to the first Observable that emits.
So if your this.pingPong$ starts emitting and then stops it makes no difference because race keeps subscribed to this.pingPong$. The other Observables don't matter any more. You might want emit one value from this.pingPong$ and the repeat the whole process. For example like the following:
Observable.race(
Observable.of('timeout').delay(5000).repeat(),
this.pingPong$
)
.pipe(
take(1), // complete the chain immediately
repeat() // resubscribe after take(1) completes the chain
)
.subscribe(...);
Obviously it mostly depends on what you want to do but I hope you get the point.
The following mocha it statement:
it('should do truthy and falsey things', function() {
var val = true;
assert.equal(val, true, "true should be true!");
console.log('state 1:', val);
val != val;
assert.equal(true, false, "should continue running even after failing");
console.log('state 2:', val);
val != val;
assert.equal(false, false, "false should be false!");
});
results in the following log:
state 1: true
1) should do truthy and falsey things
And that's it. Once the second assert fails, the rest of the function doesn't run.
Is it possible to have the rest of the function (in this case, the last three lines), and if so, how?
The most direct way to prevent an assertion failure from ending the test right there would be to catch the exception thrown by the failure and rethrow it later. However, I do not recommend doing this because this creates complications. If more than one test fails, which exception are you going to rethrow? Also, you have to remember to rethrow the exception you caught, otherwise the test will seem to be passing.
Generally I just live with the fact that if I have a test with multiple assertions then the the test will stop on the first failure. In cases where I cannot live with it, I've generally adopted a strategy of recording checks into a structure and then comparing the structure to an expected value at the very end of the test. There are many ways a structure can be constructed. Here's an example:
it('should allow multiple tests', function () {
var expected = {
"something should be true": true,
"something should be false": false,
"forgot this one": 1
};
var actual = {};
var thing_to_test = true;
// First test
actual["something should be true"] = thing_to_test;
// Second test
actual["something should be false"] = thing_to_test;
// We forget the third test.
assert.equal(actual, expected);
});
When I run the test above, I get the following output:
1) should allow multiple tests
0 passing (12ms)
1 failing
1) should allow multiple tests:
AssertionError: { 'something should be true': true,
'something should be false': true } == { 'something should be true': true,
'something should be false': false,
'forgot this one': 1 }
+ expected - actual
{
- "something should be false": true
+ "forgot this one": 1
+ "something should be false": false
"something should be true": true
}
at Context.<anonymous> (test.js:22:12)
Does anyone have any idea if this is possible? Most of the sample for node-inspector seemed geared toward debugging an invoked webpage. I'd like to be able to debug jasmine-node tests though.
In short, just debug jasmine-node:
node --debug-brk node_modules/jasmine-node/lib/jasmine-node/cli.js spec/my_spec.js
If you look at the source of the jasmine-node script, it just invokes cli.js, and I found I could debug that script just fine.
I wanted to use node-inspector to debug a CoffeeScript test. Just adding the --coffee switch worked nicely, e.g.
node --debug-brk node_modules/jasmine-node/lib/jasmine-node/cli.js --coffee spec/my_spec.coffee
I ended up writing a little util called toggle:
require('tty').setRawMode(true);
var stdin = process.openStdin();
exports.toggle = function(fireThis)
{
if (process.argv.indexOf("debug")!=-1)
{
console.log("debug flag found, press any key to start or rerun. Press 'ctrl-c' to cancel out!");
stdin.on('keypress', function (chunk, key) {
if (key.name == 'c' && key.ctrl == true)
{
process.exit();
}
fireThis();
});
}
else
{
console.log("Running, press any key to rerun or ctrl-c to exit.");
fireThis();
stdin.on('keypress', function (chunk, key) {
if (key.name == 'c' && key.ctrl == true)
{
process.exit();
}
fireThis();
});
}
}
You can drop it into your unit tests like:
var toggle = require('./toggle');
toggle.toggle(function(){
var vows = require('vows'),
assert = require('assert');
vows.describe('Redis Mass Data Storage').addBatch({
....
And then run your tests like: node --debug myfile.js debug. If you run debug toggle will wait until you anything but ctrl-c. Ctrl-c exits. You can also rerun, which is nice.
w0000t.
My uneducated guess is that you'd need to patch jasmine, I believe it spawns a new node process or something when running tests, and these new processes would need to be debug-enabled.
I had a similar desire and managed to get expressso working using Eclipse as a debugger:
http://groups.google.com/group/nodejs/browse_thread/thread/af35b025eb801f43
…but I realised: if I needed to step through my code to understand it, I probably need to refactor the code (probably to be more testable), or break my tests up into smaller units.
Your tests is your debugger.