Nightwatch waitForElementVisible - abortOnFailure parameter set to false - test exit status non-zero - nightwatch.js

Question

I am using the waitForElementVisible(<selector>, <timeout>, false) command from the Nightwatch API docs and it is not behaving quite like I expected. How can I tweak this code in order to get the intended behavior?
Intended behavior:
call .waitForElementVisible('foobar', 10, false)
see command fail and continue execution with the next command
all other commands pass
see exit status of 0 from script
Actual behavior:
call .waitForElementVisible('foobar', 10, false)
see command fail and continue execution with the next command
all other commands pass
see exit status of 1 from script
Here is example code to reproduce
module.exports = {
tags: ['smoke'],
before: browser =>
browser
.maximizeWindow('current').url('https://google.com'),
after: browser => browser.end(),
'smoke test': browser =>
browser
.waitForElementVisible('foobar', 10, false)
.waitForElementVisible('img')
.assert.visible('img'),
};
And here is the console output from running that command:
Starting selenium server in parallel mode... started - PID: 75459
Started child process for: 01_smoke
01_smoke \n
01_smoke [01 Smoke] Test Suite
=========================
01_smoke
01_smoke Results for: smoke test
01_smoke ✖ Timed out while waiting for element <foobar> to be present for 10 milliseconds. - expected "visible" but got: "not found"
01_smoke at Object.smokeTest [as smoke test] (/path/to/tests/01_smoke.js:12:8)
at _combinedTickCallback (internal/process/next_tick.js:131:7)
01_smoke ✔ Element <img> was visible after 33 milliseconds.
01_smoke ✔ Testing if element <img> is visible.
01_smoke
01_smoke Retrying (1/3): smoke test
01_smoke ✖ Timed out while waiting for element <foobar> to be present for 10 milliseconds. - expected "visible" but got: "not found"
01_smoke at Object.smokeTest [as smoke test] (/path/to/tests/01_smoke.js:12:8)
at _combinedTickCallback (internal/process/next_tick.js:131:7)
01_smoke ✔ Element <img> was visible after 21 milliseconds.
01_smoke ✔ Testing if element <img> is visible.
01_smoke
01_smoke Retrying (2/3): smoke test
01_smoke ✖ Timed out while waiting for element <foobar> to be present for 10 milliseconds. - expected "visible" but got: "not found"
01_smoke at Object.smokeTest [as smoke test] (/path/to/tests/01_smoke.js:12:8)
at _combinedTickCallback (internal/process/next_tick.js:131:7)
01_smoke ✔ Element <img> was visible after 20 milliseconds.
01_smoke ✔ Testing if element <img> is visible.
01_smoke Retrying (3/3): smoke test
01_smoke ✖ Timed out while waiting for element <foobar> to be present for 10 milliseconds. - expected "visible" but got: "not found"
01_smoke at Object.smokeTest [as smoke test] (/path/to/tests/01_smoke.js:12:8)
at _combinedTickCallback (internal/process/next_tick.js:131:7)
01_smoke ✔ Element <img> was visible after 20 milliseconds.
01_smoke ✔ Testing if element <img> is visible.
01_smoke FAILED: 1 assertions failed and 2 passed (53ms)
>> 01_smoke finished.
_________________________________________________
TEST FAILURE: 1 assertions failed, 2 passed. (6.259s)
✖ 01_smoke
- smoke test (53ms)
Timed out while waiting for element <foobar> to be present for 10 milliseconds. - expected "visible" but got: "not found"
at Object.smokeTest [as smoke test] (/path/to/tests/01_smoke.js:12:8)
at _combinedTickCallback (internal/process/next_tick.js:131:7)

Answer 1

Yes, and that's the way it should be! I think you misinterpreted the way the abortOnFailure flag works for the waitForVisible command. The false flag only gives the method the character to be evaluated by the test-runner as a non-breaking
step. But that doesn't mean it doesn't keep count of that step as being a failed step.
Note: A similar thing happens in the assert/verify case (where verify is a non-breaking assertion, similar to the abortOnFailure: false parameter for waitForElementVisible).
I can see where one would get that impression though. If you read the API call's description it said:
If the element fails to be present and visible in the specified amount
of time, the test fails. You can change this by setting abortOnFailure
to false.
Which leaves you thinking that perhaps the test will end up passing, even though the waitForVisible command failed. But... the Parameters section of the API call comes to our aid, removing that false assumption:
By the default if the element is not found the test will fail. Set
this to false if you wish for the test to continue even if the
assertion fails. To set this globally you can define a property
abortOnAssertionFailure in your globals.
Lastly... where the DOCs might fail you, the code will never lie:
process.on('exit', function (code) {
var exitCode = code;
if (exitCode === 0 && globalResults && (globalResults.errors > 0 || globalResults.failed > 0)) {
exitCode = 1;
}
process.exit(exitCode);
});
That above is the is a snippet Nightwatch's test-runner (nightwatch/lib/runner/run.js). See for yourself what it considers as valid exit code 1 conditions.
Cheers!