I am using Jasmine - 3.3.1, with combination of ProtractorJS.
My requirement is to store the result for each spec (or describe / test) and update results in Testrail system using the afterEach() method. I want to store the results into the variable "testResult".
Tried with various methods - custom_reports.js etc, but could not get what I needed.
Code snippet:
var testResult;
describe('1st scenario', function () {
it('1st Test', function () {
expect(true).toBe(true);
testResult=5;
});
});
describe('2nd scenario', function () {
it('2nd Test', function () {
expect(true).toBe(true);
testResult=1;
});
});
afterEach(function () {
helper.updateResults("Section Name", testcaseID, testResult);
});
I achieved something similar by creating my own custom reporter. My reporter uploads spec results (it blocks) to a dynamoDB table after each spec finishes and uploads suite results (describe blocks) after all tests have finished. All uploads occur asynchronously but in the onComplete all of the async upload actions are awaited.
Obviously I am using the async / await approach as opposed to the SELENIUM_PROMISE_MANAGER you appear to. I would recommend making that change over.
DBReporter.js
function dbReporter() {
this.jasmineStarted = function (options) {};
this.specStarted = function (result) {};
this.specDone = async function (result) {
if (result.status == 'pending') {
}
else if (result.status == 'passed') {
}
else if (result.status == 'failed') {
//Put your testrail interaction code here
}
testResultsUploadQueue.push(result);
};
this.suiteStarted = function (result) {};
this.suiteDone = function (result) {}
this.jasmineDone = async function (result) {}
}
module.exports = dbReporter;
conf.js
onPrepare: async () => {
//require the dpReporter file
let dbReporter = require('../src/functions/db-reporter');
//Declare a global variable that will contain all the asynchronous upload actions (promises)
global.testResultsUploadQueue = [];
//initialize the dbreporer
await jasmine.getEnv().addReporter(new dbReporter());
}),
onComplete: async() => {
//Wait for all uploads to resolve before completing
let testRulesUploadValue = await Promise.all(testResultsUploadQueue);
console.log(` ${testRulesUploadValue.length} result files uploaded to dynamoDB`);
}
No changes required in your spec files
The Constraints
I had a lot of issues with handling asynchronous actions with the reporter, this was why I chose to use the queue approach. I could not figure out how to get around this but this approach does work.
Your TestRail actions must return a promise
It is important to understand the execution order of the hooks in order to understand the solution.
--- beforeLaunch
--- onPrepare
--- jasmineStarted (set in jasmine reporter)
--- beforeAll
--- suiteStarted (set in jasmine reporter)
--- specStarted (set in jasmine reporter)
--- beforeEach
+++ afterEach
+++ specDone (set in jasmine reporter)
+++ suiteDone (set in jasmine reporter)
+++ afterAll
+++ jasmineDone (set in jasmine reporter)
+++ onComplete
+++ afterLaunch
Related
I have the code bellow. when I run the code I get the data is undefined. I do need the before hook because I use async code and describe does not support async. I this case the before hook is not being ran and the execution goes directly to the test loop. Is there any way I can fix this problem by seeding my data inside the before hook prior to the test loop execution or making the describe function to execute async code ?
describe("some function", () => {
var users
data;
before(async () => {
users = await getAllUsers();
data = [
//seeding data
]
});
data.forEach(({fName, lName}) => {
it(`${fName}`, async function () {
//test execution
})
})
})
I want to skip and allow tests in the before each hook as follows
beforeEach(() =>{
if(Cypress.mocha.getRunner().suite.ctx.currentTest.title === `Skip this`){
// skip the first test case only but run the second one [How?]
}
});
it(`Skip this`, () => {
});
it(`Don't skip this`, () => {
});
In the place of [How?] I tried using the following:
cy.skipOn(true) from the cypress skip-test plugin but apparently it skips the beforeEach hook not the test itself.
this.skip() but apparently this is not a valid function. Also, if I changed the beforeEach from an arrow function expression, the skip function works but it skips the whole suite and not just the desired test case.
Any ideas?
Change the function type from arrow function to regular function, then you can use the built-in Mocha skip() method.
beforeEach(function() {
if (condition) {
this.skip()
}
})
Your code sample will look like this:
beforeEach(function() { // NOTE regular function
if (Cypress.mocha.getRunner().suite.ctx.currentTest.title === 'Skip this') {
this.skip()
}
});
it(`Skip this`, () => {
});
it(`Don't skip this`, () => {
});
Or use the Mocha context you already use for test title
beforeEach(() => { // NOTE arrow function is allowed
const ctx = Cypress.mocha.getRunner().suite.ctx
if (ctx.currentTest.title === 'Skip this') {
ctx.skip()
}
});
afterEach()
If you have an afterEach() hook, the this.skip() call does not stop it running for the skipped test.
You should check the condition inside that hook also,
afterEach(function() {
if (condition) return;
... // code that should not run for skipped tests.
})
Is it possible to set a message (mentioning reason) why a particular test is skipped, so that it could be used in reporters.
describe('xxx', function() {
checkToSkip(1)("test1", function() {\*test goes here*\});
checkToSkip(4)("test2", function() {\*test goes here*\});
});
function checkToSkip(now) {
return now > 3 ? it : xit;
//return now > 3 ? it : it.skip;
}
Here 'test1' will be skipped as 'checkToSkip' returns 'xit' (or it.skip). Is it possible to pass a message to reporter mentioning the reason for the test skip? something like below (or any other possible way).
checkToSkip(4)("test2", function() { \\ test goes here}, "My Skip message!!!!" );
Note: Im using mocha in webdriverIO.
Thanks.
I'd just slightly modify the checkToSkip function and used Pending Tests instead of skip().
function checkToSkip(now, title, testCallable) {
if (now > 3) {
it(title, testCallable);
} else {
it(title+"#My Skip message!#");
}
}
Then use it like:
describe('xxx', function() {
checkToSkip(1, "test1", function() {\*test goes here*\});
});
It's possible to modify the test title itself inside the test before skipping:
it('My cool test', async function() {
if (this.response.status !== 201) {
this._runnable.title += ' - Skipped with reason: wrong response code'
this.skip()
}
expect(this.response.data).to.have.property('mycoolproperty')
})
Also possible to formalize the check and move it to the external function:
function skipIf(that, condition, reason){
if (condition) {
that._runnable.title += ` - Skipped with reason: ${reason}`
that.skip()
}
}
So the test will look like that:
it('My cool test', async function() {
skipIf(this, this.response.status !==201, 'Wrong response status')
expect(this.response.data).to.have.property('mycoolproperty')
})
Following the testingBot example for protractor-based projects I got this code
var TestingBot = require('testingbot-api');
describe('Protractor Demo App', function () {
var tb;
beforeEach(function () {
tb = new TestingBot({
api_key: "master_key",
api_secret: "secret_007"
});
});
afterEach(function () {
browser.getSession().then(function (session) {
tb.updateTest({
'test[success]': true/*where do I get this 'test[success]' attribute? */
}, session.getId(), function () {
console.log("Hi! :D");
});
})
});
it('should have a title', function () {
browser.get('http://juliemr.github.io/protractor-demo/');
expect(browser.getTitle()).toEqual('Super Calculator');
});
});
I need to send the success of the test back through the tb.updateTest() but I don't know where I get the value of a passed or failed test. For now the value is a static true. I'd appreciate a jasmine approach too.
You can use a custom reporter with Jasmine.
There you can hook into specDone or suiteDone which has a result parameter, containing the test's success state.
You can then use this state to write a custom report or send it to somewhere else.
I have a suite which includes multiple specs. Each spec uses code on some libraries that return a rejected promise upon failure.
I can easily catch those rejected promises inside my spec. What I'm wondering about is that if I can make Protractor exit the whole suite inside that catch function, because the next specs inside the same suite are dependent on the success of the previous specs.
Pretend I have a suite called testEverything which has these specs openApp,signIn,checkUser,logout. If openApp fails, all next specs will fail due to dependency.
Consider this code for openApp:
var myLib = require('./myLib.js');
describe('App', function() {
it('should get opened', function(done) {
myLib.openApp()
.then(function() {
console.log('Successfully opened app');
})
.catch(function(error) {
console.log('Failed opening app');
if ( error.critical ) {
// Prevent next specs from running or simply quit test
}
})
.finally(function() {
done();
});
});
});
How would I exit the whole test?
There is a module for npm called protractor-fail-fast. Install the module npm install protractor-fail-fast. Here's an example from their site where you would place this code into your conf file:
var failFast = require('protractor-fail-fast');
exports.config = {
plugins: [{
package: 'protractor-fail-fast'
}],
onPrepare: function() {
jasmine.getEnv().addReporter(failFast.init());
},
afterLaunch: function() {
failFast.clean(); // Cleans up the "fail file" (see below)
}
}
Their url is here.
I have managed to come up with a workaround. Now the actual code that I used is way more complex, but the idea is the same.
I added a global variable inside protractor's config file called bail. Consider the following code at the top of config file:
(function () {
global.bail = false;
})();
exports.config: { ...
The above code uses an IIFE (Immediately Invoked Function Expression) which defines bail variable on protractor's global object (which would be available throughout the whole test).
I also have written asynchronous wrappers for the Jasmine matchers I need, which would execute an expect expression followed by a comparison, and return a promise (using Q module). Example:
var q = require('q');
function check(actual) {
return {
sameAs: function(expected) {
var deferred = q.defer();
var expectation = {};
expect(actual).toBe(expected);
expectation.result = (actual === expected);
if ( expectation.result ) {
deferred.resolve(expectation);
}
else {
deferred.reject(expectation);
}
return deferred.promise;
}
};
}
module.exports = check;
Then at the end of each spec, I set the bail value based on the spec's progress, which would be determined by the promise of those asynchronous matchers. Consider the following as first spec:
var check = require('myAsyncWrappers'); // Whatever the path is
describe('Test', function() {
it('should bail on next spec if expectation fails', function(done) {
var myValue = 123;
check(myValue).sameAs('123')
.then(function(expectation) {
console.log('Expectation was met'); // Won't happen
})
.catch(function(expectation) {
console.log('Expectation was not met'); // Will happen
bail = true; // The global variable
})
.finally(function() {
done();
});
});
});
Finally, on the beginning of next specs, I check for bail and return if necessary:
describe('Test', function() {
it('should be skipped due to bail being true', function(done) {
if ( bail ) {
console.log('Skipping spec due to previous failure');
done();
return;
}
// The rest of spec
});
});
Now I want to mention that there's one module out there called protractor-fail-fast which bails on the whole test whenever an expectation fails.
But in my case, I needed to set that bail global variable depending on which type of expectation has been failed. I ended up writing a library (really small) that distinguishes failures as critical and non-critical and then using that, specs would be stopped only if a critical failure has occurred.