I'm running tests in parallel on Cucumber using parallel_cucumber.
I'm finding that when I choose to have the reports in junit, everything runs smoothly. If I change the format of the test to html, some tests keep failing. Yet running them in isolation, no parallel_cucumber and it works fine. I switch it back to junit and the tests all pass again.
The errors I'm getting are unable to find fields on screen. Could the html be messing with the step definitions? I've searched and searched and can't figure out why it's failing.
Has anyone experienced a similar issue?
My cucumber.yml file contains:
html: --format pretty --format html --out Reports/report<%= ENV['TEST_ENV_NUMBER']%>.html --format ParallelTests::Cucumber::FailuresLogger --out Reports/cucumber_failures.log
my junit profile is:
junit: --format pretty --format junit --out Reports/ --format ParallelTests::Cucumber::FailuresLogger --out Reports/cucumber_failures.log
On trial and error, I found that it was the background steps causing the failure. I moved these to the scenarios and lo and behold, the tests stopped failing
Related
I have a file with N>10 tests defined in BDD in a feature file.
For debugging purposes, I need to execute only one of the tests many times, but I am not finding any other way than commenting out the other tests.
I am trying now to use the #focus tag in front of the test, in a way that cypress IDE actually marks the rest of the tests as sync skip; aborting execution, but the test that I need to run actually says No commands were issued in this test..
This is the example .feature file I'm playing with:
#e2eTests
#cases
Feature: Test feature
Scenario: Test 1
Given foo
When bar
Then foobar
#focus
Scenario: Test 2
Given bar
When foo
Then barfoo!
And when running npm run cypress and selecting this feature, I get:
What am I doing wrong?
Thanks
I would recommend using cypress-grep if you are looking for a flaky test, that needs repeating n times.
After installation and configuration (see here: Burning Tests with cypress-grep) in terminal indicate which test needs to be repeated and how many times (cypress 7.6) ->
npx cypress run --spec cypress/folder/with/test/testFile.spec.ts, --env grep="name/description of specific test", burn= >>numberOfIterations<<
Example:
npx cypress run --spec cypress/integration/testExample.spec.ts, --env grep="A popup should appear after login",burn=250
So check this out: it appears as though, upon running bundle exec rspec, there's a race between jekyll serve and puma/rspec's boot up. Sometimes I run the command, and my tests run fine. Other times, I get the error for each of my spec files: cannot load such file -- /path/to/project/sitename_jekyll/_layouts/spec/form_spec.rb which is interesting cause that's not where my spec files are located. They're in /path/to/project/sitename_jekyll/spec/form_spec.rb.
What's crazy is that I can literally just re-run the command over and over and over again and sometimes it'll go through and run the spec tests in the correct location, and sometimes it'll look for them in _layouts and error out. It probably runs correctly maybe once out of ever three or five attempts. All the other times I get the following errors:
Here's what my spec_helper.rb looks like: https://gist.github.com/johnhutch/2cddfafcde0485ff021501d5696c0c2d
And here's an example test file:
https://gist.github.com/johnhutch/a35d15c170f5fd9ca07998bf035d111d
My .rspec only contains two lines:
--color
--require spec_helper
And here's the output, both successful and unsuccesful, back to back:
https://gist.github.com/johnhutch/7927d609170ef5c70a595735502b128d
HEEELLLLLP!
This sounds like jekyll is changing the current directory while building the site, which since it is being run in a thread also affects the tests RSpec is trying to run (See https://bugs.ruby-lang.org/issues/9785 for why Dir.chdir is not threadsafe) - leading to attempts to load things from incorrect locations.
A potential solution to this would be to wait for the Jekyll site to be built before actually running your tests. A comment in your spec_helper seems to state that someone thought passing force_build: true would do this but from a quick perusal of the jekyll-rack code I don't think that's true and you actually need to wait for compiling? to return false (v 0.5) (complete? to return true in the current master branch) to ensure building has finished (as well as passing force_build). This could either be done in a loop sleeping and checking (simpler)
sleep 0.1 while <jekyll app>.compiling?
or (if using the master branch) via the mutex/conditional Rack::Jekyll exposes like in its test suite - https://github.com/adaoraul/rack-jekyll/blob/master/test/helper.rb#L49
Note: Also check my comment about your tests that aren't actually testing anything.
As per Thomas Walpole's super helpful responses this ended up working:
sleep 0.1 while Capybara.app.compiling?
inserted right after:
51 Capybara.app = Rack::Jekyll.new(force_build: true)
in my spec_helper.rb
Thanks again, Thomas!
I am new to rspec testing.
I have a folder which contains _spec.rb files. When I run the specs using:
rspec --format html --out rspec_results.html "/Users/phukb/prpc-platform/prpc-platform/test/giza/coreui/ClientPerformance/RDL/spec/"
I see that the tests are run successfully and the results are posted in rspec_results.html.
However, when I create a .rspec file in my project with:
--format html --out rspec_results.html
and run the specs using
rspec "/Users/phukb/Desktop/prpc-platform/prpc-platform/test/giza/coreui/runtime_regression/featuresets/DynamicLayout/Repeating/MicroDC/Pagination/spec"
I see that the results_rspec.html is not getting created. Am I missing something here?
here is my folder structure
If you are running the rspec command from ../RDL folder,then the .rspec file should also be there.
That is, the path of the file should be ../RDL/.rspec, not ../RDL/spec/.rspec.
I followed this guide to get coverage report integrated in WebStorm.
https://www.jetbrains.com/help/webstorm/mocha.html
When I click "Run Mocha test with coverage" the coverage window shows up after running the test. But I don't see any coverage reported. It is not even showing 0%, it is not showing any numbers at all in the Statistics column.
Any help will be appreciated.
Some more background:
WebStorm 2017.2.3
I'm referring Mocha v3.2.0 in package.json
The code is written in ES2015.
The mocha Run configuration uses these environment variables
BABEL_ENV=test and these extra mocha options: --compilers
js:babel-core/register
Thanks in advance
Further information: when I run with coverage in WebStorm I see this command being output to the console:
/Users/nikolaschou/.nvm/versions/node/v8.1.2/bin/node /Users/nikolaschou/Dev/dsg/bifrostshop/node_modules/nyc/bin/nyc.js --reporter=lcovonly --extension=.ts --extension=.js --exclude=test/unitmocha//.spec. --exclude=test/unitmocha//.test. --report-dir=/private/var/folders/q7/kn0zjzks5dz0q2bx0kpg2yhw0000gn/T/mocha-intellij-coverage- /Users/nikolaschou/Dev/dsg/bifrostshop/node_modules/mocha/bin/_mocha --compilers js:babel-core/register --ui bdd --reporter "/Applications/WebStorm 2.app/Contents/plugins/NodeJS/js/mocha-intellij/lib/mochaIntellijReporter.js" --recursive /Users/nikolaschou/Dev/dsg/bifrostshop/test/unitmocha
Add a file called .nycrc in the project root with these contents:
{ "exclude": [] }
This overrides nyc default config and win is restored.
In my case I had this problem because mocha did not exit properly. As the coverage reports seem to be generated on program exit, they are never generated.
Mocha 4 especially has changed behaviour from Mocha 3 as it does "no longer automatically kill itself via process.exit() when it thinks it should be done running".
However you can force that old behaviour with the --exit option.
The better option is probably to examine what keeps your program from terminating and perform a clean shut down. (Maybe with some help of wtfnode for example)
Readings:
Mocha Won't Force Exit
mocha 4 doesn't exit unlike mocha 3
UPDATE: when I run rake test from root, I get:
C:\workspace\faker>rake test
DL is deprecated, please use Fiddle
C:/RailsInstaller/Ruby2.1.0/bin/ruby.exe -w -I"lib;test;." -I"C:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib" "C:/RailsInstaller/Ruby2.1.0/lib/ruby/gems/2.1.0/gems/rake-10.4.2/lib/rake/rake_test_loader.r
b" "test/test_array_sample_method_compat.rb" "test/test_avatar.rb" "test/test_en_au_ocker_locale.rb" "test/test_en_ca_locale.rb" "test/test_en_locale.rb" "test/test_en_ug_locale.rb" "test/test_en_us_locale.rb" "test/test_es_l
ocale.rb" "test/test_faker.rb" "test/test_faker_app.rb" "test/test_faker_bitcoin.rb" "test/test_faker_book.rb" "test/test_faker_business.rb" "test/test_faker_city.rb" "test/test_faker_code.rb" "test/test_faker_color.rb" "test
/test_faker_commerce.rb" "test/test_faker_company.rb" "test/test_faker_date.rb" "test/test_faker_hacker_talk.rb" "test/test_faker_internet.rb" "test/test_faker_lorem.rb" "test/test_faker_name.rb" "test/test_faker_number.rb" "
test/test_faker_shakespear.rb" "test/test_faker_shakespeare.rb" "test/test_faker_slack_emoji.rb" "test/test_faker_street.rb" "test/test_faker_team.rb" "test/test_faker_time.rb" "test/test_faker_university.rb" "test/test_flexi
ble.rb" "test/test_helper.rb" "test/test_locale.rb" "test/test_pl_locale.rb" "test/test_uk_locale.rb"`
I don't fully understand what rake is doing at this point, but the system hangs infinitely. What am I missing?
So I have built some added functionality for the faker gem, and want to run the tests I wrote before submitting a pull request. However, all my attempts to run the existing tests, and my new tests, have failed. I am worried that it is a Windows machine related issue.
I have tried running rake test:units but it gives me an error and then hangs, while also ringing the Bell on my system 5 times. Strange right?
I have also tried running bundle exec rspec spec, but no tests are found becasue they are not RSpec unit tests, or at least not in the right location.
To see how the gem is formatted, check it out at the repo
All the test files, including my new one, live in the test directory. But for the life of me, I can't run them.
I tried this and works fine:
bundle install
bundle exec ruby test