This question already has answers here:
Force retesting or disable test caching
(5 answers)
Closed 4 years ago.
Golang 1.10 introduced caching for test, but there is no obvious way to disable test caching.
So the question is, how to temporal dislable it and how to force rebuild it.
I did read the documentation: https://golang.org/cmd/go/#hdr-Build_and_test_caching but did not found any obvious answer to that question.
The idiomatic way to bypass test caching is to use -count=1. This is the recommended way for doing that in release note as well
The go test command now caches test results: if the test executable
and command line match a previous run and the files and environment
variables consulted by that run have not changed either, go test will
print the previous test output, replacing the elapsed time with the
string “(cached).” Test caching applies only to successful test
results; only to go test commands with an explicit list of packages;
and only to command lines using a subset of the -cpu, -list,
-parallel, -run, -short, and -v test flags. The idiomatic way to bypass test caching is to use -count=1.
Refer : https://golang.org/doc/go1.10#test
Related
I want to know when checking Cache compiled script if available checkbox is wrong,
Following Best practices there are some situations that Cache compiled script shouldn't be used, but the example of not using ${varName} is wrong, I did a test and the value it's taking is the updated value of ${varName} and not the first value.
When using JSR 223 elements, it is advised to check Cache compiled
script if available property to ensure the script compilation is
cached if underlying language supports it. In this case, ensure the
script does not use any variable using ${varName} as caching would
take only first value of ${varName}.
Does someone knows of a real case it's wrong to use caching?
EDIT
I check using ${varName} in script and there are similar results with/without caching:
I define variable in Jmeter called aa with value 1, and created a script:
def aa = "2";
aa = "3";
log.info("${aa}");
Value 1 was return in both cases of checkbox, so it doesn't relate to caching
Also tried with Beanshell (not compiled language without def aa = "2";) and got same results.
What is meant by documentation is that whenever ${varName} has a different value a new entry is stored in cache which will end up filling it with useless data.
So in this case it is wrong and ${varName} should be replaced with
vars.get("varName")
In fact I don't see real reason for unchecking this option provided you use the right JMeter syntax
The option is unchecked by default due to the risk described above and also for "non consensus" reasons:
https://bz.apache.org/bugzilla/show_bug.cgi?id=56554
http://mail-archives.apache.org/mod_mbox/jmeter-dev/201212.mbox/%3CCAH9fUpZ7dMvuxq9U5iQYoBtVX7G-TqOx4aJjjOnPcp%3D5Fc%3D8Qw%40mail.gmail.com%3E
http://mail-archives.apache.org/mod_mbox/jmeter-dev/201301.mbox/%3CCAH9fUpbnde3mnyKmHXwoQeL-SqZUA0Wt1%3D%3D-XMxQWq3ZAS6Pvw%40mail.gmail.com%3E
As to performance, it is exactly the same wether or not you check it for a language that does not support compilation as the first thing JMeter does is to check "supportsCompilable" before using the checkbox, see:
https://github.com/apache/jmeter/blob/trunk/src/core/org/apache/jmeter/util/JSR223TestElement.java#L171
You should not be using caching when you use a scripting engine which does not support caching if compiled scripts. Since only Groovy is capable of compiling scripts you should tick this box for Groovy and untick for the other engines (the code block which does not make sense will be triggered each time the script will be called)
Well-behaved Groovy engine should:
Compile script in the runtime to avoid interpretation each time
Cache compiled script to avoid recompilation
Recompile script and update the cache if any changes are being made to it.
Inlining JMeter functions and variables into scripts is a little bit dangerous for any language as it might resolve into something which cause compilation failure or even even worse result in code which you don't expect. In case of Groovy JMeter Variables syntax conflicts with Groovy GString template
So inlining variables will cause overhead for recompiling the script each time it's called.
So please continue following JMeter's Best Practices and remember one more little tip: avoid scripting where possible as none scripting option performance behaves as fast as Java does, check out Beanshell vs JSR223 vs Java JMeter Scripting: The Performance-Off You've Been Waiting For! guide for details.
I have compiled and tested an open-source command line SIP client for my machine which we can assume has the same architecture as all other machines in our shop. By this I mean that I have successfully passed a compiled binary to others in the shop and they were able to use them.
The tool has a fairly esoteric invocation, a simple bash script piped to it prior to execution as follows:
(sleep 3; echo "# 1"; sleep 3; echo h) | pjsua sip:somephonenumber#ip --flag_1 val --flag_2 val
Note that the leading bash script is an essential part of the functioning of the program and that the line itself seems to be the best practice for use.
In the framing of my problem I am considering the following:
I don't think I can expect very many others in the shop to
compile the binary for themselves
Having a common system architecture in the shop it is reasonable to think that a repo can house the most up-to-date version
Having a way to invoke the tool using Ruby would be the most useful and the most accessible to the
most people.
The leading bash script being passed needs to be wholly extensible. These signify modifiable "scenarios" e.g. in this case:
Call
Wait three seconds
Press 1
Wait three seconds
Hang up
There may be as many as a dozen flags. Possibly a configuration file.
Is it a reasonable practice to create a gem that carries at its core a command line tool that has been previously compiled?
It seems reasonable to create a gem that uses a command line tool. The only thing I'd say is to check that the command is available using system('which psjua') and raising an informative error if it hasn't been installed.
So it seems like the vocabulary I was missing is extension. Here is a great stack discussion on wrapping up a Ruby C extension in a Ruby Gem.
Here is a link to the Gem Guides on creating Gems with Extensions.
Apparently not only is it done but there are sets of best practices around its use.
If I have different packages and each have a test file (pkg_test.go) is there a way for me to make sure that they run in a particular order ?
Say pkg1_test.go gets executed first and then the rest.
I tried using go channels but it seems to hang.
It isn't obvious, considering a go test ./... triggers test on all packages... but runs in parallel: see "Go: how to run tests for multiple packages?".
go test -p 1 would run the tests sequentially, but not necessarily in the order you would need.
A simple script calling go test on the packages listed in the right expected order would be easier to do.
Update 6 years later: the best practice is to not rely on test order.
So much so issue 28592 advocates for adding -shuffle and -shuffleseed to shuffle tests.
CL 310033 mentions:
This CL adds a new flag to the testing package and the go test command
which randomizes the execution order for tests and benchmarks.
This can be useful for identifying unwanted dependencies
between test or benchmark functions.
The flag is off by default.
If -shuffle is set to on then the system
clock will be used as the seed value.
If -shuffle is set to an integer N, then N will be used as the seed value.
In both cases, the seed will be reported for failed runs so that they can reproduced later on.
Picked up for Go 1.17 (Aug. 2021) in commit cbb3f09.
See more at "Benchmarking with Go".
I found a hack to get around this.
I named my test files as follow:
A_{test_file1}_test.go
B_{test_file2}_test.go
C_{test_file3}_test.go
The A B C will ensure they are run in order.
Closed. This question needs to be more focused. It is not currently accepting answers.
Want to improve this question? Update the question so it focuses on one problem only by editing this post.
Closed 8 years ago.
Improve this question
Though I have been using CasperJS for some time, and rely on console logging for debugging. I was wondering if there is any IDE which support CasperJS step by step debugging or there is other way(remote debugging) to step in to CasperJS code? Has anybody successfully done it? Any information will be helpful.
Thanks,
When I want to debug with CasperJS, I do the following : I launch my script with slimerJS (it opens a firefox window so I can easilly see click problem, problems of form filling-ajax return error, media uploading...-, and in which step the code blocks).
With that I don't often need to look at the console and I don't call this.capture('img.jpg') several times to debug (for now i don't test responsive design so i don't need to use capture, take a look at PhantomCSS if you test it).
I use slimerJS to debug (always with casper), but phantomJS in continuous Integration-jenkins- (headless), though you can use slimerjs too (being headless) with xvfb on linux or Mac.
But sometimes i have to look at the console for more details, so I use these options (you can call them in command line too) :
casper.options.verbose = true;
casper.options.logLevel ="debug";
Name your closures will be useful with these options, because the name will be displayed.
I don't think there is an IDE : if a step fail, the stack with all the following steps stops anyway (well it's still possible to do a 'sort of hack' using multiple wait -and encapsulate them- to perform differents closures and have the result of all of them, even if one of them fail; but in this case we are not stacking synchronous steps, but executing asynchronous instructions : take care about timeout and logical flow if you try it). Care : i said 'if a step fail', if it's just an instruction in a closure which fails, of course the following steps will be execute.
So a closure which fails -> following steps are executed.
A step which fails (ex : thenOpen(fssfsf) with fssfsf not defined), the stack will stop.
Multiple wait() can be done asynchronously.
So if you have a lot of bugs and execute your tests sequentially (stacking steps), you can only debug them one by one, or by closure for independant step functions-I think an IDE could work in this case- (->for one file. The stacks are of course independent if you launch a folder). And usually at the beginning you launch your file each time you finish a step. And when you're used to the tool, you write the whole script at once.
In most cases, bug are due to asynchrone, scope, context problems (actually js problems, though casper simplifies asynchronous problems : "The callback/listener stuff is an implementation of the Promise pattern.") :
If you want to loop a suit of instructions, don't forget the IIFE or use the casper each function, and encompass all with a then() statement (or directly eachThen). I can show some exemples if needed. Otherwise it will loop the last index 'i.length' times, there isn't loop scope in js, by default i has the same reference.
When you click on a link at the end of a step, use a wait()-step function too- statement after; instead of a then(). If i've well understood the then() statement, it's launched when the previous step is completed. So it's launched just after the click(). If this click launches an ajax return, and your following step scrape or test the result of this ajax return, it will randomly fail, because you haven't explicitly ask to wait for the resource. I saw some issues like that in my first tests.
Don't mixt the two context : casper environment and page DOM environment. Use the evaluate function() for passing from one to the other. In the evaluate function, you can pass an argument from the casper context to the page DOM...
...Like that :
var casperContext = "phantom";
casper.evaluate(function(pageDomContext) {
console.log("will echo ->phantom<- in the page DOM environment : " + pageDomContext + ", use casper.on('remote.message') to see it in the console");
}, casperContext);
Or you can see it directly in the browser with slimerJS using alert() instead of console.log().
Use setFiltrer to handle prompt and confirm box.
If your website exists in mobile version too, you can manipulate the userAgent for your mobile tests.
You can call node modules in a casperJS file, in files using the tester module too. Well it's not entirely true, see use node module from casper. Some core node features are implemented in phantom (and slimer too), like fs, child process, but they are not always well documented. I prefer to execute my tests with node so. Node is useful to launch your tests in parallel (child process). I suggest you to execute as many processes as you have core. Well, it depends of your type of script, with just normal scenario (open a page and check some elements) I can execute 10 child process in parallel whithout random failure (local computer), but with some elements which are slow to load (as multi svg, sometimes xml...), use require('os').cpus().length or a script like that : Repeat a step X times. Otherwise you will have random failure, even if you increase the timeout. When it crashes, you can't do anything other that reload() the page.
You can then integrate your tests in jenkins using the xunit command. Just specify differents index for each log.xml files, jenkins (XUnit -> JUnit) will manage them : pattern *.xml.
I know i didn't really answer your question, but i think to debug, list the main specific problems remains the best way.
There are still useful functions to debug :
var fs = require('fs');
fs.write("results.html", this.getPageContent(), 'w');
I prefer this way rather than this.debugHTML(). I can check in my results.html file if there are missing tags (relative to the browser using firebug or another tool). Or sometimes if i need to check just one tag, output the result in the console isn't a problem, so : this.getHTML("my selector"); and you can still pipe the log result : casperjs test test.js > test.html
Another trick : if you execute your tests in local, sometimes the default timeout isn't sufficient (network freeze) (5sec).
So -> 10sec :
casper.options.waitTimeout = 10000;
Some differences between Phantom and Slimer :
With slimer, if you set casper.options.pageSettings.loadImages = false; and in your file you try to scrape or test the weight/height.... of an element, it will work with slimer but not with phantom. So set it to true in the specific file to keep the compatibility.
Need to specify an absolute path with slimer (with include, import->input media, ...).
Example :
this.page.uploadFile('input[name="media"]', fs.absolute(require('system').args[4]).split(fs.separator).slice(0, -1).join(fs.separator) + '/../../../../avatar.jpg');
To include a file from the root folder (work in every subdir/OS, better than previous inclusion) -you could also do it nodeLike using require()-:
phantom.injectJs(fs.workingDirectory + '/../../../../global.js');
Too often people write tests that don't clean up after themselves when they mess with state. Often this doesn't matter since objects tend to be torn down and recreated for most tests, but there are some unfortunate cases where there's global state on objects that persist for the entire test run, and when you run tests, that depend on and modify that global state, in a certain order, they fail.
These tests and possibly implementations obviously need to be fixed, but it's a pain to try to figure out what's causing the failure when the tests that affect each other may not be the only things in the full test suite. It's especially difficult when it's not initially clear that the failures are order dependent, and may fail intermittently or on one machine but not another. For example:
rspec test1_spec.rb test2_spec.rb # failures in test2
rspec test2_spec.rb test1_spec.rb # no failures
In RSpec 1 there were some options (--reverse, --loadby) for ordering test runs, but those have disappeared in RSpec 2 and were only minimally helpful in debugging these issues anyway.
I'm not sure of the ordering that either RSpec 1 or RSpec 2 use by default, but one custom designed test suite I used in the past randomly ordered the tests on every run so that these failures came to light more quickly. In the test output the seed that was used to determine ordering was printed with the results so that it was easy to reproduce the failures even if you had to do some work to narrow down the individual tests in the suite that were causing them. There were then options that allowed you to start and stop at any given test file in the order, which allowed you to easily do a binary search to find the problem tests.
I have not found any such utilities in RSpec, so I'm asking here: What are some good ways people have found to debug these types of order dependent test failures?
There is now a --bisect flag that will find the minimum set of tests to run to reproduce the failure. Try:
$ rspec --bisect=verbose
It might also be useful to use the --fail-fast flag with it.
I wouldn't say I have a good answer, and I'd love to here some better solutions than mine. That said...
The only real technique I have for debugging these issues is adding a global (via spec_helper) hook for printing some aspect of database state (my usual culprit) before and after each test (conditioned to check if I care or not). A recent example was adding something like this to my spec_helper.rb.
Spec::Runner.configure do |config|
config.before(:each) do
$label_count = Label.count
end
config.after(:each) do
label_diff = Label.count - $label_count
$label_count = Label.count
puts "#{self.class.description} #{description} altered label count by #{label_diff}" if label_diff != 0
end
end
We have a single test in our Continuous Integration setup that globs the spec/ directory of a Rails app and runs each of them against each other.
Takes a lot of time but we found 5 or 6 dependencies that way.
Here is some quick dirty script I wrote to debug order-dependent failure - https://gist.github.com/biomancer/ddf59bd841dbf0c448f7
It consists of 2 parts.
First one is intended to run rspec suit multiple times with different seed and dump results to rspec_[ok|fail]_[seed].txt files in current directory to gather stats.
The second part is iterating through all these files, extracts test group names and analyzes their position to the affected test to make assumptions about dependencies and forms some 'risk' groups - safe, unsafe, etc. The script output explains other details and group meanings.
This script will work correctly only for simple dependencies and only if the affected test is failing for some seeds and passes for another ones, but I think it's still better than nothing.
In my case it was complex dependency when effect could be cancelled by another test but this script helped me to get directions after running its analyze part multiple times on different sets of dumps, specifically only on the failed ones (I just moved 'ok' dumps out of current directory).
Found my own question 4 years later, and now rspec has a --order flag that lets you set random order, and if you get order dependent failures reproduce the order with --seed 123 where the seed is printed out on every spec run.
https://www.relishapp.com/rspec/rspec-core/v/2-13/docs/command-line/order-new-in-rspec-core-2-8
Its most likely some state persisting between tests so make sure your database and any other data stores (include class var's and globals) are reset after every test. The database_cleaner gem might help.
Rspec Search and Destroy
is meant to help with this problem.
https://github.com/shepmaster/rspec-search-and-destroy