In RSpec I would use the following to obtain a Random which is seeded from the test order random seed. This would give us a reproducible RNG for things like fuzzing:
reproducible_random = Random.new(RSpec.configuration.seed)
Now that I am working with a project that uses Minitest (the Rails default) I can't seem to find how to obtain the Minitest seed value and produce a reproducible test. Is there a way to access it someplace from inside the tests themselves?
As of Minitest 5.16.0, the seed is available via Minitest.seed
There is an open issue about it in mintest from 2016.
I believe as of now it is not possible.
You can use the --seed parameter when calling minitest or the SEED environment variable, [source code](https://github.com/seattlerb/minitest/blob/fe3992e85b40792cf7bff2a876887d8d9e392068/lib/minitest.rb#L190
Related
I've created a very simple OpenAI gym (banana-gym) and wonder if / how I should implement env.seed(0).
See https://github.com/openai/gym/issues/250#issuecomment-234126816 for example.
In a recent merge, the developers of OpenAI gym changed the behavior of env.seed() to not call the method env._seed() anymore. Instead the method now just issues a warning and returns. I think if you want to use this method to set the seed of your environment, you should just overwrite it now.
The docstring of the env.seed() function (which can be found in this file) provides the following documentation on what the function should be implemented to do:
Sets the seed for this env's random number generator(s).
Note:
Some environments use multiple pseudorandom number generators.
We want to capture all such seeds used in order to ensure that
there aren't accidental correlations between multiple generators.
Returns:
list<bigint>: Returns the list of seeds used in this env's random
number generators. The first value in the list should be the
"main" seed, or the value which a reproducer should pass to
'seed'. Often, the main seed equals the provided 'seed', but
this won't be true if seed=None, for example.
Note that, unlike what the documentation and the comments in the issue you linked to seem to imply, it doesn't seem (to me) like env.seed() is supposed to be overridden by custom environments. env.seed() has a very simple implementation, where it only calls and returns the return value of env._seed(), and it seems to me like that is the function which should be overridden by custom environments.
For example, OpenAI gym's atari environments have a custom _seed() implementation which sets the seed used internally by the (C++-based) Arcade Learning Environment.
Since you have a random.random() call in your custom environment, you should probably implement _seed() to call random.seed(). In that way, users of your environments can reproduce experiments by making sure to call seed() on your environment with the same argument.
Note: Messing around with the global random seed like this may be unexpected though, it may be better to create a dedicated random object when your environment gets initialized, seed that object, and make sure to always obtain your random numbers if you need them in the environment from that object.
env.seed(seed) works like a charm. The key is to seed the env not just at the beginning, but EVERY time the reset() function is called. Since we invariably end up playing multiple games during training this seeding function should be inside one of the loops and will be executed multiple times. Possibly this is the reason why it is deprecated now in favor of env.reset(seed=seed)
Of course needless to say, if you are using randomness in the agent, you need to seed that as well. In that case, seeding once at the start of the training would be fine. You may want to seed the NN framework as well.. A typical seeding function (Pytorch) would be:
def seed_everything(seed):
random.seed(seed)
np.random.seed(seed)
os.environ['PYTHONHASHSEED'] = str(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.backends.cudnn.deterministic = True
env.seed(seed)
##One call at beginning is enough
seed_everything(SEED)
Howver remember to call env.seed every time you reset the env:
curr_state = env.reset()
env.seed(SEED)
or simply use the new API: env.reset(seed=seed)
BUT - While this determinism may be used in early training to debug your code, it is recommended not to use the the same(ie env.seed(SEED)) in your final training. This is because, by nature, the start position of the environment is supposed to be random and your RL code is expected to function considering that randomness. If you make your start position deterministic then your model will not perform well in the live environment
I have following cucumber test
Scenario Outline:
Given site <url> is available
Then I see all all content
Examples:
|url|
|"google.com|
In the test case is dynamic and is generated in ruby code.
Problem:
I want to replace google.com with a ruby variable e.g., <%URL%>. Is that possible to embed ruby code in cucumber tests and evaluate ?
I think you should not do that on the feature steps. If you need a ruby variable here it means that you are doing something wrong. Check some examples around
Link here
The features should be clear text so anyone can read, specially non-programmers. so that is why you should now start mixing code with features. The code goes behind, in your step definition.
You can eval this string it if you trust code in this feature file. But it may be a bad idea for the reasons outlined by #SnakeSanders.
Can Anyone point Me to a tool to detect unused code, objects, methods, parameters, etc., in Ruby code?
I saw nitpick but it does not seem to give me the coverage I want. I also checked laser and reek but their respective gems seem to have issues which prevent them from running.
I thought at one point the Ruby binary had a mode which would detect unused constructs but I do not seem to be able to find it.
It might help if we had a little more context in how you want to "detect unused code" - is this code coverage of your tests you're looking into? Otherwise, how would you know from run to run whether you hit all the use cases? Or are you looking for a statistical "heat map" of coverage over time for e.g. performance reasons?
In any case, for code coverage while testing I use SimpleCov - it uses Ruby 1.9's built-in Coverage library with some nice sugar on top.
You can also use a mutation tester that mutates your code. In case the mutation tester can delete a construct without your tests noticing. You found an unused construct.
I know two mutation testers for ruby:
Heckle
Mutant
Disclaimer, I'm the author of mutant.
Depending on your setup, your ruby version, spec layout, test framework heckle and or mutant can do the job for you.
Here you can see mutant in action: http://ascii.io/a/1707
JetBrains RubyMine http://www.jetbrains.com/ruby/quickstart/index.html
I am testing a piece of Rails code that reads:
sleep(10.0)
In my RSpec tests, calling:
Kernel.should_receive(:sleep).exactly(1).time
failed and the test slept for ten seconds. This led me to conclude that sleep() in a Rails program isn't calling Kernel.sleep(). I verified this by changing my Rails code to:
Kernel.sleep(10.0)
... after which my RSpec tests passed (and the test didn't sleep).
This leads to a specific and a general question:
What implementation of sleep() does Rails use (I'm running Ruby 1.9.3 / Rails 3.2.1)?
From the interpreter, what's the easiest way to find the source code for any function?
The implicit receiver, when you don't specify an explicit one, is self, not Kernel. (Why would you think that?)
So,
sleep(10.0)
is roughly the same as
self.sleep(10.0)
and not at all the same as
Kernel.sleep(10.0)
So, it is calling Kernel#sleep on self and not on Kernel. Which means you need to set an expectation on whatever object self is in that particular method.
I have a bunch of RSpec Rake tasks defined that I'd like to run in a specific order when I run the entire suite of tests.
I've tried something like this:
task :run_in_order => [:one, :two, :three] do
puts "Run suite of tests"
end
And it runs the first test and then stops and doesn't run the rest of the tasks.
So if using Rake syntax it doesn't work. Is there a way to note task dependencies for RSpec Rake tasks?
Check this http://blog.davidchelimsky.net/2012/01/04/rspec-28-is-released/
–order rand We added an --order option with two supported values: rand
and default.
rspec --order random (or rand) tells RSpec to run the groups in a
random order, and then run the examples within each group in random
order. We implemented it this way (rather than complete randomization
of every example) because we don’t want to re-run expensive
before(:all) hooks. A fair tradeoff, as the resulting randomization is
just as effective at exposing order-dependency bugs.
When you use --order random, RSpec prints out the random number it
used to seed the randomizer. When you think you’ve found an
order-dependency bug, you can pass the seed along and the order will
remain consistent:
--order rand:3455
--order default tells RSpec to load groups and examples as they are declared in each file
Basically you should order your tests in a spec file an then execute it with --order default option.
.