One of my automated test cases (TC) fails predictably after a prod refresh that takes place every few months.
For the TC to pass, there should be 'N/A' for values, which is a precondition. After getting the 'N/A' text, I do insert into a table to create values and then do other steps.
After the refresh, there are values (monies) instead of the 'N/A'.
What are the ways to avoid the failure? Run Keyword If and Run Keyword And Expect Failure would invalidate the original TC and and it will always pass, which I apparently don't need.
There might be other approaches too, however, one of the ways to approach this problem is
You can define init file in directory
__init__.robot
That suite setup and suite teardown in the file would run before anything in the underlying folders.
make use of set global variable with N/A and update the same when you see actual values. i.e every test case would verify whether the variable contains N/A or actual values(i.e not N/A), this can be done using Test Setup with keyword.
NOTE: You can also use set suite variable for the same
Related
I am trying to build a jmeter testplan, where all the test values are sent from a csv datafile.I want to add assertions(provided in the datafile) to my HTTP Request at runtime and execute the test. The reason behind doing this is to keep the plan flexible according to the number of assertions. In my case, the assertions are getting added at the runtime; however they fail to get executed. May I know what should be done to get the components added and executed in the same flow?
For example: A part of plan looks like:
XYZ
--HTTP Sampler
-- Response Assertion1
-- Response Assertion2
-- JSON Extractor
where XYZ -->keyword based transaction controller(reusable component)
Everytime I have a request of type XYZ, this chunk of components will get executed. In my case, I do not want to place anything such as Assertions, pre/post processors, extractors in the test plan already. I want to generate these components at run time and execute them (as per my test requisites).
Issue: The problem here is that I cant load the components programmatically and execute them in the same flow. The reason being, the compiler does not know beforehand what all components it needs to execute, so it bypasses the newly added components.
So, I need some alternative solution to execute this.
You can add Response Assertion (or multiple) with Pattern to test filled with a variable as ${testAssert1} and set the variable by default as empty, for example
Put in User Defined Variables name testAssert1 with empty value.
Your assertion(s) will pass until you on run time set the variable with a different value, for example using User Parameters Pre Processor.
In the set of my TeamCity configurations, I decided to make something like an aging test*. And run a single configuration for a 100 times.
Can I make in a few simple clicks?
*aging test - test that is showing, that due time/aging, results will not be changed.
As of now, this is not possible from UI. If you run one build configuration few times without any changes, they will be merged and only 1 will be executed. If you want to run 100, you have to trigger them one by one, after the previous one finished executing.
But the better solution is to trigger builds from script using REST API (for more details see the documentation here), if builds have different values in custom parameters they all will be put in the queue.
HOW: Define a dummy custom parameter, and trigger the build from script within a loop. Pass the value of iterating variable as parameter value. So, TeamCity will think those are different builds and execute all of them.
Using ruby / cucumber, I know you can explicitly call a fail("message"), but what are your other options?
The reason I ask is that we have 0... I repeat, absolutly NO control over our test data. We have cucumber tests that test edge cases that we may or may not have users for in our database. We (for obvious reasons) do not want to throw away the tests, because they are valuable; however since our data set cannot test that edge case, it fails because the sql statement returns an empty data set. Right now, we just have those tests failing, however I would like to see something along the lines of "no_data" or something like that if the sql statement returns an empty data set. So the output would look like
Scenarios: 100 total (80 passed, 5 no_data, 15 fail)
I am willing to use the already implemented "skipped" if there is a skip("message") function.
What are my options so we can see that with the current data, we just don't have any test data for those tests? making these manual tests is also not an option. They need to be run ever week with our automation, but somehow separate from the failures. Failure means defect, no_data found means it's not a testable condition. It's the difference between a warning: we have not tested this edge case, and Alert: broken code.
You can't invoke 'skipped', but you can certainly call pending with or without an error message. I've used this in a similar situation to yours. Unless you're running in strict mode then having pending scenarios won't cause any failures. The problem I encountered was that occasionally a step would get mis-spelled causing cucumber to mark that as pending, since it was not matching a step definition. That then became lost in the sea of 'legitimate' pending scenarios and was weeks before we discovered it.
I'm testing a Test Case with a few steps in Microsft Test Manager.
When I run this Test Case, I want to execute only a few steps and then assign another tester to this Test Run.
E.g.
I have three steps. The first two steps are for me to test.
After those two steps, I want to stop testing and assign another tester so that he can test the third step.
But I can't find a way to stop testing, and assign a new user to this Test Case.
Does anyone know if this is possible?
Thanks!
This definitely cannot be done. When you run a Test Case a new Test Run is created and stored in the tfs database. The steps executed for this run and their result, comments, attachments e.t.c. are saved and cannot be edited.
From a test point of view, I think that even if you could do this, you shouldn't. Every test case should be as simple as possible so everyone can execute it. If you really need this, perhaps you should split the test case to two different tests, and the second one will have the first as prerequisite.
I'm testing a large BizTalk system using Visual Studio Load Test. Load Test to pushes messages into MQ, these are picked up by BizTalk and then processed.
Rather than having the test finish (and all performance counters ending) as soon as Visual Studio has finished injecting messages to MQ, I want the test to end if and only if some condition is met (in my case if (SELECT COUNT(*) FROM BizTalkMsgBoxDb.dbo.Spool) == 4).
I can see a bunch of ways to run stuff after the test is complete, but no obvious way to extend the test and continue monitoring unless some user-defined exit condition is met.
Is this possible, or if not, does anyone have an idea for a good work-around/hack to achieve this?
You'll want to write a custom load test plugin. Details begin at this URL: http://msdn.microsoft.com/en-us/library/ms243153.aspx
The plugin can manipulate the scenario, extending the duration of the test until your condition is met.
I imagine you want to keep the load test running after queueing up a bunch of requests in order to continue to monitor the performance while the requests are processed. Although we can't control the load test duration, there is a way to achieve this.
Don't limit the test duration: Set the load test duration (or number of iterations) to a very large value -- larger than you anticipate (or know) it will take for the end condition to be satisfied.
Limit the scenario that queues up requests: In the load test scenario properties, in the Options section, set the Maximum Test Iterations so that the user load will drop to zero after sending the desired number of requests. If setting an iteration limit is not possible for some reason, you can instead write a load test plugin that sets the user load to zero in a specified scenario after a certain amount of test time has elapsed.
Check for end condition: Write a web test plugin that checks the database for your end condition. Attach this plugin to a new webtest in a new scenario and set Think Time Between Test Iterations on the scenario so that the test runs only as often as needed (60 seconds?). When the condition is reached, the plugin should write a predetermined value into the user context (the user context is accessible in the web test context as $LoadTestUserContext, and is only available in a load test, not when running a webtest standalone).
Abort the test: Write a load test plugin that looks for the flag value in the user context in the TestFinished event. When the value is found, the plugin calls LoadTest.Abort().
There is one minor disadvantage to this method: the test state is marked as Aborted in the results database.
At time of writing there is (still) no way to extend the duration of the test using a custom load test plugin, nor by having a virtual user type that refused to exit, nor by locking the close-down period of the test and preventing it from exiting that way.
The only way we managed to do something like this was to directly manipulate the LoadTest database and inject performance counter data in afterwards from log files, but this is neither smart nor recommended.
Oh well..