I want to use Cucumber to test my application which takes snapshots of external websites and logs changes.
I already tested my models separatly using RSpec and now want to make integration tests with Cucumber.
For mocking the website requests I use VCR.
My tests usually follow a similar pattern:
1. Given I have a certain website content (I do this using VCR cassettes)
2. When I take a snapshot of the website
3. Then there should be 1 "new"-snapshot and 1 "new"-log messages
Depending if the content of the website changes, a "new"-snapshot should be created and a "new"-log message should be created.
If the content stays the same, only a "old"-log message should be created.
This means, that the the application's behaviour depends on the current existing snapshots.
This is why I would like to run the different scenarios without resetting the DB after each row.
Scenario Outline: new, new, same, same, new
Given website with state <website_state_1>
When I take a snapshot
Then there should be <1> "new"-snapshot and <1> "old"-log messages and <1> "new"-log messages
Examples:
| state | snapshot_new | logmessages_old | logmessages_new |
| VCR_1 | 1 | 0 | 1 |
| VCR_2 | 2 | 0 | 2 |
| VCR_3 | 2 | 1 | 2 |
| VCR_4 | 2 | 2 | 2 |
| VCR_5 | 3 | 2 | 3 |
However, the DB is resetted after each scenario is run.
And I think that scenario outline was never intended to be used like this. Scenarios should be independent from each other, right?
Am I doing something wrong trying to solve my problem in this way?
Can/should scenario outline be used for that or is there another elegant way to do this?
J.
Each line in the Scenario Outline Examples table should be considered one individual Scenario. Scenarios should be independent from each other.
If you need a scenario to depend on the system being in a certain state, you'll need to set that state in the Given.
Related
I started to read famous Martin Fowler book (Patterns of Enterprise Application Architecture)
I have to mention that I am reading the book translated into my native language so it might be a reason of my misunderstanding.
I found their definitions (back translation into English):
Response time - amount of time to process some external request
Latency - minimal amount of time before getting any response.
For me it is the same. Could you please highlight the difference?
One way of looking at this is to say that transport latency + processing time = response time.
Transport latency is the time it takes for a request/response to be transmitted to/from the processing component. Then you need to add the time it takes to process the request.
As an example, say that 5 people try to print a single sheet of paper at the same time, and the printer takes 10 seconds to process (print) each sheet.
The person whose print request is processed first sees a latency of 0 seconds and a processing time of 10 seconds - so a response time of 10 seconds.
Whereas the person whose print request is processed last sees a latency of 40 seconds (the 4 people before him) and a processing time of 10 seconds - so a response time of 50 seconds.
As Martin Kleppman says in his book Designing Data Intensive Applications:
Latency is the duration that a request is waiting to be handled - during which it is latent, awaiting service. Used for diagnostic purposes ex: Latency spikes
Response time is the time between a client sending a request and receiving a response. It is the sum of round trip latency and service time. It is used to describe the performance of application.
This article is a good read on the difference, and is best summarized with this simple equation,
Latency + Processing Time = Response Time
where
Latency = the time the message is in transit between two points (e.g. on the network, passing through gateways, etc.)
Processing time = the time it takes for the message to be processed (e.g. translation between formats, enriched, or whatever)
Response time = the sum of these.
If processing time is reasonably short, which in well designed systems is the case, then for practical purposes response time and latency could be the same in terms of perceived passage of time. That said, to be precise, use the defined terms and don't confuse or conflate the two.
I differentiate this using below example,
A package has been sent from A-B-C where
A-B took 10 sec, B (processing) took 5 sec, B-C took 10 sec
Latency = (10 + 10) sec = 20 sec
Response time = (10 + 5 + 10) sec = 25 sec
Latency
The time from the source sending a packet to the destination receiving it
Latency is the time it takes for a message, or a packet, to travel from its point of origin to the point of destination. That is a simple and useful definition, but it often hides a lot of useful information — every system contains multiple sources, or components, contributing to the overall time it takes for a message to be delivered, and it is important to understand what these components are and what dictates their performance.
Let’s take a closer look at some common contributing components for a typical router on the Internet, which is responsible for relaying a message between the client and the server:
Propagation delay
Amount of time required for a message to travel from the sender to receiver, which is a function of distance over speed with which the signal propagates.
Transmission delay
Amount of time required to push all the packet’s bits into the link, which is a function of the packet’s length and data rate of the link.
Processing delay
Amount of time required to process the packet header, check for bit-level errors, and determine the packet’s destination.
Queuing delay
Amount of time the packet is waiting in the queue until it can be processed.
The total latency between the client and the server is the sum of all the delays just listed
Response Time
Total time taken between the packet to send and receive the packet from the receiver
Q : Could you please highlite the difference?
Let me start using a know-how from professionals in ITU-T ( former CCITT ), who have for decades spent many thousands of man*years of efforts on the highest levels of professional experience, and have developed an accurate and responsible methodology for measuring both:
What have the industry-standards adopted for coping with this ?
Since early years of international industry standards ( well, as far as somewhere deep in 60-ies ), these industry-professionals have created concept of testing complex systems in a repeatable and re-inspectable manner.
System-under-Test (SuT), inter-connected and inter-acting across a mediation service mezzo-system
SuT-component-[A]
|
+------------------------------------------------------------------------------------------------------------[A]-interface-A.0
| +------------------------------------------------------------------------------------------------[A]-interface-A.1
| |
| | SuT-component-[B]
| | |
| | +-------------------------[B]-interface-B.1
| | | +---------[B]-interface-B.0
| | ???????????????? | |
| | ? mezzo-system ? | |
+-----------+ ???????????????? +---------------+
| | ~~~~~~~~~~~~~~~~~~~~~~~~ ??? ... ... ??? ~~~~~~~~~~~~~~~~~~~~~~~~ | |
| | ~~<channel [A] to ???>~~ ??? ... ... ??? ~~<channel ??? to [B]>~~ | |
| | ~~~~~~~~~~~~~~~~~~~~~~~~ ??? ... ... ??? ~~~~~~~~~~~~~~~~~~~~~~~~ | |
+-----------+ ???????????????? +---------------+
|
No matter how formal this methodology may seem, it brings both clarity and exactness, when formulating ( the same when designing, testing and validating ) the requirements closely and explicitly related to SuT-components, SuT-interfaces, SuT-channels and also constraints for interactions across exo-system(s), including limitations of responses to an appearance of any external ( typically adverse ) noise/disturbing events.
At the end, and to the benefit of clarity, all parts of the intended SuT-behaviour could be declared against a set of unambiguously defined and documented REFERENCE_POINT(s), for which the standard defines and documents all properties.
A Rule of Thumb :
The LATENCY, most often expressed as a TRANSPORT-LATENCY ( between a pair of REFERENCE_POINTs ) is related to a duration of a trivial / primitive event-propagation across some sort of channel(s), where event-processing does not transform the content of the propagated-event. ( Ref. memory-access latency - does not re-process the data, but just delivers it, taking some time to "make it" )
The PROCESSING is meant as that kind of transforming an event, in some remarkable manner inside a SuT-component.
The RESPONSE TIME ( observed on REFERENCE_POINT(s) of the same SuT-component ) is meant as a resulting duration of some kind of rather complex, End-to-End transaction processing, that is neither a trivial TRANSPORT LATENCY across a channel, nor a simplistic in-SuT-component PROCESSING, but a some sort of composition of several ( potentially many ) such mutually interacting steps, working along the chain of causality ( adding random stimuli, where needed, for representing noise/errors disturbances ). ( Ref. database-engine response-times creep with growing workloads, right due to increased concurrent use of some processing resources, that are needed for such requested information internal retrieval, internal re-processing and for final delivery re-processing, before delivering resulting "answer" to the requesting counter-party )
|
| SuT_[A.0]: REFERENCE_POINT: receives an { external | internal } <sourceEvent>
| /
| / _SuT-[A]-<sourceEvent>-processing ( in SuT-[A] ) DURATION
| / /
| / / _an known-<channel>-transport-LATENCY from REFERENCE_POINT SuT-[A.1] to <mezzo-system> exosystem ( not a part of SuT, yet a part of the ecosystem, in which the SuT has to operate )
| / / /
| / / / _a mezzo-system-(sum of all)-unknown-{ transport-LATENCY | processing-DURATION } duration(s)
| / / / /
| / / / / _SuT_[B.1]: REFERENCE_POINT: receives a propagated <sourceEvent>
| / / / / /
| / / / / / _SuT_[B.0]: REFERENCE_POINT: delivers a result == a re-processed <sourceEvent>
| / / / / | / | /
|/ /| /................ / |/ |/
o<_________>o ~~< chnl from [A.1] >~~? ??? ... ... ??? ?~~<chnl to [B.1]>~~~~~~? o<______________>o
| |\ \ \
| | \ \ \_SuT-[B]-<propagated<sourceEvent>>-processing ( in SuT-[B] ) DURATION
| | \ \
| | \_SuT_[A.1]: REFERENCE_POINT: receives \_an known-<channel>-transport-LATENCY from <mezzo-system to REFERENCE_POINT SuT_[B.1]
| |
| | | |
o<--------->o-----SuT-test( A.0:A.1 ) | |
| | | |
| | o<-------------->o---SuT-test( B.1:B.0 )
| | | |
| o<----may-test( A.1:B.1 )-------------------------------------------->o |
| | exo-system that is outside of your domain of control, | |
| indirectly, using REFERENCE_POINT(s) that you control |
| |
| |
o<-----SuT-End-to-End-test( A.0:B.0 )------------------------------------------------------------->o
| |
Using this ITU-T / CCITT methodology, an example of a well defined RESPONSE TIME test would be a test of completing a transaction, that will measure a net duration between delivering a source-event onto REFERENCE_POINT [A.0] ( entering SuT-component-[A] ) and waiting here until the whole SuT delivers an answer from any remote part(s) ( like a delivery from [A]-to-[B], plus a processing inside a SuT-component-[B] and an answer delivery from [B]-back-to-[A] ) until an intended response is received back on a given REFERENCE_POINT ( be it the same one [A.0] or another, purpose-specific one [A.37] ).
Being as explicit as possible saves potential future mis-understanding ( which the international industry standards fought to avoid since ever ).
So a requirement expressed like:
1) a RESPONSE_TIME( A.0:A.37 ) must be under 125 [ms]
2) a net TRANSPORT LATENCY( A.1:B.1 ) ought exceed 30 [ms] in less than 0.1% cases per BAU
are clear and sound ( and easy to measure ) and everybody interested can interpret both the SuT-setup and the test-results.
Meeting these unambiguous requirements qualify a such defined SuT-behaviour to safely become compliant with an intended set of behaviours, or let professionals to cheaply detect, document and disqualify those, who do not.
I Want to execute Background only once in each cucumber feature files for multiple scenarios. how can i do that in step files?
Feature: User can verify...........
Background:
Given Enter test data for a specific logic
Scenario: Verify ......... 1
When A1
And B1
Then C1
Scenario: Verify ......... 2
When A2
And B2
Then C2
Scenario: Verify ......... 2
When A3
And B3
Then C3
Tests should be isolated. That is the way Cucumber is designed, and there's a very good reason for it. I would strongly urge against it unless you absolutely have to.
If you have something that needs to be executed before your entire test suite, consider a #BeforeAll hook.
That said, I've come across this before. We had tests which would start a process that took a long time (such as provisioning a VM, which would take 10 minutes a time..), but could be skipped in other tests if it had already been done.
So you may want to design steps along the lines of..
"Given that .. X has been done"
and in the step detect whether X is done or not, and if not, then do X. If it has, then skip it. For example, say creating users is a process that takes absolutely ages. Then we could do..
Given that user "Joe Bloggs" has been created
The step definition would first try to determine if Joe existed, and then if they didn't, create the user. You have an initial start up problem, but then other tests will be able to safely assume Joe exists.
WHY YOU SHOULD NOT DO THIS
If you do this, there's a high probability your tests will conflict with one another.
Say you have lots of tests that use the Joe Bloggs user. Maybe some will delete him from the system. Some might temporarily deactivate the user, add roles, change their name.. all sorts of things. All tests assume certain things about the system they're testing, and you are intentionally harming your tests assumptions about the environment.
It's particularly bad if you are running parallel tests. Maybe your system has a restriction that only one person can log in at a time as Joe. Or every test is changing loads of things about Joe, and no test can assume anything about the state of the Joe user. Then you'll be in a huge mess.
The best solution is often to create brand new data for each test you run. Open up those APIs and create disposable data for each test run. Here's a good blog post about it: https://opencredo.com/test-automation-concepts-data-aliases/
Background is designed to run each time before each scenario. Its not good and standard as well to hack Background.
If you want your background to be run only once. You can add condition with an instance variable ex, i==0 then execute the logic and increment i at the end of the method.
For the next scenario, i value is 1 which is not equal to 0,the it won't execute the logic.
We can have both scenario and scenario outline in single feature file. With this scenario will run only once and scenario outline would run based on the data given in Example table
We had similar problem and couldn't find any solution for background for multiple scenario. Background is design to run for all scenario after every scenario it will run Background. If you have examples in scenario in this case it will run after each examples.
We had to option to fix this problem.
1) Used #BeforeClass Annotation of junit
2) Create setup scenario and It will always execute in first place.
For example : In API testing, You take login once and used that session every time to access other API
Feature:Setup Data
Given Customer logs in as System Admin
Scenario: Verify ......... 1 When A1 And B1 Then C1
Scenario: Verify ......... 2 When A2 And B2 Then C2
Scenario: Verify ......... 2 When A3 And B3 Then C3
After first scenario it will execute all scenario and You don't need to use background.
I would say using background every time it must be part of business requirement else it will create unwanted test data and load on testing environment and may result in slow down testing execution time.
Please let me know if you find other solution.
Running some steps (Background) before each set of scenarios or a Scenario Outline can also be achieved by creating a tagged #Before method and passing a Scenario object as parameter. Within the before method, execute your logic only if the scenario name is different as compared to the last scenario.
Below is a how you can do it:
Feature:Setup Data Given Customer logs in as System Admin
#BeforeMethodName
Scenario Outline: Verify ......... 1
When <Variable1> And <Variable2>
Then <Variable3>
Examples:
| Variable1 | Variable2 | Variable3 |
| A1 | B1 | C1 |
| A2 | B2 | C2 |
| A3 | B3 | C3 |
| A4 | B4 | C4 |
#BeforeMethodName
Scenario Outline: Verify ......... 2
When <Variable1> And <Variable2>
Then <Variable3>
Examples:
| Variable1 | Variable2 | Variable3 |
| X1 | Y1 | Z1 |
| X2 | Y2 | Z2 |
| X3 | Y3 | Z3 |
| X4 | Y4 | Z4 |
And define the #BeforeMethodName as below:
private static String scenarioName = null;
public className BeforeMethodName(Scenario scene) {
if(!scene.getName().equals(scenarioName)) {
// Implement your logic
scenarioName = scene.getName()
}
return this;
}
This way BeforeMethodName will be called before each scenario but will execute the logic only once per Scenario Outline.
Old issue, but adding incase others find this.
As has been mentioned, cucumber should only be used to structure your code.
You can use tagged hooks to create items which are used on a sub-set of tests. Furthermore you could isolate the code into Helpers, and then conditionally call these helpers inside your ruby steps.
Probably need a bit more clarity to make a judgement
I've run into a scenario which requires nesting of the MVP pattern. It is probably best to explain using a visual example:
------------------------------
| [View] |
| | |
| +----[Presenter] |
| | |
| +------[Model] |
|____________________________|
|
+----[View]
|
+----[Presenter]
|
+------[Model]
This is how the two MVP layers should interact. My question is in regards to the connection between the two. I can envision several ways to connect the two:
The Presenter from Tier 1 is connected to the View for Tier 2.
The Model from Tier 1 is connected to the View for Tier 2.
The View for Tier 2 IS the Tier 1 (the View contains references to the M, V, & P of Tier 1).
The Presenter or Model from Tier 1 IS the View for Tier 2.
Which of these is correct, if any?
Then there is the question of HOW they are connected. Should the Tier 2 View have a reference to one of the objects from Tier 1? Should the interaction be purely event based? A combination of the two (if so, where should the reference lie?)?
I'm pretty new to using these types of patterns, so any insight would be appreciated.
I've recently read the article CQRS à la Greg Young and am still trying to get my head around CQRS.
I'm not sure about where input validation should happen, and if it possibly has to happen in two separate locations (thereby violating the Don't Repeat Yourself rule and possibly also Separation of Concerns).
Given the following application architecture:
# +--------------------+ ||
# | event store | ||
# +--------------------+ ||
# ^ | ||
# | events | ||
# | v
# +--------------------+ events +--------------------+
# | domain/ | ---------------------> | (denormalized) |
# | business objects | | query repository |
# +--------------------+ || +--------------------+
# ^ ^ ^ ^ ^ || |
# | | | | | || |
# +--------------------+ || |
# | command bus | || |
# +--------------------+ || |
# ^ |
# | +------------------+ |
# +------------ | user interface | <-----------+
# commands +------------------+ UI form data
The domain is hidden from the UI behind a command bus. That is, the UI can only send commands to the domain, but never gets to the domain objects directly.
Validation must not happen when an aggregate root is reacting to an event, but earlier.
Commands are turned into events in the domain (by the aggregate roots). This is one place where validation could happen: If a command cannot be executed, it isn't turned into a corresponding event; instead, (for example) an exception is thrown that bubbles up through the command bus, back to the UI, where it gets caught.
Problem:
If a command won't be able to execute, I would like to disable the corresponding button or menu item in the UI. But how do I know whether a command can execute before sending it off on its way? The query side won't help me here, as it doesn't contain any business logic whatsoever; and all I can do on the command side is send commands.
Possible solutions:
For any command DoX, introduce a corresponding dummy command CanDoX that won't actually do anything, but lets the domain give feedback whether command X could execute without error.
Duplicate some validation logic (that really belongs in the domain) in the UI.
Obviously the second solution isn't favorable (due to lacking separation of concerns). But is the first one really better?
I think my question has just been solved by another article, Clarified CQRS by Udi Dahan. The section "Commands and Validation" starts as follows:
Commands and Validation
In thinking through what could make a command fail, one topic that comes up is validation. Validation is
different from business rules in that it states a context-independent fact about a command. Either a
command is valid, or it isn't. Business rules on the other hand are context dependent.
[…] Even though a command may be valid, there still may be reasons to reject it.
As such, validation can be performed on the client, checking that all fields required for that command
are there, number and date ranges are OK, that kind of thing. The server would still validate all
commands that arrive, not trusting clients to do the validation.
I take this to mean that — given that I have a task-based UI, as is often suggested for CQRS to work well (commands as domain verbs) — I would only ever gray out (disable) buttons or menu items if a command cannot yet be sent off because some data required by the command is still missing, or invalid; ie. the UI reacts to the command's validness itself, and not to the command's future effect on the domain objects.
Therefore, no CanDoX commands are required, and no domain validation logic needs to be leaked into the UI. What the UI will have, however, is some logic for command validation.
Client-side validation is basically limited to format validation, because the client side cannot know the state of the data model on the server. What is valid now, may be invalid 1/2 second from now.
So, the client side should check only whether all required fields are filled in and whether they are of the correct form (an email address field must contain a valid email address, for instance of the form (.+)#(.+).(.+) or the like).
All those validations, along with business rule validations, are then performed on the domain model in the Command service. Therefore, data that was validated on the client may still result in invalidated Commands on the server. In that case, some feedback should be able to make it back to the client application... but that's another story.
I'm testing an app that's basically a message-handling application - a message comes in, it's content is analysed, then it's sent somewhere else for processing. The app isn't built in Ruby.
As you might imagine, the main testing approah consists of generating a whole bunch of different types of (quite complex) messages, loading them into the app, waiting a few seconds then ensuring that they get sent to the correct place.
Functionally, the testing's going well, but I've currently got all of the test messages & desired message destinations defined in Ruby code - I'd like to move them to either a YAML file, or (second choice) a database. I'd prefer to use a YAML file over a database because it's easier to version control, and for non-technical testers to edit the message content directly.
Is there a "recommended" way to implement this sort of data management in Cucumber? It sort of smells like a fixtures approach makes sense, but fixtures to me have always involved populating a database from a file and then using the DB for testing, and I'm not 100% sure this is the best/simplest fit for this particular problem.
I believe what you will be most happy with is a Scenario Outline. You could perhaps create a yaml file an load it from a step, but that would not make a very useful test output. What you (I think) would really like is to see each message and its destination sorted by weather it passed or failed. The example below is for Failed Logins, but it gets the point accross.
Scenario Outline: Failed Login
Given I am not authenticated
When I go to "/login"
And I fill in "login" with "<mail>"
And I fill in "password" with "<password>"
And I press "Log In"
Then the login request should fail
Then I should see an error message
Examples:
| mail | password |
| not_an_address | nil |
| not#not | 123455 |
| 123#abc.com | wrong_paasword |
Each Example will turn green, red or yellow depending on whether it worked, failed or was pending.