I'm relatively inexperienced when it comes to Unit Testing/Automated Testing, so pardon the question if it doesn't make any sense.
The current code base I'm working on is so tightly coupled that I'll need to refactor most of the code before ever being able to run unit tests on it, so I read some posts and discovered Selenium, which I think is a really cool program.
My client would like specific automated tests to run every ten minutes on our production server to ensure that our site is operational, and that certain features/aspects are running normally.
I've never really thought to run tests against a production server because you're adding additional stress to the site. I always thought you would run all tests against a staging server, and if those work, you can just assume the prouction site is operational as long as the hosting provider doesn't experience an issue.
Any thoughts on your end on testing production code on the actual production server?
Thanks a lot guys!
Maybe it would help if you thought of the selenium scripts as "monitoring" instead of "testing"? I would hope that every major web site out there has some kind of monitoring going on, even if it's just a periodic PING, or loading the homepage every so often. While it is possible to take this way too far, don't be afraid of the concept in general. So what might some of the benefits of this monitoring/testing to you and your client?
Somehow not all the best testing in the world can predict the odd things users will do, either intentionally or by sheer force of numbers (if 1 million monkeys on typewriters can write Hamlet, imagine what a few hundred click happy users can do? Pinging a site can tell you if it's up, but not if a table is corrupted and a report is now failing, all because a user typed in a value with a umlaut in it.
While your site might perform great on the staging servers, maybe it will start to degrade over time. If you are monitoring the performance of those selenium tests, you can stay ahead of slowness complaints. Of course as you mentioned, be sure your monitoring isn't causing problems either! You may have to convince your client that certain test are appropriate to run every X minutes, and others should only be run once a day, at 3am.
If you end up making an emergency change to the live site, you'll be more confident knowing that tests are running to make sure everything is ok.
I have worked on similar production servers from long time. From my experience, i can say is that, Always it is better to test our change changes/patches in Stage environment and just deploy it, in production servers. This is because, both Staging and Production environments are alike, except the volume of data.
If really required, it would be ok, to run few tests on Production servers, once the code/patch is installed. But it is not recommended/good way to run the tests always on the production server.
My suggestion would be to shadow the production database down to a staging/test environment on a nightly basis and run your unit tests there nightly. The approach suggested by the client would be good for making sure that new data introduced into the system didn't cause exceptions within the system, but i do not agree with doing this in production.
Running it in a staging environment would give you the ability to evaluate features as new data flows into the system without using the production environment as a test bed.
[edit] to make sure the site is up, you could write a simple program which pings it every 10 minutes rather than running your whole test suite against it.
What will change in production environment that you would need to run automated tests? I understand that you may need monitoring and alerts to make sure the servers are up and running.
Whatever the choice, whether it be a monitoring or testing type solution, the thing that you should be doing first and foremost for your client is warning them. As you have alluded to, testing in production is almost never a good idea. Once they are aware of the dangers and if there are no other logical choices, carefully construct very minimal tests. Apply them in layers and monitor them religiously to make sure that they aren't causing any problems to the app.
I agree with Peter that this sounds more like monitoring than testing. A small distinction but an important one I think. If the client's requirements relate to Service Level Agreements then their requests do not sound too outlandish.
Also, it may not be safe to assume that if the service provider is not experiencing any issues that the site is functioning properly. What if the site becomes swamped with requests? Or perhaps SQL that runs fine in test starts causing issues (timeouts, blocking etc.) with a larger production database?
Related
I have taken as an example for learning and gathered some information about tools, objectives,scenarios, but I need your inputs. Please assist me.
I am new to Performance testing and would like to test the following website www.volkswagen.co.nz
Can you tell me, what are need to be tested? What are the scenarios and activities for each scenario? What metrics do I need to add? Which is the best and free tool for testing it? How to test if it is deployed in cloud like AWS?
Please let me know, Thanks in advance.
Performance testing needs,
Identify critical/heavy/important scenario in your webapp (irrespective of deployment cloud/standalone)
Identify service level agreements in terms of response times, throughput, latency etc.
Identify workload model i.e. how much user load application is expecting. this should be as fine grained as possible (avg users per transaction/workflow at a point of time)
Identify tools (JMeter is freeware and best but if you can afford paid then look at loadrunner, neoload etc.)
Record the script for workflows and parameterise and correlate.
generate test setup for load test and execute the load test.
monitor system utilization, collect metrics like response time, throughput, error rate, latency etc.
This all comes in load testing. For more you can read http://www.guru99.com/performance-testing.html
I am new to Performance testing and would like to test the following website www.volkswagen.co.nz
That is a recipe for disaster. No one new should be allowed to work on their own without a full period of training and internship with a master in the field. This is true of stone masons, electricians, plumbers, barbers, accountants, engineers and physicians. And it is most certainly true of performance testers/engineers.
There are dozens of foundation skills you need to master before you touch any tool, open source or otherwise. Until you show mastery of those items along with tool mechanics for your tool you should not be allowed to test any website, particularly a production website. And, if you don't work for this company what you are engaging in is a denial of service attack and could leave you with exposed legal liability.
I strongly agree with James on this one.
Do not touched the site if:
it's not yours
not sure what you are doing
the owner gave you explicit (and sounds like irresponsible) permission
don't know or don't have the support to restore the environment into a working state
If you do work for the company then you need to have a test environment first, a playground where you can mess around and nobody would mind if you take it down.
Firstly get information from the business on which use cases needs to
be tested.
Get response times target for user actions and for environments utilisation.
Get response time targets for environments utilisation: define environment monitoring tactics.
Found a tool that can fit for purpose: Jmeter, Gattling,etc, lot's of free ones available.
Get a test environment, preferably similar scale to production
Create scripts to cover critical use cases
Comply scripts into scenarios
Create a reporting framework
Kick off monitoring
Kick off scenario
Collect and analyse results
Be mindful of the free editions of load testing tools: they tend to be easy to use at first but soon as you start to outgrow it it can cost a fortune and more often then not it's hard to port scripts/scenarios to another tool.
I'm setting up my front-end application to use continuous integration in CircleCI. Unit tests work fine, but end-to-end tests are not.
The problem is that it requires the backend (API) server to be running, and ours is in another completely different application. So, what is the best way to setup this backend server (thinking about CI)?
I thought about uploading it on heroku, but then I'd have to keep manually updating the code via git. Another option was to download the code to the CI VM and run the server directly there, but it is just too much work (install ruby, postgres, gems...), and it doesn't seem in no way the best option.
Have anyone passed through the same situation? How do you guys usually deal with this kind of situations?
I ended up doing everything inside the CI. I made some custom scripts that configure the backend project every time the test suite is ran. Also, I cached the folder with the backend code and the gems (which was taking ~2min to install).
The configuring part now adds ~20 seconds to the total time, so it wasn't a big deal. Although I still think that this is probably not the best way to do this, it has some advantages, such as not worrying about updating the backend code (it pulls from master automatically) or its database (it runs rake db:reset after updating the code).
Assuming the API server is running somewhere, configure the front-end application to point there while in the test/CI environment, at least to start out. If there are multiple API environments, choose the one the most closely matches the front-end environment (e.g. dev, staging, etc).
It gets more complicated if/when you need to run the e2e tests each time the API is built or match up specific build versions of the front-end and the API. In that case you will have to run the API server as part of the test.
On our applications we have a lot of functional tests through selenium.
We understand that it is a good practice to have the server where the tests are ran as similar as possible to the production servers, and we try to follow it as much as possible.
But that is very hard to achieve in 100%, so we have a different settings file for our server for some changes that we want in the staging environment (for example, we opt to turn e-mail sending off because of the additional required architecture).
In fact, lots of server frameworks recommend having an isolated front-controller (environment) for testing to easily achieve this small changes.
By default, most frameworks such as ours recommend that their testing environment should have its cache turned off. WHY?
If we want to emulate production as much as possible, what's the possible advantage of having the server's cache turned off when performing functional tests? There can be bugs that are only found with the cache on, and having it on might also have the benefit of accelerating our tests execution!
Don't we just need to make sure that the cache is cleared before starting a new batch of functional tests, the same way we clear the cache when deploying a new version to production?
A colleague of mine suggests that the reason for this is could be that cache can generate false-positives, errors that are not caused by badly implemented features (that are the main target of those tests), but of the cache system itself... but even if those really happen (I suppose it depends on how advanced is the way the cache is used), why would they be false-positives?
To best answer this question I will clarify some points.
(be aware that this is based on my experience)
Integration tests using the browser are typically "Black Blox Tests" , which means that they are made without knowledge of the code. That is, without knowing whether the cache is being used or not.
These tests are usually designed based on certain tasks that are performed during the normal use of the system. But, these tasks are chosen for automation depending on certain conditions of use (mainly reusability, and criticality/importance but also the cost of implementation). So most of the times, we will not need/wont to test caching behaviour.
By convention, a test (any) must be created with a single purpose and have the less possible dependencies. Why?
When the test fails , we can quickly find the source of the failure.
Smaller tests are easier to extend, fix, remove...
We do not spend too much time, first debugging the test code and then
debugging the system code.
Integration testing should follow this convention.
Answering the question:
If we want to check a particular task, we must isolate it as possible.
For example, if we want to verify that the user correctly logs in, we have to delete the cookies to be sure that they do not influence the result (because they may). If on the other hand, we want to test the cookies we have to somehow use an environment where they are not deleted.
so, in short:
If there is need to test the caching behaviour then we need to create an "isolated" environment where this is possible.
The usual integration tests purpose is to test the functionality, so the framework default value it's to have the cache disabled.
This does not means that we shouldn't create our own environment to test the caching behaviour.
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 1 year ago.
The community reviewed whether to reopen this question 1 year ago and left it closed:
Original close reason(s) were not resolved
Improve this question
In the company where I work, we have 2 environments: test and production. We are not currently starting a new environment, because of cost.
Here is the procedure we follow: business makes a feature request, development makes it happen and deploys it on test environment. Then business tests it (UAT), and if it's OK, the feature will be included into next production deployment.
The problem is best practices for test DB. Developers treat test environment as their playground, and sometimes they reset the DB to initial state for testing purposes. On the other hand, business people think that test DB must be stable, and should not be reset. We would like to resolve this issue, and decide if test environment should belong to development team or business team. (Developers don't want business to put their nose in test env., but business team is paying for servers.)
What is the best practice about environments? Can you recommend an article about this?
I've worked in many companies, each one with a different set of environments, the one I have like the most had 5 environments:
1) Local: Basically your machine. Here is where you code and test your changes before even asking a peer for review.
2) Dev: If for some reason you cannot test your code locally (Dependencies issues mainly: "My code has neves compiled in my machine but it compiles perfectly in Jenkins/Bamboo/Travis"), then you push your changes to your feature branch in Git and make Bamboo compile it and deploy it to a dev server where you can test (you are still not sure if it will work, so no peer review so far).
3) Staging: You think your code work and you love how it looks like. You create a Pull Request in order for you peers to take a look at it before it gets merged to the master branch. Here they make comments and you fix possible issues, since you are more sure about your changes you make Bamboo deploy it to the Staging environment where more "stable" code lives and more realistic data is stored in whatever database is there. Once deployed another developer/tester can check your changes actually work and make you a "QA Sign off in Staging Env."
4) Stable: Ok, now you are ultimate sure your changes work since you deployed to Staging and nothing got broken. You merge your branch to master and Bamboo compiles master and deploys to another sets of servers in a Stable Environment (no one else should merge to master until you finish your deployment to Production, to avoid a merge of not related merges). This environment should be a replica from Production, data, code and server conditions. Here is where you show your changes to your manager, product owner or the person in charge to validate your work before sending it to production. You get the final approval, everything is good, you are sweaty, you have worked for 30 days in a row to get this change done, your wife divorced you but you are very confident it works perfectly.
5) Production: Where the clients connect to consume the company services, or the final build of your software to send to customers. With a few clicks in Bamboo you make it deploy to Production servers or compile the final build. It is green, everything seems to be ok. You check Splunk looking for errors, everything is good, life is good, you drink another sip of coffee before leaving, you drive home and sleep all weekend with your dog by your side.
It's a happy ending because having so many "Test" environments ensure quality, no change gets to production until EVERYBODY (not just you) are completely sure that changes is working.
At our company there are two databases too, a test and a production database. The test database is mainly used for testing by developers but sometimes for business tests too. This database is refreshed daily using an actual copy of the production database. So this database can be both a playground and a serious testing database. But a third, development, database is the best option. We had one, but it is broken at the moment. But when you get one of those, you should make sure it is refreshed often enough. When developers use it as a playground, it will stray away from the production environment, and its data will be both old and currupt. Because of this, developers won't be able to test well themselves. So make sure you refresh this database periodically (maybe daily too, or at least once a week).
If possible, give each developer their own database on their local machine. That way they can do whatever they want with it without affecting anyone else. This should significantly decrease the desire to play with the test database, providing a more stable environment for business UAT.
I believe in order to establish an environment strategy that supports all ALM/SDLC activities 4 requirements should exist:
1) Development environment: to allow Dev to play around with new code/concepts and typically unit test with some basic integration testing using stubs and drivers. This environment should have loose change control procedures and would typically not be anywhere near the same scale as production. This is where the Dev team can build and tear down setups as required.
2) Interop environment: where integration of systems can be further tested and increased capability for non-functional testing I.e might be a resilient env with greater scalability than Dev. I'd see this environment having tighter change control and management. Test would perform integration and system testing in this environment.
3) Reference Architecture: This is what some might call pre-prod but is essentially the same as production in terms of scale and resilience. This would have change control and management procedures akin to prod. This env would support further test activities especially full scale performance testing, failover as well as operational fault triage and maintenance activities once a product is launched to customers.
4) Production: This environment will support live customers and so test activities will be limited once this is the case. This will be fully managed and have strict change management and config management processes.
Hope this helps
You could give each developer the latest database docker image to play on their local environment. If the data is corrupted, they can just recreate the container.
In our company we have staging and production servers. I'm trying to have them in state 1:1 after latest release. We've got web application running on several host and many instances of it.
The issue is that I am an advocate of having the same architecture (structure) of web applications on staging and production servers to easily test new features and avoid creating of new bugs with new releases.
But not everyone agree with me, and for them is not a such big deal to have different connection between staging application instances. Even maybe to have more application and connections between application on staging than on production server.
I would like to ask about pros and cons of such an approach? I mean some good points to agree with me, or some bad why maybe i don't have right. Some examples of consequences and so forth.
If your staging server is substantially different from your production server, then successful deployment and testing on the staging server does not tell you much about whether the world will come down crashing on you when you finally deploy to the production server.
I do not see any real advantage to your colleagues' preferred chaotic situation, to compensate for this obvious disadvantage. What do they claim they gain by letting the staging server's configuration get totally out of sync with that of the production server...?!
Staging is like the dress rehearsal of deployment. If you're not wearing the same costume you will be wearing on the night, how do you know it's going to fit, or you're not going to trip over the dangly bits.
More formally, you try to keep the staging environment as close as possible to the production environment in order to minimize differences which may cause or hide issues in the deployment. Note that I say "close as possible", since it's not always possible to have the same model of disk, or the same network interconnects, but you try to minimize those things that you can within the resources you have available.
Martin Fowler recently blogged about having identical environments that could be cut over from one to the other, so your staging environment becomes your production environment after testing. He says:
One of the challenges with automating deployment is the cut-over itself, taking software from the final stage of testing to live production. You usually need to do this quickly in order to minimize downtime. The blue-green deployment approach does this by ensuring you have two production environments, as identical as possible. At any time one of them, let's say blue for the example, is live. As you prepare a new release of your software you do your final stage of testing in the green environment. Once the software is working in the green environment, you switch the router so that all incoming requests go to the green environment - the blue one is now idle.
I think an approach like this would be a great alternative to the seemingly chaotic environment you have today. Good luck convincing your team!
I would go with "as close as possible" approach as well, as ptomli suggested... and that's mostly due to cost factor. If it is a farm that contains 5 servers, I would never recommend the staging to be just 1 stand alone server. This helps in scenarios where network layer is involved as well. If there are patches that affect network connectivity due to any reason (like security!) a single box staging server might not reflect the "real affect" of patching.