I'm looking to implement continuous deployment, pretty much as a proof of concept using Windows Azure, deploying the packages and switching staging <-> production is all fine, however, I would like to add some smarts to the way it is deployed.
Essentially, if I have 10 instances, I want to deploy to all in the staging slot, and switch say 3 of them to production and monitor to make sure there is no statistical error difference between those 3 and the other 7 before switching all to production, or if there is, switch those 3 back to the original production which is now running in staging.
Essentially I want to mimic the sym link switching as described at http://timothyfitz.wordpress.com/2009/02/10/continuous-deployment-at-imvu-doing-the-impossible-fifty-times-a-day/
From what I can see, Azure only allows an all or nothing approach when switching between production and staging? I also thought about having two sets of roles defined, but the issue is there the same end point cant be used in two roles (I dont think?).
Anyone know of a way to do this?
Do a manual in-place upgrade. Then the update will happen one update domain at a time (and you can define how many update domains you want... default is five). If you set it to manual, you're in charge of when you move on to the next update domain. If something goes wrong, you start a new in-place upgrade to the old bits again.
Related
I want to deploy hybris builds with zero down time. Our technical architecture consist of two frontend servers, two backend servers, two master/slave solr clusters, but a single DB server (MS SQL 2012). A new build may require patch execution which changes the DB schema.
Would it be possible to achieve this in a single DB landscape?
If two DB's are required (blue and green), then what is the best practice for DB replication in case of hybris?
Hybris does provide a rolling update feature (when you're running it in a cluster environment).
This is targeted to allow for zero downtime.
You can find more information on the hybris help pages, e.g.
https://help.hybris.com/6.5.0/hcd/8c455268866910149b25f7b53d1af3e1.html
Looking at the first picture there it seems to be pretty much fitting for the architecture you describe.
(But to be honest I have no experience with it, so I can't tell you whether or or how well it works :) )
If you have risky changes or end up needing to rollback your rolled out update you will have to do quite a bit of db cleanup etc.
From that perspective a blue/green setup might sound better although with db replication you would end up with the same problem (as your updated schema would be replicated as well I assume).
Hybris only adding new columns to db, never change their type or remove them. So single DB can be OK. I didn't test this using store front while updating system. I think it will be OK.
On the other hand you need development for empty/null check for new attributes in development.
How would I go about setting up a backup for heroku downtimes set up on a vps like linode? (using nginx/unicorn)
Essentially very simply, but also with a whole world of hurt.
Simply create an instance of your application of said VPS.
Then you need to ensure that you're able to flip your DNS from Heroku to said VPS without waiting for a TTL to expire, or someway of letting the world know your application has moved.
Then figure out a reliable way of ensuring that the code on both environments is exactly the same, and works on both different server setups
Then figure out how you can keep the data up to date in both environments so that when you do need to flip, the data will be the same in both environments.
Then you need to figure out a way to remind yourself to keep this secondary VPS up to date from a server management point of view. Software updates, security patches etc etc.
Then you need to figure out a way that you can notified when Heroku is down 24/7
Then you need to hope that when Heroku is down that Linode isn't
... or just accept that any host will go down, and it can cost a hell of a lot of money to ensure that your site doesn't. To be honest, it's probably better for you to look at some sort of hosting setup that allows redundancy and failover across several locations (which won't be cheap)
There are third party services which provide the ability to keep your site (parts of) up if your server goes down - At least it appears to the user that your site is up but it's not working properly behind the scenes. CloudFlare is one such service. It sits in front of your site/application and performs magic (quite simply). It works with static/dynamic sites - and if your server goes offline then they are able to serve static parts of your site. See http://support.cloudflare.com/kb/what-do-the-various-cloudflare-settings-do/what-does-enabling-cloudflare-offline-browsing-do
Using CouchDB to create a hosted app for clients. I have a dev database I work from, as well as separate DBs for each client. Works well, problem is when I make a change on dev, I have to manually copy the view code into each separate DB. It's fine now that I have 2 clients. But my hope is to grow to 100 clients. One small change could take a very long time!
Am I missing something simple in regards to replicating ONLY the views?
Thanks!
Here is how I usually work.
I have my local dev db. create and update my design docs (containing the views).
Have a production deployment db that will be visible to all the clients. I usually use iriscouch. Keep no data in this db.
When setting up a client, make sure you setup one way replication from #2 to this client db.
So to deploy to all clients, I put my latest design docs on the master, then all the clients will then be updated. There are some caveats to this. You have to make sure when you deploy to the master db, that you respect the revisions, so the client dbs will know to update.
Here is a quote from the master, Jason Smith:
The Good Way: Work with _rev
I think your application has a concept of "upgrading" from one
revision to another. There is staging or development code, and there
is production code. Periodically you promote development code to
production. That sounds like two Git branches and it also sounds like
two doc ids. (Or two sets of doc ids.)
You can test and refactor your code all day long, in the temporary doc
(_design/dev). But in production (_design/pro), it's just like a long
Git history. Every revision built from the one previous, to the
beginning of time.
If you want to promote _design/dev, the latest deploy is
_rev=4-abcdef. So this will be the fifth revision deployed, right?
Hey! Stop reading the "_rev" field! But yeah, probably.
COPY /db/_design/dev
Destination: _design/pro?rev=4-abcdef
{"id":"_design/pro","rev":"5-12345whatever"}
Notice that each deployed _design/pro builds from the other, so it
will naturally float out to the slaves when they replicate.
In real-life, you may have add a middle step, pushing design documents
to production servers before actually publishing them. Once you push,
how long will it take couch to build new views? The answer is,
"Christ, who knows?"
Therefore you have to copy _design/dev to _design/staging and then
push that out into the wild. Then you have to query its views until
you are satisfied that they are fresh and fast. (You can compare
"update_seq" from /db vs. "update_seq" from /db/_design/ddoc/_info).
And only then do you HTTP copy from _design/staging to _design/pro and
let that propagate out.
Source
Its not as confusing as it may sound. But to simplify the process, you can use Reupholster
(I admit, I have written this tool). It is mainly for couchapps, but even if you are just promoting design docs, it might be worth you just using reupholster to deploy to your master db. Reupholster adds in some handy info to the design doc, like date/time svn or git info. That way when you look at a clients db you can tell which design doc they are on.
Good luck
You can replicate just the design docs;
http://wiki.apache.org/couchdb/Replication#Named_Document_Replication
Closed. This question is opinion-based. It is not currently accepting answers.
Want to improve this question? Update the question so it can be answered with facts and citations by editing this post.
Closed 1 year ago.
The community reviewed whether to reopen this question 1 year ago and left it closed:
Original close reason(s) were not resolved
Improve this question
In the company where I work, we have 2 environments: test and production. We are not currently starting a new environment, because of cost.
Here is the procedure we follow: business makes a feature request, development makes it happen and deploys it on test environment. Then business tests it (UAT), and if it's OK, the feature will be included into next production deployment.
The problem is best practices for test DB. Developers treat test environment as their playground, and sometimes they reset the DB to initial state for testing purposes. On the other hand, business people think that test DB must be stable, and should not be reset. We would like to resolve this issue, and decide if test environment should belong to development team or business team. (Developers don't want business to put their nose in test env., but business team is paying for servers.)
What is the best practice about environments? Can you recommend an article about this?
I've worked in many companies, each one with a different set of environments, the one I have like the most had 5 environments:
1) Local: Basically your machine. Here is where you code and test your changes before even asking a peer for review.
2) Dev: If for some reason you cannot test your code locally (Dependencies issues mainly: "My code has neves compiled in my machine but it compiles perfectly in Jenkins/Bamboo/Travis"), then you push your changes to your feature branch in Git and make Bamboo compile it and deploy it to a dev server where you can test (you are still not sure if it will work, so no peer review so far).
3) Staging: You think your code work and you love how it looks like. You create a Pull Request in order for you peers to take a look at it before it gets merged to the master branch. Here they make comments and you fix possible issues, since you are more sure about your changes you make Bamboo deploy it to the Staging environment where more "stable" code lives and more realistic data is stored in whatever database is there. Once deployed another developer/tester can check your changes actually work and make you a "QA Sign off in Staging Env."
4) Stable: Ok, now you are ultimate sure your changes work since you deployed to Staging and nothing got broken. You merge your branch to master and Bamboo compiles master and deploys to another sets of servers in a Stable Environment (no one else should merge to master until you finish your deployment to Production, to avoid a merge of not related merges). This environment should be a replica from Production, data, code and server conditions. Here is where you show your changes to your manager, product owner or the person in charge to validate your work before sending it to production. You get the final approval, everything is good, you are sweaty, you have worked for 30 days in a row to get this change done, your wife divorced you but you are very confident it works perfectly.
5) Production: Where the clients connect to consume the company services, or the final build of your software to send to customers. With a few clicks in Bamboo you make it deploy to Production servers or compile the final build. It is green, everything seems to be ok. You check Splunk looking for errors, everything is good, life is good, you drink another sip of coffee before leaving, you drive home and sleep all weekend with your dog by your side.
It's a happy ending because having so many "Test" environments ensure quality, no change gets to production until EVERYBODY (not just you) are completely sure that changes is working.
At our company there are two databases too, a test and a production database. The test database is mainly used for testing by developers but sometimes for business tests too. This database is refreshed daily using an actual copy of the production database. So this database can be both a playground and a serious testing database. But a third, development, database is the best option. We had one, but it is broken at the moment. But when you get one of those, you should make sure it is refreshed often enough. When developers use it as a playground, it will stray away from the production environment, and its data will be both old and currupt. Because of this, developers won't be able to test well themselves. So make sure you refresh this database periodically (maybe daily too, or at least once a week).
If possible, give each developer their own database on their local machine. That way they can do whatever they want with it without affecting anyone else. This should significantly decrease the desire to play with the test database, providing a more stable environment for business UAT.
I believe in order to establish an environment strategy that supports all ALM/SDLC activities 4 requirements should exist:
1) Development environment: to allow Dev to play around with new code/concepts and typically unit test with some basic integration testing using stubs and drivers. This environment should have loose change control procedures and would typically not be anywhere near the same scale as production. This is where the Dev team can build and tear down setups as required.
2) Interop environment: where integration of systems can be further tested and increased capability for non-functional testing I.e might be a resilient env with greater scalability than Dev. I'd see this environment having tighter change control and management. Test would perform integration and system testing in this environment.
3) Reference Architecture: This is what some might call pre-prod but is essentially the same as production in terms of scale and resilience. This would have change control and management procedures akin to prod. This env would support further test activities especially full scale performance testing, failover as well as operational fault triage and maintenance activities once a product is launched to customers.
4) Production: This environment will support live customers and so test activities will be limited once this is the case. This will be fully managed and have strict change management and config management processes.
Hope this helps
You could give each developer the latest database docker image to play on their local environment. If the data is corrupted, they can just recreate the container.
In our company we have staging and production servers. I'm trying to have them in state 1:1 after latest release. We've got web application running on several host and many instances of it.
The issue is that I am an advocate of having the same architecture (structure) of web applications on staging and production servers to easily test new features and avoid creating of new bugs with new releases.
But not everyone agree with me, and for them is not a such big deal to have different connection between staging application instances. Even maybe to have more application and connections between application on staging than on production server.
I would like to ask about pros and cons of such an approach? I mean some good points to agree with me, or some bad why maybe i don't have right. Some examples of consequences and so forth.
If your staging server is substantially different from your production server, then successful deployment and testing on the staging server does not tell you much about whether the world will come down crashing on you when you finally deploy to the production server.
I do not see any real advantage to your colleagues' preferred chaotic situation, to compensate for this obvious disadvantage. What do they claim they gain by letting the staging server's configuration get totally out of sync with that of the production server...?!
Staging is like the dress rehearsal of deployment. If you're not wearing the same costume you will be wearing on the night, how do you know it's going to fit, or you're not going to trip over the dangly bits.
More formally, you try to keep the staging environment as close as possible to the production environment in order to minimize differences which may cause or hide issues in the deployment. Note that I say "close as possible", since it's not always possible to have the same model of disk, or the same network interconnects, but you try to minimize those things that you can within the resources you have available.
Martin Fowler recently blogged about having identical environments that could be cut over from one to the other, so your staging environment becomes your production environment after testing. He says:
One of the challenges with automating deployment is the cut-over itself, taking software from the final stage of testing to live production. You usually need to do this quickly in order to minimize downtime. The blue-green deployment approach does this by ensuring you have two production environments, as identical as possible. At any time one of them, let's say blue for the example, is live. As you prepare a new release of your software you do your final stage of testing in the green environment. Once the software is working in the green environment, you switch the router so that all incoming requests go to the green environment - the blue one is now idle.
I think an approach like this would be a great alternative to the seemingly chaotic environment you have today. Good luck convincing your team!
I would go with "as close as possible" approach as well, as ptomli suggested... and that's mostly due to cost factor. If it is a farm that contains 5 servers, I would never recommend the staging to be just 1 stand alone server. This helps in scenarios where network layer is involved as well. If there are patches that affect network connectivity due to any reason (like security!) a single box staging server might not reflect the "real affect" of patching.