I'm interested in IaC's continuous deployment and continuous integration. As we need to ensure the infrastructure is ready to be deployed for production, my initiation thinking was to have separate environments for dev and prod. But the point that I'm a little bit hesitant to consider it as a problem or usual stuff is about the cost efficiency of this idea. Because we are running the infrastructure exactly twice, the infrastructure cost will double.
Is this usual to have two separate and identical infrastructure for dev and prod?
Or is there any working solution that makes the continuous integration for infrastructure considering high availability and high resiliency more cost-efficient?
Yes! having a separate setup for each environment is highly recommended. So you will have a more visible workflow between different environments. But do you need to manage identical setup for dev and prod? Please No. You can keep your dev setup at a minimum. Yes, It will cost some bucks, but it will help you to keep your production bug-free and save more money, and time that you will have to put on troubleshooting.
Related
Based on the parameters
Cost
Hardware Requirements (RAM,Nodes etc)
Also elaborate availability requirements for production and development environments.
It depends upon your development / production setup environment. For both development and production- its better to go with Cloud setup.
Because - assuming you need to serve reasonable load in production, you would need less or more instances ( cloud instance such as redis on docker ) so that you can get the benefit of cloud auto-scaling.
For pre-prod and prod - you would need more load - may be two (min - for disaster recovery or blue green deployment) to more instances. For dev just one instance should be sufficient.
The question is tied more to CI/CD practices and infrastructure. In the release we follow, we club a set of microservices docker image tags as a single release, and do CI/CD pipeline and promote that version.yaml to staging and production - say a sort of Mono-release pattern. The problem with this is that at one point we need to serialize and other changes have to wait, till a mono-release is tested and tagged as ready for the next stage.A little more description regarding this here.
An alternate would be the micro-release strategy, where each microservice release in parallel through production through the CI/CD pipeline. But then would this mean that there would be as many pipelines as there are microservices? An alternate could have a single pipeline, but parallel test cases and a polling CD - sort of like GitOps way which takes the latest production tagged Docker images.
There seems precious little information regarding the way MS is released. Most talk about interface level or API level versioning and releasing, which is not really what I am after.
Assuming your organization is developing services in microservices architecture and is deploying in a kubernetes cluster, you must use some CD tool (continuous delivery tool) to release new microservices services, or even update a microservice.
Take a look in tools like Jenkins (https://www.jenkins.io), DroneIO (https://drone.io)... Some organizations use Python scripts, or Go and so on... I, personally, do not like this approch, I think the best solution is to pick a tool from CNCF Landscape (https://landscape.cncf.io/zoom=150) in Continuous Integration & Delivery group, these are tools test and used in the market.
An alternate would be the micro-release strategy, where each microservice release in parallel through production through the CI/CD pipeline. But then would this mean that there would be as many pipelines as there are microservices?
It's ok in some tools you have a parameterized pipeline thats build projects based in received parameters, but I think the best solution is to have one pipeline per service, and some parameterized pipelines to deploy, or apply specific tests, archive assets and so on... Like you say micro-release strategy
Agreed, there is little information about this out there. From all I understand the approach to keep one pipeline per service sounds reasonable. With a growing amount of microservices you will run into several problems:
how do you keep track of changes in the configuration
how do you test your services efficiently with regression and integration tests
how do you efficiently setup environments
The key here is most probably that you make better use of parameterized environment variables that you then look to version in an efficient manner. This will allow you to keep track of the changes in an efficient manner. To achieve this make sure to a.) strictly paramterize all variables in the container configs and the code and b.) organize the config variables in a way that allows you to inject them at runtime. This is a piece of content that I found helpful in regard to my point a.);
As for point b.) this is slightly more tricky. As it looks you are using Kubernetes so you might just want to pick something like helm-charts. The question is how you structure your config files and you have two options:
Use something like Kustomize which is a configuration management tool that will allow you to version to a certain degree following a GitOps approach. This comes (in my biased opinion) with a good amount of flaws. Git is ultimately not meant for configuration management, it's hard to follow changes, to build diffs, to identify the relevant history if you handle that amount of services.
You use a Continuous Delivery API (I work for one so make sure you question this sufficiently). CDAPIs connect to all your systems (CI pipelines, clusters, image registries, external resources (DBs, file storage), internal resources (elastic, redis) etc. They dynamically inject environment variables at run-time and create the manifests with each deployment. They cache these as so called "deployment sets". Deployment Sets are the representation of the state of an environment at deployment time. This approach has several advantages: It allows you to share, version, diff and relaunch any state any service and application were in at any given point in time. It provides a very clear and bullet proof audit auf anything in the setup. QA environments or test-feature environments can be spun of through the API or UI allowing for fully featured regression and integration tests.
How to proceed with performing performance testing for an application that is not in production.
If the environment of the application under test is the same as it will be in production - the approach should not be different.
If the environment of the application under test is different, i.e. has less servers, the servers has less memory, etc. - in the absolute majority of cases you will not be able to calculate and predict the performance on more powerful hardware as there are too many factors to consider. You can inform the client about it right away.
However there are still some types of testing you can perform, i.e.:
You could run an integration test to check whether your application is configured for high load
You could run a scalability test to check whether and how does your application scale
You could run a soak test to check for possible memory leaks
You could use profiling tools to check bottlenecks in terms of code quality
You could identify slow DB queries and look for a way to optimise them
etc.
More information: Performance Testing in a Scaled Down Environment. Part Two: 5 Things You Can Test
If you meant how to judge on how much volume to test if application is not already in production, the answer is simple, you have to predict. The prediction can be based on survey, reports from your business analysts.
If none of the above are available. Just test how much your application can withstand in a test environment of similar configuration as production. This will give you an idea of when you need to start worrying while the application is live.
I decided to move to MicroService architecture, divide a project into multiple services and run those services on DCOS.It really gives a good story to project deployment and maintenance. But it makes development process complex.
For the developer, it was easy to run the application locally while implementation is in progress.Now the project is divided into multiple services and runs on DCOS which require good configuration. so to test application for the developer in the middle of implementation becomes a nightmare.
Guys, anyone is using DCOS with Microservice, can you please suggest what process you are following for internal development.
DCOS is just a tool for your deployment, so in order to give a better answer you'd have to share more about your technology stack in your question. But here some general thoughts: There are different types/levels of testing and there are different considerations for each.
On unit level - This depends on what technology you use for implementing your services and is the reason why languages like go become more and more popular for server development. If you use go for example, you can easily run any service you are currently developing locally (not containerized) on the dev machine. And you can easily run attached unit tests. You would either run dependent services locally or mock them up (probably depending on effort). You may also prefer asking each service team to provide mock services as part of their regular deliveries.
And you will require special environment settings and service configuration for the local environment.So summarized this approach will require you to have means in place to run services locally and depending on the implementation technologies you use it will be easier or harder.
On deployment/integration level - Setting up a minimal cluster on the dev's local machine and/or using dedicated testing- and staging clusters. This allows you to test the services including containerization and in a more final deployment environment with dependencies. You would probably write special test clients for this type of test. And this approach will also require you to have separate environment settings, configuration files, etc. for the different environments. Tools like Jaeger become more popular for helping with debugging errors through multiple services here.
Canary testing. Like the name suggests - You deploy your latest service version to a small portion of your production cluster in order to test it on a limited number of users first before rolling it out to the masses. In this stage you can run user level tests and in fact your users become the testers, so it is to be used carefully. Some organizations prefer to have special beta-type-users that will only get access to those environments.
In our company we have staging and production servers. I'm trying to have them in state 1:1 after latest release. We've got web application running on several host and many instances of it.
The issue is that I am an advocate of having the same architecture (structure) of web applications on staging and production servers to easily test new features and avoid creating of new bugs with new releases.
But not everyone agree with me, and for them is not a such big deal to have different connection between staging application instances. Even maybe to have more application and connections between application on staging than on production server.
I would like to ask about pros and cons of such an approach? I mean some good points to agree with me, or some bad why maybe i don't have right. Some examples of consequences and so forth.
If your staging server is substantially different from your production server, then successful deployment and testing on the staging server does not tell you much about whether the world will come down crashing on you when you finally deploy to the production server.
I do not see any real advantage to your colleagues' preferred chaotic situation, to compensate for this obvious disadvantage. What do they claim they gain by letting the staging server's configuration get totally out of sync with that of the production server...?!
Staging is like the dress rehearsal of deployment. If you're not wearing the same costume you will be wearing on the night, how do you know it's going to fit, or you're not going to trip over the dangly bits.
More formally, you try to keep the staging environment as close as possible to the production environment in order to minimize differences which may cause or hide issues in the deployment. Note that I say "close as possible", since it's not always possible to have the same model of disk, or the same network interconnects, but you try to minimize those things that you can within the resources you have available.
Martin Fowler recently blogged about having identical environments that could be cut over from one to the other, so your staging environment becomes your production environment after testing. He says:
One of the challenges with automating deployment is the cut-over itself, taking software from the final stage of testing to live production. You usually need to do this quickly in order to minimize downtime. The blue-green deployment approach does this by ensuring you have two production environments, as identical as possible. At any time one of them, let's say blue for the example, is live. As you prepare a new release of your software you do your final stage of testing in the green environment. Once the software is working in the green environment, you switch the router so that all incoming requests go to the green environment - the blue one is now idle.
I think an approach like this would be a great alternative to the seemingly chaotic environment you have today. Good luck convincing your team!
I would go with "as close as possible" approach as well, as ptomli suggested... and that's mostly due to cost factor. If it is a farm that contains 5 servers, I would never recommend the staging to be just 1 stand alone server. This helps in scenarios where network layer is involved as well. If there are patches that affect network connectivity due to any reason (like security!) a single box staging server might not reflect the "real affect" of patching.