So I've been investigating ci/cd pipelines using concourse and cloud foundry lately, and I've been confused about what the best way to do this is. So I've been thinking about how the overall flow would go from development to release. There are a lot of talks and videos that discuss this at a very high level, but often they abstract away too much of the actual implementation details for it to be useful. Like how do people actually roll this out in actual companies? I have a lot of questions, so I will try to list a few of them here in the hope that someone could enlighten me a little.
What does the overall process and pipeline look like conceptually from development to prod? So far I have something along the lines of :
During development each product team is under their own org, with each developer possibly having their own development "space" that they could manually cf push to and just develop against. There will be development spaces that devs can just directly push to as well as spaces that can only be used by the automated pipeline to deploy artifacts for functional tests.
Once devs finish a feature they would make a pull request, which would trigger a smaller pipeline with some tests using something like the git-multibranch-resource or the git-pullrequest-resource, which would hook into the github required status check hooks and report back if any particular PRs are able to be merged into master or not
Once all checks pass and the pull request is merged into master the below pipeline is kicked off, which validates the master branch before releasing the artifact to prod.
code repo [master] -> build -> snapshot artifact repo -> deploy to test space -> run functional tests -> deploy to staging space -> run smoke tests and maybe other regression tests -> deploy artifact to prod -> monitoring/rollbacks (?)
What other things could/should be added to this pipeline or any part of this process?
Once you automate deployment how do you do also automate things like canary releases or rollbacks once something happens? Should this be part of the pipeline or something completely separate?
I've been playing with the idea of creating spaces temporarily and then tearing them down for the functional testing phase, would there be any benefit to doing that? The idea is that the apps being deployed would have their own clean environments to use, but this could also potentially be slow, and it is difficult to know what services are required inside of each space. You would have to read the manifest, which only specifies service-names, which seems to necessitate some sort of canonical way of naming service instances within the same space? The alternative is managing a pool of spaces which also seems complicated...
Should the pipeline generate the manifest files? Or should that be completely up to the developers? Only the developers know which services the app needs, but also it seems like things like instance count, memory etc should be something that the performances tests/pipeline should be able to determine/automate. You could generate a manifest inside the pipeline, but then you would not know which services the app needs without reading a manifest....chicken and egg problem?
I have many more burning questions, but I will cut it off here for now. I know the subjects have kind of bounced back and forth between Concourse and Cloud Foundry, but it seems when discussing CI/CD concepts the nitty gritty implementation details are often the actual tricky bits which tangle the two rather tightly together. I am also aware that the specific implementation details are often very specific to each company, but it would be great if people could talk about how they have implemented these pipelines / automated pipelines using Concourse and Cloud Foundry at their companies (if you can spare the details of course). Thanks everyone!
During development each product team is under their own org, with each developer possibly having their own development "space" that they could manually cf push to and just develop against. There will be development spaces that devs can just directly push to as well as spaces that can only be used by the automated pipeline to deploy artifacts for functional tests.
Honestly it doesn't matter if you create multiple orgs in your CloudFoundry. If your CI/CD system runs on the same director that is (ab)used by other developers you going to have a hard time probably (i was there).
Once devs finish a feature they would make a pull request, which would trigger a smaller pipeline with some tests using something like the git-multibranch-resource or the git-pullrequest-resource, which would hook into the github required status check hooks and report back if any particular PRs are able to be merged into master or not
We are doing almost the exact thing. For PR's checkout jtarchie's PR resource here https://github.com/jtarchie/github-pullrequest-resource.
The only difference is that we are not using Github checks. The problem with them is, that you have to select a set of checks fixed for a branch.
But in case i just changed manifest xyz in the PR, i don't want to run all tests. You can overcome that problem by using the Github Status API only with the pending and successful status.
Once all checks pass and the pull request is merged into master the below pipeline is kicked off, which validates the master branch before releasing the artifact to prod.
We make PR's into the develop branch and following the Git Flow system. Our releases are merged into master manually.
You want to check first which updates you want to carry out before you merge every PR into master and trigger an update of the production system. Your test cases might be good, but you can always miss something.
What other things could/should be added to this pipeline or any part of this process?
You can have a pipeline which updates releases/stemcells in your manifests automatically.
Once you automate deployment how do you do also automate things like canary releases or rollbacks once something happens? Should this be part of the pipeline or something completely separate?
Test your stuff on a staging system before you go to production. Otherwise you a) won't know if the update is happening at zero downtime and b) to prevent a potential problem in production is always better than doing rollbacks.
Ofc you can also create a rollback pipeline, but if you come to that point something else might be wrong with your setup.
Should the pipeline generate the manifest files? Or should that be completely up to the developers? Only the developers know which services the app needs, but also it seems like things like instance count, memory etc should be something that the performances tests/pipeline should be able to determine/automate. You could generate a manifest inside the pipeline, but then you would not know which services the app needs without reading a manifest....chicken and egg problem?
We write our manifests by ourselves and use the CI/CD system to update/deploy/test them.
But if you find a valid case and a concept which lets you apply your manifest generating pipeline for many cases, i would just try it out.
At the very end you have to decide if a certain atomisation holds a business value for your company.
cheers, Dennis
Related
When there is a team using terraform to manage cloud resources (via a shared git repository), and most members do not have direct individual access to the state file (for security reasons), what practices are available for testing and debugging change (e.g. feature) proposals prior to circulating pull requests?
Without a state file (or admin credentials), can terraform locally produce a speculative plan for the changes with respect to a base commit? Or would it be necessary to produce separate speculative plans (for the main branch and the new feature branch) then perform a diff? Like, can another speculative plan be used in place of a state file? Or is the standard workflow to keep pushing unfinished work to a temporary remote branch, with continuous-integration pipelines configured to automatically generate plans against the actual state?
I guess what you intend is to create workflow like this:
Junior DevOps commits code with proposal of some feature which needs some kind of resource in cloud. (we don't trust them enough yet, so he can't provision it resource by himself)
CI is making some checks about the code (checkov, terraform verify, etc.)
CI tests are passing -> we'll come back here
Code is reviewed by senior Devops and gets accepted
CI executes terraform apply
If so, let's focus on question...
What does it mean to test terraform changes?
Depending on case it could be:
terraform plan prints valid plan
terraform apply works on test environment (separated from production, with empty state file - remember to clean this after tests)
this case also allows to run some real tests (e2e, simple curl, integration tests, etc.)
Of course second one is a lot better, but not always you can (or it makes sense) to create test resources. Usually it can be made by setting some different name in resource.
What test probably doesn't mean is to apply changes on your production.
Diff from plan also isn't best 'test' available since also two terraform plan run one after another can differ.
The question is tied more to CI/CD practices and infrastructure. In the release we follow, we club a set of microservices docker image tags as a single release, and do CI/CD pipeline and promote that version.yaml to staging and production - say a sort of Mono-release pattern. The problem with this is that at one point we need to serialize and other changes have to wait, till a mono-release is tested and tagged as ready for the next stage.A little more description regarding this here.
An alternate would be the micro-release strategy, where each microservice release in parallel through production through the CI/CD pipeline. But then would this mean that there would be as many pipelines as there are microservices? An alternate could have a single pipeline, but parallel test cases and a polling CD - sort of like GitOps way which takes the latest production tagged Docker images.
There seems precious little information regarding the way MS is released. Most talk about interface level or API level versioning and releasing, which is not really what I am after.
Assuming your organization is developing services in microservices architecture and is deploying in a kubernetes cluster, you must use some CD tool (continuous delivery tool) to release new microservices services, or even update a microservice.
Take a look in tools like Jenkins (https://www.jenkins.io), DroneIO (https://drone.io)... Some organizations use Python scripts, or Go and so on... I, personally, do not like this approch, I think the best solution is to pick a tool from CNCF Landscape (https://landscape.cncf.io/zoom=150) in Continuous Integration & Delivery group, these are tools test and used in the market.
An alternate would be the micro-release strategy, where each microservice release in parallel through production through the CI/CD pipeline. But then would this mean that there would be as many pipelines as there are microservices?
It's ok in some tools you have a parameterized pipeline thats build projects based in received parameters, but I think the best solution is to have one pipeline per service, and some parameterized pipelines to deploy, or apply specific tests, archive assets and so on... Like you say micro-release strategy
Agreed, there is little information about this out there. From all I understand the approach to keep one pipeline per service sounds reasonable. With a growing amount of microservices you will run into several problems:
how do you keep track of changes in the configuration
how do you test your services efficiently with regression and integration tests
how do you efficiently setup environments
The key here is most probably that you make better use of parameterized environment variables that you then look to version in an efficient manner. This will allow you to keep track of the changes in an efficient manner. To achieve this make sure to a.) strictly paramterize all variables in the container configs and the code and b.) organize the config variables in a way that allows you to inject them at runtime. This is a piece of content that I found helpful in regard to my point a.);
As for point b.) this is slightly more tricky. As it looks you are using Kubernetes so you might just want to pick something like helm-charts. The question is how you structure your config files and you have two options:
Use something like Kustomize which is a configuration management tool that will allow you to version to a certain degree following a GitOps approach. This comes (in my biased opinion) with a good amount of flaws. Git is ultimately not meant for configuration management, it's hard to follow changes, to build diffs, to identify the relevant history if you handle that amount of services.
You use a Continuous Delivery API (I work for one so make sure you question this sufficiently). CDAPIs connect to all your systems (CI pipelines, clusters, image registries, external resources (DBs, file storage), internal resources (elastic, redis) etc. They dynamically inject environment variables at run-time and create the manifests with each deployment. They cache these as so called "deployment sets". Deployment Sets are the representation of the state of an environment at deployment time. This approach has several advantages: It allows you to share, version, diff and relaunch any state any service and application were in at any given point in time. It provides a very clear and bullet proof audit auf anything in the setup. QA environments or test-feature environments can be spun of through the API or UI allowing for fully featured regression and integration tests.
I have question about microservices and repositories. We are a small team (5 people) and we creating new project in microservices. Expected microservice applications in our project is between 10-15.
We are thinking about one repository for all microservices in structure like that:
-/
--/app1
--/app2
--/app3
-./script.sh
-./script.bat
What do you think about this design? Can you recommend something better? We think if we will have repository per app it will be overkill for that small project in one team. As our applications you can imagine spring boot or spa applications in angular. Thank you in advice.
In general you can have all your micro-services in one repository but I think while the code grows for each of them it can be difficult to manage that.
Here are some things that you might want to consider before deciding to put all your micro-services in one repository:
Developer discipline:
Be careful with coupling of code. Since the code for all your micro-services is in one repository you don't have a real physical boundary between them, so developers can just use some code from other micro-services like adding a reference or similar. Having all micro-services in one repository will require some discipline and rules for developers not to cross boundaries and misuse them.
Come into temptation to create and misuse shared code.
This is not a bad thing if you do it in a proper and structured way. Again this leaves a lot of space for doing it the wrong way. If people just start using the same shared jar or similar that could lead to a lot of problems. In order to have something shared it should be isolated and packaged and ideally should have some versioning with support for backwards compatibility. This way each micro-service when this library is updated would still have a working code with the previous version. Still it is doable in the same repository but as with the 1. point above it requires planing and managing.
Git considerations:
Managing a lot of pull requests and branches in one repository can be challenging and can lead to the situation: "I am blocked by someone else". Also as possibly more people are going to work on the project and will commit to your source branch you will have to do rebase and/or merge source branch to your development or feature branch much more often(even if you do not need the changes from other services). Email notifications configured for the repository can be very annoying as you will receive Emails for things which are not in your micro-service code. In this case you need to create some Filters/Rules in your Email clients to avoid the Emails that you are not interested in.
Number of micro-services grow even further then your initial 10-15. The number can grow? If not, all fine. But if it does, at some point you could maybe consider to split each micro-service in a dedicated repository. Doing this at the point where you are in later stage of project can be challenging and could require some work and in worst case you will find out that there are some couplings that people made over time which you will have to resolve at this stage.
CI pipelines considerations:
If you use something like Jenkins to build, test and/or deploy your code
you could encounter some small configuration difficulties like the integration between Jenkins and GitHub. You would need to configure a pipeline which would only build/test a specific part of the code (or one micro-service) if someone creates a merge/pull request against that micro-service. I never tried to do such a thing but I guess you will have to figure out how to do it (script and automate this). It is doable I guess but will required some work to achieve it.
Conclusion
Still all or most of these points can be resolved with some extra management and configuration but it is still worth knowing what additional effort you could encounter. I guess there are some other points to be taken into considerations as well but my general advice would be to use separate repositories for each micro-service if you can (Private Repository pricing and similar reasons). This is a decision which is made project by project.
I've been researching cloud based CI systems for a while now and cannot seem to find any systems that can address a major need of mine.
I'm building CI processes for development on Salesforce, but this question is more generally about builds which rely on an external resource. In our builds, we deploy code into a cloud hosted Salesforce instance and then run the tests in that instance. During a build, the external resource is effectively locked and build failures will occur if two builds target the same external resource at the same time. This means that the normal concurrency model of cloud based CI systems would start tripping over the Salesforce instance (external resource) with a concurrency greater than 1.
To complicate things a bit more, we actually have 5 different external resources for each project (feature, master, packaging, beta, and release) and need to control the concurrency of any builds relying on an external resource to 1. For example, all our feature branches build against the feature external resource. We can identify these builds by the branch name which uses the pattern feature/* and need to ensure that only one feature build runs at a time. However, the feature build doesn't tie up the other 4 external resources so ideally any builds that would need those resources should still be able to run concurrently.
I currently accomplish this in Jenkins using the Throttle Concurrent Builds plugin and assign a throttle group to each build identifying the external resource it relies on. This has been successful at preventing concurrent builds from tripping over external resources.
A few clarifications:
I'm not asking how to reduce concurrency to 1 at the repo level. I know every cloud CI system can do that. I should be able to set repo concurrency to N external resources (in my case, 5).
Ideally, I'd like to be able to use a regex pattern on branch name as the "group" with which to block concurrence. So, a setting like: If branch name matches 'feature/.*' then limit concurrency to 1. I want to avoid having to manually configure new feature branches in the build system and instead match on pattern.
I have to say, it's been nearly impossible to find a restrictive Google search term that would help me answer this question. Hopefully someone out there has faced this problem before and can shed some light for me :)
With Jenkins Pipeline plugin you can set the stage concurrency to 1 - and only 1 thing will pass through that stage at a time. The stage was designed to be able to represent things like this.
https://www.cloudbees.com/blog/parallelism-and-distributed-builds-jenkins
stage "build"
node {
sh './test-the-awesome'
}
stage name: "environment test", concurrency: 1
node {
sh 'tests that lock the environment'
}
You can put the build pipeline in a Jenkinsfile in a repo too: https://documentation.cloudbees.com/docs/cookbook/pipeline-as-code.html (so any branches that build, also obey that lock).
As pointed out by #Jesse Glick in the comments below, perhaps a more general solution (not yet compatible with pipeline) is to use the Lockable Resources Plugin - which will then work across jobs, of any type.
I accomplish this with a Drone.io setup.
Essentially, I use a grunt plugin to access a Redis db hosted externally. It provides semaphore locking on any param you'd like.
Determine if the lock is free for that Env.
If so that Env's Key with a reasonable timeouts
Run the tests
Clear the lock
If the lock is held, get it's expiration time, and sleep until then.
I am not aware of any cloud based CI tools that can manage external resources the way you want to, unless you include the logic as part of the build script, which you've already said you'd prefer not to do. If you decide you want to do that you could do it with Snap CI or Drone or any of the other cloud tools I imagine.
In this sort of situation, I would usually recommend an agent-based system such as Go.cd
Using CouchDB to create a hosted app for clients. I have a dev database I work from, as well as separate DBs for each client. Works well, problem is when I make a change on dev, I have to manually copy the view code into each separate DB. It's fine now that I have 2 clients. But my hope is to grow to 100 clients. One small change could take a very long time!
Am I missing something simple in regards to replicating ONLY the views?
Thanks!
Here is how I usually work.
I have my local dev db. create and update my design docs (containing the views).
Have a production deployment db that will be visible to all the clients. I usually use iriscouch. Keep no data in this db.
When setting up a client, make sure you setup one way replication from #2 to this client db.
So to deploy to all clients, I put my latest design docs on the master, then all the clients will then be updated. There are some caveats to this. You have to make sure when you deploy to the master db, that you respect the revisions, so the client dbs will know to update.
Here is a quote from the master, Jason Smith:
The Good Way: Work with _rev
I think your application has a concept of "upgrading" from one
revision to another. There is staging or development code, and there
is production code. Periodically you promote development code to
production. That sounds like two Git branches and it also sounds like
two doc ids. (Or two sets of doc ids.)
You can test and refactor your code all day long, in the temporary doc
(_design/dev). But in production (_design/pro), it's just like a long
Git history. Every revision built from the one previous, to the
beginning of time.
If you want to promote _design/dev, the latest deploy is
_rev=4-abcdef. So this will be the fifth revision deployed, right?
Hey! Stop reading the "_rev" field! But yeah, probably.
COPY /db/_design/dev
Destination: _design/pro?rev=4-abcdef
{"id":"_design/pro","rev":"5-12345whatever"}
Notice that each deployed _design/pro builds from the other, so it
will naturally float out to the slaves when they replicate.
In real-life, you may have add a middle step, pushing design documents
to production servers before actually publishing them. Once you push,
how long will it take couch to build new views? The answer is,
"Christ, who knows?"
Therefore you have to copy _design/dev to _design/staging and then
push that out into the wild. Then you have to query its views until
you are satisfied that they are fresh and fast. (You can compare
"update_seq" from /db vs. "update_seq" from /db/_design/ddoc/_info).
And only then do you HTTP copy from _design/staging to _design/pro and
let that propagate out.
Source
Its not as confusing as it may sound. But to simplify the process, you can use Reupholster
(I admit, I have written this tool). It is mainly for couchapps, but even if you are just promoting design docs, it might be worth you just using reupholster to deploy to your master db. Reupholster adds in some handy info to the design doc, like date/time svn or git info. That way when you look at a clients db you can tell which design doc they are on.
Good luck
You can replicate just the design docs;
http://wiki.apache.org/couchdb/Replication#Named_Document_Replication