I'm wondering if there is some good way to make CI/CD for single git repository that contains many projects. I often work with microservices, but I always preferred to have separate repo for each project and thanks to that I had separate pipelines for each repo. Now I'm in situation where all microservices are in single repo and I would like to know if there is some good way to make it work. The question is general because I don't have any CI tool right now, probably going to use Gitlab CI or Jenkins if I'm gonna need something more complex.
Yes it is possible, but you need to understand that your pipeline can be too big depending how many microservices you have in this repo
look here: https://docs.gitlab.com/ee/ci/yaml/#onlyexcept-basic
you are able to add a lot of only/except conditions to avoid or force some specific steps
example
build only if folder x is changed
test only if microservice x was built.... except if branch is example
Related
Both Github Actions and Bitbucket Pipelines seem to fill similar functions at a surface level. Is it trivial to migrate the YAML for Actions into a Pipeline - or do they operate fundamentally differently?
For example: running something simple like SuperLinter (used on Github Actions) on Bitbucket Pipelines.
I've searched for examples or explanations of the migration process but with little success so far - perhabs they're just not compatible or am I missing something. This is my first time using Bitbucket over Github. Any resources and tips welcome.
They are absolutely unrelated CI systems and there is no straightforward migration path from one to another.
Both systems base their definitions in YAML, just like GitLab-CI, but the only thing that can be reused is your very YAML knowledge (syntax and anchors).
As CI systems, both will start some kind of agent to run a list of instructions, a script, so you can probably reuse most of the ideas of your scripts. But the execution environment is very different so be ready to write tons of tweaks like Benjamin commented.
E.g: about that "superlinter", just forget about it. Instead, Bitbucket Pipelines has a concept of pipes which have a similar purpose but are implemented in a rather different approach.
Another key difference: GHA runs on VMs and you configure whatever you need with "setup-actions". BBP runs on docker containers that should feature most of the runtime and tooling you will need upfront, as "setup-pipes" can not exist. So you will end up installing tooling on every run (via apt, yum, apk, wget...) so as to not maintain and keep updated a crazy amount of images with tooling and language runtimes: https://stackoverflow.com/a/72959639/11715259
Hey guys I work with a team of four people and only two of us really push changes to the Master Branch. As a result we communicate regularly to ensure we are not working on the same file of the solution to avoid merge conflicts. Definitely not the best practices.
However, we will be bringing in more people to work with us and it was proposed to switch to a Continuous Integration and Continuous Deployment environment. I understand the concept of a CICD environment, but creating such environment will be the responsibility of another team.
My task however is coming up with a document for developers on how we should program going forward. Every time I Google something like "CICD Best Practices" I get results for building the CICD environment, however I am interested on best practices developers must follow when programming in a CICD environment as opposed to building a CICD environment.
I am aware of the obvious "Commit your changes often, document your code, use good names". So I am trying to track down any publication or sources (from Microsoft, Amazon, etc) that might mention best practices that are not so obvious. Could anyone point me in the right direction?
Please and thanks!
From my experience the best workflow is to restrict access to the master branch, no one should be allowed to push to it.
All changes should be made through PRs and pass all the tests defined in your cl pipeline.
In that way you can keep your master branch always with working code and deploy it directly to a staging environment. Once you have acceptable code on master you can make a release and have your CD deploy it to production.
Is there an abstraction for defining continuous integration pipelines which can then produce individual config files for particular CI providers? This would be handy especially for projects intended to serve as boilerplate.
Currently I am finding myself needing to manually write and sync both a .gitlab-ci.yml and a .github/workflows/ci.yml.
This is an interesting question, unless you can abstract all your CI scripts into shell scripts, then from what I can see there would lots of periodical porting process between the different CI providers.
Also, different CI providers has its own ideology of the perfect build pipeline as well as predefined setup.
With that being said, I would love to see some utility tools help me migrate the scripts and converge my CI setup into Github Action world.
I have a single repository consisting of various microservices written in node js, I am trying to deploy by triggering a Jenkins pipeline for those microservices where a change is done and trying to maintain only one Jenkins file for the repository. Can this be done? I have been exploring for quite a while now, and I didn't find any useful resources. Any leads to solving this problem statement would be of great help!
Could you suggest any other famous open source CI/CD tools to achieve the same?
I'm going to assume you have all the core Jenkins setup done and don't need help with git hooks or integrations to Github, but feel free to clarify if I'm mistaken. I believe you're specifically asking about how to build/deploy individual microservices housed within a mono-repo.
I personally don't like mono-repos for microservices for reasons like this, and prefer to share common libraries via package managers (now commonly built into version control systems like Github and Gitlab), but if you're dead-set on trying to perform individual builds and releases from a mono-repo then you'll need a script that helps you determine which folders contain changes so you can trigger their respective build pipelines. A simple one that leverages git to evaluate the most recent diff is as follows:
$ git diff --dirstat=files,0 HEAD~1 | sed -E 's/^[ 0-9.]+% //g' | sed -E 's/\/.*$//g'
source: git diff - only show which directories changed
The above will parse the most recent change in the mono-repo and list out the top-level directories containing changes. You'll then be able to iterate through each and run a build pipeline specific to each folder. If each contains a Dockerfile, this may be as simple docker build -f $FOLDER/Dockerfile, but you could also elect to force each folder to contain a custom build script and execute in the same iterator (e.g. $FOLDER/bin/build.sh).
Yeah, you can achieve what you need using Gitlab. We have pretty much the same use case, multiple Lambda functions in the same repo. The idea is not to redeploy everything each time a function is updated. This is a good starting point: https://aarongorka.com/blog/gitlab-monorepo-pipelines/
I've started working on a new Perl module and I've decided that I want to make a Ruby version of it as well (once I finish the Perl version). Do people tend to make separate repositories for each language? Or put them in the same repository?
I can easily see how the two sets of code are different enough to be treated as separate projects. But at the same time it's the same functionality written in two languages, so from that perspective it seems like a single project with two language ports.
What's considered best practice in this situation?
FWIW, I'm using git.
EDIT: I should be more clear here. These aren't modules in the sense of git submodules. They're modules that will be submitted to CPAN and RubyGems. Users of this project will likely be installing it via cpan or gem and then using/requiring it in the normal fashion.
In the course of my group's research, we have a couple repos, some with different technologies in each. We divide the repos by research question and checkout only the projects we are working on, with the repos having a uniform hierarchal directory structure that is the same for all projects. Since we already know the repo directory structural, running scripts and finding data becomes much easier.
I would recommend taking the same approach. The higher the division between the two technologies, the easier it will be to contribute to one of them without being confused by the presence of the other.
In the end ask yourself this: If I were to add another language, would I still keep it in one repo? If the answer is yes, keep doing what you're doing. If not, keep these libraries in two separate repos and manage the projects and contributers distinctly.
my experience in this kind of case is to have 2 smaller git repos for each of the modules and either cloning one branch into the consumer projects repo makes it quite simple. another way is to create a naked clone from the module's repo into the consumer projects repo, then just keep updating it as each module's development progresses. the consumer project should ignore the injected repos.
once other dev clones module A, and/or B, then he/she can just push to consumer project, as permissions allow. this is either a pro or a con depends.