I'm new to continuous integration. I'm interested in systems that would be able to test if the changes that I made to a code break the compilation of the code on a list of different build types.
Properties of code (Which I will call CodeA):
1.) Has dependencies to numerical libraries like SUNDIALS and PETSC
2.) Has dependencies on two other codes (CodeB CodeC) which themselves have dependencies to things like HDF5, MPI, etc.
Is it feasible to use the CI feature of GitLab to set up a system that would be able to build CodeA (linked with CodeB and CodeC) on Linux machines with different system flavors (Ubuntu, OpenSuSe, RHEL, Fedora, etc)?
Most of the examples that I've found of using GitLab for CI have been things like testing to see if HelloWold.cpp compiles if lines are changed on it. Just simple builds with very little external dependency management/integration.
So it sounds like you've got a few really great questions in here. I'll break them apart as I see them and you let me know if this fully answers your question.
How can I build in different flavors of linux?
The approach I would take would be to to use docker files as Connor Shea mentioned in the comment. This allows you to continue using generic build agents in your CI system but test across multiple platforms.
Another option would be to look at how you're distributing your application and see if you could use a snap package. That would allow you to not have to worry about the environment you're deploying to.
How do I deal with dependencies?
This is where it's really useful to consider and artifact repository. Both jfrog's artifactory and sonatype's nexus work wonders here. This will allow you to hook up your build pipeline for any app or library and push an artifact that the others can consume. These can be locked down with a set of credentials that you supply to your build.
I hope this helped.
Related
Both Github Actions and Bitbucket Pipelines seem to fill similar functions at a surface level. Is it trivial to migrate the YAML for Actions into a Pipeline - or do they operate fundamentally differently?
For example: running something simple like SuperLinter (used on Github Actions) on Bitbucket Pipelines.
I've searched for examples or explanations of the migration process but with little success so far - perhabs they're just not compatible or am I missing something. This is my first time using Bitbucket over Github. Any resources and tips welcome.
They are absolutely unrelated CI systems and there is no straightforward migration path from one to another.
Both systems base their definitions in YAML, just like GitLab-CI, but the only thing that can be reused is your very YAML knowledge (syntax and anchors).
As CI systems, both will start some kind of agent to run a list of instructions, a script, so you can probably reuse most of the ideas of your scripts. But the execution environment is very different so be ready to write tons of tweaks like Benjamin commented.
E.g: about that "superlinter", just forget about it. Instead, Bitbucket Pipelines has a concept of pipes which have a similar purpose but are implemented in a rather different approach.
Another key difference: GHA runs on VMs and you configure whatever you need with "setup-actions". BBP runs on docker containers that should feature most of the runtime and tooling you will need upfront, as "setup-pipes" can not exist. So you will end up installing tooling on every run (via apt, yum, apk, wget...) so as to not maintain and keep updated a crazy amount of images with tooling and language runtimes: https://stackoverflow.com/a/72959639/11715259
We recently brought Golang into the company that I work in for general use, but we hit a snag in the roll out because Go can use the go get command to get packages from the internet. Typically when we roll out Java and Python we are able to limit where the developer can pull packages from by pointing them to our internal artifactory.
So with Python we can change where they pull from by altering the pip command to pull from our internal artifactory, and with Java we can alter their settings.xml and pom.xml to point to our internal packages.
I know that during development you can fetch and pull in dependencies into your local then compile them into a standalone binary. What i am looking for is some mechanism that stops people from going out and pulling from the open internet.
Does something like this exist in Go? Can I stop people from going to the internet and go get 'ing packages?
Any help would be greatly appreciated.
It depends on your definition of "roll out", but typically there are three stages:
Development - at this point you can't prevent arbitrary go get calls, apart from putting the development machines behind restrictive proxies or similar technical measures.
Deployment - since Go programs can (should) be deployed as single binaries, go get is not used at all during deployment.
Building deployment artefacts - this is probably your issue:
The usual approach is not to fetch dependencies when building Go programs. Instead, dependencies are fetched during development, and made part of the source tree using vendoring, for example by using the dep tool.
At this point, the build step no longer needs to fetch any dependencies. The choice of which dependencies are allowed now becomes part of the rest of your process, such as code reviews.
Our company currently uses TFS for source control and build server. Most of our projects are written in C/C++, but we also have some .NET projects and wouldn't want to be limited if we need to use other languages in the future.
We'd like to use Git for our source control and we're trying to understand what would be the best choice for a build server. We have started looking into TeamCity, but there are some issues we're having trouble with which will probably be relevant regardless of our choice of build server:
Build dependencies - We'd like to be able to control the build dependencies for each <project, branch>. For example, have <MyProj, feature_branch> depend on <InfraProj1, feature_branch> and <InfraProj2, master>.
From what we’ve seen, to do that we might need to use Gradle or something similar to build our projects instead of plain MSBuild. Is this correct? Are there simpler ways of achieving this?
Local builds - Obviously we'd like to be able to build projects locally as well. This becomes somewhat of a problem when project dependencies are introduced, as we need a way to reference these resources or copy them locally for the build to succeed. How is this usually solved?
I'd appreciate any input, but a sample setup which covers these issues will also be a great help.
IMHO both issues you mention fall really in the config management category, thus, as you say, unrelated to the build server choice.
A workspace for a project build (doesn't matter if centralized or local) should really contain all necessary resources for the build.
How can you achieve that? Have a project "metadata" git repo with a "content" file containing all your project components and their dependencies (each with its own git/other repo) and their exact versions - effectively tying them together coherently (you may find it useful to store other metadata in this component down the road as well, like component specific SCM info if using a mix of SCMs across the workspace).
A workspace pull wrapper script would first pull this metadata git repo, parse the content file and then pull all the other project components and their dependencies according with the content file info. Any build in such workspace would have all the parts it needs.
When time comes to modify either the code in a project component or the version of one of the dependencies you'll need to also update this content file in the metadata git repo to reflect the update and commit it - this is how your project makes progress coherently, as a whole.
Of course, actually managing dependencies is another matter. Tons of opinions out there, some even conflicting.
My case is about having too much complex project configuration logic inside Jenkins jobs definition and in time this becoming harder and hard to deal with. This also prevents you from easily execute build jobs under other build/CI tools.
If those projects would be Java based anyone would probably tell me to use maven as I could put most of the things inside the pom.xml files and have them with the project. Still, in my case is more about C/C++ or even .NET projects for which the all the build scripts are usually in bash (cygwin being a dependency on Windows).
I do know that theoretically I could code the parts that are now inside jenkins job configuration in those bash files but this would clearly require significant effort and would be really hard to tune them to allow to enable and disable different steps based on external conditions.
So, what I am trying here is to achieve a high level of independency regarding the build system, so if I want I could switch it in the long future.
What would you recommend as a solution for that? Obviously I need something that can be used multiplatform and not tightened to a specific build system.
Does it make sense to use maven for that, even if those projects are not Java ones? Personally I am not a big fan of XML configuration files, YAML, JSON and INI being seen as more friendly.
What kind of extra logic existing in Jenkins configuration are we talking about?
One would deployment, as I want to be able to deploy to Nexus or similar repositories, executing tests, code coverage and maybe posting the results somewhere.
As a sidenote, looking at Travis configuration files makes me wonder why Jenkins didn't go for such approach.
Look at Groovy. Jenkins allows direct Groovy code manipulating almost everything. A Groovy script could be used to take care of everything from project specific configuration, and it could even be checked in together with the source code. Then in the Jenkins job, you just have a single build step to call the Groovy script.
The above suggestion, however, is very Jenkins dependent.
Another possibility is an Ant script. The AntExec plugin allows to execute Ant script, along with ant-contrib if needed, using the same tools installation process that the rest of Jenkins use. Therefore, you don't need to worry about Ant being installed on the node: Jenkins will take care of it on demand.
The benefits of the Ant script is that it's not tied to Java concepts as Maven is, it's cross platform (Windows and Linux), and just like the Groovy script example above, it can be checked in along with the rest of the source code.
I have recently been charged with building out our "software infrastructure" and so I am putting together a continuous integration server.
After a build completes would it be considered bad form for the CI system to check in some of the artifacts it creates into a tag so that it can be fetched easily later (or if the build breaks you can more easily recreate the problem.)
For the record we use SVN and BuildMaster (free edition) here.
This is more of a best practices question rather than a how-to question. (It is pretty easy to do with BuildMaster)
Seth
If you believe this approach would be beneficial to you, go ahead and do it. As long as you maintain a clear trace of what source code was used to build each artifact, you'll be fine.
You should keep this artifact repository separated from the source code repository.
It is however a little odd to use a source code repository for this - these are typically used for things that will change, something your artifacts most definitely should not.
Source code repositories are also often used in a context where you want to check out "everything", for example the entire trunk. With artifacts you are typically looking for a specific version, and checking out all of the would only be done if exporting them to some other medium.
There are several artifact repositories specialized for this, for example Artifactory or Apache Archiva, but a properly backed up file server will thought-through access settings might be a simple and good-enough solution.
I would say it's a smell to check in binaries as a tag. Your build artifacts should be associated with a particular build version in your build system, and that build should be associated with a particular checkin. You should be able to recreate the exact source code from that information. If what you're looking for is a one-stop-function to open the precise source-code revision that generated the broken build, I'd suggest that you invest some time into building a Powershell module that will do that for you.
Something with a signature like:
OpenBuild -projectName "some project name" -buildNumber "some build number"