I just got to know about Travis CI and went through some of their docs. It seems to be a nice solution for open source projects.
With my reading so far through Travis docs, I am doubtful whether I will be able to connect it to my personal hardware in some manner.
I am working on some IoT related project written in C/C++ hosted on github. Building and publishing images on artifactory, on Travis CI should not be a problem. But when it comes to testing, definitely it cannot be tested on their (Travis's) hardware. The binaries need to be put on my development board (raspberry pi) and then test cases should be executed. Once test suit finishes, Travis CI should be notified of the results.
Is Travis allows such functionality? If not, then it would be a great limitation.
But when it comes to testing, definitely it cannot be tested on their
(Travis's) hardware
By this statement you answered your own question. If you want to use Travis for showing test status publicly, your best chance is running the tests on premises using Jenkins, GitLab CI or any other CI server and then offering an API to Travis (e.g. a file with the test results). All Travis would then do is fetching and showing the test results.
If you want to control things from within Travis, you could trigger builds from Travis using a HTTP call and then wait for the remote build to finish before showing its results. Both Jenkins and GitLab CI offer remote triggering. However, this requires your CI servers to be publicly accessible, which might be a security risk (e.g. people triggering non-stop builds).
TravisCI is primarily useful for testing libraries and projects that can be run and tested on common systems (linux, etc.) I don't believe there is a way to run TravisCI locally however, MinionCI seems to offer a solution for running a CI server locally following a style similar to TravisCI, check it out here.
Related
I have yet to find clear guidance on semantically versioning software releases using Azure DevOps (Server) CI/CD.
My basic understanding is that I would set up a CI pipeline for our team so that we get the benefits of build failure notifications, test execution, code coverage, static code analysis and even more goodies.
The CD pipeline picks up on the CI pipeline, rolling out the artifacts on the different environments upon their completion.
Using this approach seems to not make sense to me. What about builds that fail because a dev didn't pay attention or that the team wants to discard? Such builds won't find their way into production but might use up version numbers, leading to gaps in our versioning scheme.
What is your experience or approach for semantically versioning software releases using CI/CD pipelines? Do you cherry-pick builds? Do you have a separate build pipeline for building releases?
Builds that fail don't get deployed. For builds that succeed but fail QA or integration testing when deployed to lower environments, you can put approval gates so that someone has to approve the release before it proceeds to higher environments.
If you build version 1.0.1 of your application and deploy it to dev and it's no good, that doesn't mean that version 1.0.1 doesn't exist. It exists. It was comprised of specific code assets. It was bad, and your users will never see it, but that's fine. If the users see version 1.0.1 jump to 1.0.94, why does it matter?
What is the point of using a Continuous Integration system to test your code if you already have a system like Husky running that allows you to test you code before pre-commit and pre-push?
Pre-commit and pre-push hooks are great for quick operations and tests. Sometimes you can even setup a hook in your IDE that will run quick unit tests every time you save a file. But usually you have multiple suites of tests and unlike unit tests functional, integration and performance tests often take longer time to run, which is not feasible for hooks.
Also, you want to run your tests in the same environment where you build your deliverables, which is usually not your local machine.
Another reason to use CI system is to run post-merge tests to verify that there are no issues introduced by multiple parallel merges.
All-in-all, the more tests you run, the better and a CI system allows you to run both pre-merge tests usually triggered by some sort of pull request hook and post-merge tests. And all of that in a controlled reliable environment.
I'm not really interested about whether it passes in your local environment, where you may have a different version of some dependent library on your environment path. I want to know for sure that anyone's contributions don't break the software when linked against the specific library versions that we ship with.
One reason to test using a Continuous Integration platform like Travis would be to assure developers haven't circumvented their own local development environment's testing git hooks.
CI is not only tests, it's a lot more, but the test stage is of course a very important part of the flow.
As you said in your own answer, local environments could be changed, the tests on the CI could have stricter settings, the environment you test on could be more like the environment that the end-user uses (say, set versions of software or even hardware).
Say for example that you develop a PHP package. The package have support for everything between php 5.6 and 7.2, it should also support multiple types of operating systems and should behave differently if ext/open_ssl is installed or not. A local test suite would rarely have a setup allowing the developer to test each of the possible versions on each of the required platforms, but a test suite set up in a CI pipeline could.
And honestly, it's always a good idea to test one more time, just to be safe! ;)
In certain useful and reasonable workflows, it is acceptable to commit and push broken commits (though not to the master branch). Preventing such workflows with git hooks is annoying.
Rebasing or merging as an example does not run hooks again, even though files are changed.
Hooks are also very difficult to get right. They check a local state which might not be what gets pushed (if certain files are present that are not in git).
CI servers also provide a stable predictable environment. E.g. consider a CI server having Linux and developers using MacOs laptops. The git hooks run on MacOs, which has case insensitive file system, allowing tests to pass even if filenames are wrong.
Hooks also punish diligent developers who run checks manually before committing, because tests are just run again one more time.
Each professional project should have CI. The real question is why any project should maintain annoying slow fragile broken local hooks when you already have CI.
Use hooks only for private toy projects.
The best part of UrbanCode Deploy is it models a component based architecture application, and its deployment environment so well that everybody can understand in 10 minutes. Very initiative, flexible and powerful. Don't know if there is another tool does this well.
Jenkins Pipeline can orchestra the Continuously Delivery workflow at the higher level to include the build, test, etc.
Does it make sense?
There's a new UCD plugin for Jenkins that adds nice integrations with the Jenkins 2.0 pipeline. I'm going to poke the developers since there doesn't seem to be a nice video showing it, but there is documentation (and a link to the plugin) out here:
https://developer.ibm.com/urbancode/docs/jenkins-build-step-integration-with-ibm-urbancode-deploy/
I think the idea is that you can use Jenkins pipeline to govern the flow of a build through early test environments, while UCD owns the late test environments / production when the pipeline operates more at the snapshot level. Would love your feedback!
Today, in my production environment, i use Jenkins to manage my build (like a "build pipeline" with some tests) and put all my build results into Urbancode Code station. Urbancode is doing all my deploy work perfectly, the integration with Jenkins is beautiful, easy and fast. I have read some articles about Jenkins delivery pipeline and do not recommend use it.
Check it out
https://www.thoughtworks.com/radar/tools
I've been wondering how you go about doing CI-style testing when you're dealing with physical devices.
I imagine you have a suite of tests, and a pool of devices against which they can be run.
Additionally:
Some tests may require specific device models.
Some tests may require the use of more than one device.
What CI servers have support for this?
I'm still interested in those which have partial support, either natively or through plugins, as I'm interested in how it's done.
Continuous Integration enables a team integrate & test their work frequently. Automated builds are meant to compile, link and run unit-tests. You want your CI to run fast, especially if you run it at every check-in. That is why you would want to restrict CI activities to simple confirmation of the build and unit tests alone. What you're asking seems more-along the lines of quality assurance (QA) testing...and having QA failures mixed into your CI efforts would detract development efforts from progressing.
As such, I'm more under the impression activities associated with CI are not dependent upon the final physical machine said work may eventually be migrated to.
Now...this doesn't mean you CAN'T take the CI-compiled package and run it against some final target-machine....but again...that is really considered a seperate activity.
This seems to be re-enforced in the following article by Martin Fowler.
Notice he doesn't talk about the final targeted devices...only the build machine.
I can suggest Test Manager that is part of the TFS suite of Microsoft. I have not tried it with many different environments apart from windows based though I know there are many connectors. For windows based environments I believe it will satisfy most needs.
I use it for nightly builds to perform smoke tests (Turn it on, see if any smoke comes out) but you have to be careful to keep tests small in order to have them finished in a matter of hours and not days, if you want it to be part of your CI.
Then when you have a good enough quality you can proceed onto regression tests and integration tests if needed.
I wouldn't get too caught up in what a CI system is supposed to do or not do. Instead I would focus on the problem you are trying to solve. It sounds like that problem is to facilitate development on multiple platforms. You can use the concept of Continuous Integration and add to it successfully address the issue. I know, because I've done it in the past.
I implemented a build system for code that needed to compile and test successfully on 4 different platforms (nt, wince, linux-arm, linux-x86). The CI server would:
Used a linux and winnt build server compilation (and cross compilation)
The compiled tests and supporting libs would then be copied to the appropriate devices and an automated test run executed.
After the test suite was completed the log would be copied back, (or it was written to a network mounted fs)
If the test suite was successful we would tag the source, and package the libs and executables.
This same platform was reused for developer verification before commits. Developers would run a partial build and test (only updated source would be recompiled and those tests rerun). The CI would execute a full build (from scratch).
Our build were pretty fast because we had a proper DAG for build dependencies. This allowed for concurrent compilation within a platform build. Each platform build was also concurrent. As a result partial builds took a few seconds, full builds took ~30 minutes. Our build servers were quite beefy (optimized for fast compiles) and the codebase was of moderate size (I don't remember the stats).
I am referred to Hudson today.
I have heard about continuous integration before, but I have no idea what the heck is a ci-server.
Hudson is really easy to install in Ubuntu and in several minutes I managed to set up an instance of it.
But I don't quite understand the workflow of a ci-server, or how am I supposed to use it?
Please tell me if you have experience about ci, thanks in advance.
Edit:
I am currently using Mercurial as my SCM, and I wonder what is the right way to use it with Hudson.
I have installed the Mercurial Plugin of Hudson, and I create a new job with a local repository. When I commit in the repository the Hudson job is built with the latest version of my source code.
If what I used is a remote repository, what's the workflow like?
Is it something like the following?
Set up a Hudson job with the repository
Developer makes a local clone of the repository
Developer commit and push changes
The remote repository update with the incoming changeset
Run a Hudson build
There may be something I misunderstanded at all, please help me point it out.
Continuous Integration is the process of "integrating software" continuously i.e. as frequently as possible (ultimately after each set of changes) to avoid any big-bang integration and all subsequent problems by getting immediate feedback.
To implement Continuous Integration, you first need to automate the build of your software (where build means of course compiling sources, packaging them, but also compiling tests, running the tests, running quality checks, etc, anything that will help to get feedback on the health of your code). Then you need to trigger the build on the latest version of the sources on a particular event (a change in the repository, a temporal event), to generate reports and to send notifications upon failure (by mail, twitter, etc).
And this is precisely the responsibility of a CI engine: offering trigger mechanisms, being able to get the latest version of the sources, running the build, generating and publishing reports, sending notifications. CI engines do implement this.
And because running a build is CPU and Disk intensive, CI engines usually run on a dedicated machine (or even a farm of machines if you want to build lots of projects).
Back to your question now. Once you've got Hudson running, configure it (Manage Hudson > Configure System): setup the JDK, build tools, etc. Then setup an Hudson Job and follow the steps: configure the location of the source repository, the build tool, the trigger, a notification channel and you're done (you can do more complex things but that's a start).
For more details on the setup, check:
The official Use Hudson guide for more details. << START HERE
Continuous Integration with Hudson - Tutorial.
Spot defects early with Continuous Integration.
Martin Fowler's overview of continuous integration is one of the canonical references. In my opinion, using automation to make sure your code base is healthy is one of the most useful things that you can set up.
Update Sorry that I didn't have much time earlier to expand on my reply. #Pascal_Thivent is right that in order to effectively use CI, you need to be able to automate your builds, tests, etc. CI is actually a good forcing function for this. For me, it's one of those little warning flags if I start to think that it would be too painful to put a build into Hudson. It means that something is not quite right.
What I like about Hudson is that it's flexible enough to accommodate different workflows. We use it for both builds / unit tests and releases. And it eliminates a lot of the worry about certain release procedures only working in one person's environment.
What I don't like about Hudson is that it is occasionally unstable when new builds break plugins. I've had a couple of upgrades (2 out of 10 or so) go bad because of incompatibilities. I do two things now:
I never upgrade my team's Hudson server to the latest and greatest right away. I generally only upgrade when there are significant new features, or bug fixes.
I now have a basic Hudson instance set up with all my plugins on a virtual machine with some dummy builds that I fire up to test out any new upgrades before doing it on the public server.