I've been looking through the site and I have found some information with regards to this topic, but most of the information is old and possibly outdated.
example: Continuous Integration tools
We are: We're a SaaS product with a microservice (200+) architecture.
We have: We currently do our building through bamboo, and we use nexus as an artifact manager with proper versioning. We deploy those artifacts using bamboo to many different machines. For our frontend deployment we build our code through continua and use AWS codedeploy to handle the deployment. We use Bitbucket and Jira for our development. We have done a POC with bitbucket pipelines but we were lacking proper version management there as well as proper environment management. Setting up 10 servers for every repository manually is just something that we don't want to do.
We want: Since bamboo is EOL next year and since there are many alternatives with different levels of complexity we are currently unsure about the tools that are most suited to our needs. We are currently running everything on dedicated linux machines, but we want to switch to docker containers in AWS in the near future. Support for running gulp scripts etc. would be great since that could help us move from continua and bamboo to one single solution.
The setup of bamboo has been a struggle in the past due to difficulties with the software itself. A nice balance between features and complexity would be best. Does anybody have experience with one or more of the options out there? Some that come to mind are CircleCi, teamCity, GitLab, Jenkins and AWS codePipeline.
Many thanks,
Kenny
Bamboo doesn't EOL next year, but Atlassian forces to switch from perpetual licenses to DC licences to be renewed every year. You can get discount prices when switch to Server to DC licenses. See details at https://www.atlassian.com/licensing/data-center
I would propose Kraken CI. It is open-source and can work on-premise but in the cloud as well. In the cloud, it has support for AWS and Azure, and can do autoscaling depending on a number of tasks.
If you are interested please contact me.
Related
I am a developer from who wants to create an ecosystem around micro services. My research led to your software projects, which are outstanding in many respects.
Unfortunately, one of the components I couldn't get running for an initial review was the portal.
The build failed due to a missing light-4j version (1.5.29).
The light-4j master branch is at 1.5.23, hence I checked out the portal at a version that meets the light-4j version. With this the docker-compose-hybrid.yml script failed, due to other missing libraries. Considering the fact that I even reverted to an older version of the portal sources, I am almost sure that I am on the wrong track.
Do you have any advice for me how to get this solved?
Thank you in advance.
Thanks a lot for your interest in the light platform. The light-portal is still in heavy development in develop branch which is depending on the develop branch of light-4 and other libraries. The easiest way to build it is through light-bot which is our own DevOps tool for microservices as you can see you are dealing with too many dependencies and most DevOps tools on the market can only deal with one repository each time.
https://github.com/networknt/light-config-test/tree/master/light-bot/develop-build/build-portal
Also, please be aware that light-portal services are built on top of light-hybrid which is a serverless framework. The build process just creates small jar files and copy to the read and write service folders. You need to start a compose to start the two services to load all the services.
The following the docker-compose to start the light-portal locally.
https://github.com/networknt/light-config-test/tree/master/light-portal
I am starting to write a light-portal tutorial but there are still a lot of topics are missing. Please let me know if you see any gaps so that I can add more info into it.
https://doc.networknt.com/tutorial/portal/
The mail might not be the best channel as the communication is on a private channel. In the future, you can ask questions on gitter as other people might know the answer when our team members are not available immediately. Also, the answers on the public channel might help other users to learn the platform.
https://gitter.im/networknt/light-4j
The best part of UrbanCode Deploy is it models a component based architecture application, and its deployment environment so well that everybody can understand in 10 minutes. Very initiative, flexible and powerful. Don't know if there is another tool does this well.
Jenkins Pipeline can orchestra the Continuously Delivery workflow at the higher level to include the build, test, etc.
Does it make sense?
There's a new UCD plugin for Jenkins that adds nice integrations with the Jenkins 2.0 pipeline. I'm going to poke the developers since there doesn't seem to be a nice video showing it, but there is documentation (and a link to the plugin) out here:
https://developer.ibm.com/urbancode/docs/jenkins-build-step-integration-with-ibm-urbancode-deploy/
I think the idea is that you can use Jenkins pipeline to govern the flow of a build through early test environments, while UCD owns the late test environments / production when the pipeline operates more at the snapshot level. Would love your feedback!
Today, in my production environment, i use Jenkins to manage my build (like a "build pipeline" with some tests) and put all my build results into Urbancode Code station. Urbancode is doing all my deploy work perfectly, the integration with Jenkins is beautiful, easy and fast. I have read some articles about Jenkins delivery pipeline and do not recommend use it.
Check it out
https://www.thoughtworks.com/radar/tools
We use Perforce to manage our Software Configuration.
My question is how to achieve continuous integration in Perforce?
Are there any tools built on top of Perforce to run continuous integration?
The official one is Electric Cloud, through its "Continuous Delivery" feature. See the press release from May 2012.
However, the tool itself is only one part of the process. The other is how you access the code with Perforce: the blog post "Massive Automation: Agile, Continuous Integration, and Perforce" details how:
Automation, particularly at this scale, puts a tremendous load on the SCM server. Build automation can account for well over 50% of the load on a Perforce server, and that can start impacting the end user experience
So it insists on the presence of replica in order to alleviate the charge put on the central Perforce server.
You can see another process for CI with Perforce in the article "Compartmentalized Continuous Integration".
Perforce has a full list of their Build partners (there are ahem many) here:
http://www.perforce.com/product/components/integrations/build_tools
Jenkins is an awesome and widely used continuous integration tool.
There's a perforce plugin (among lots of other valuable plugins).
Our team has a full licence for the TeamCity server, as well as 7 additional agents. Another unrelated team has reached the limits of their free TeamCity licence and is eyeing our licences up.
The powers that be think it's a good idea to run both teams using the same enterprise licence, which means that we'd be hosting the TeamCity configurations on the same server, and either sharing agents or somehow assigning some agents to one team, some to another.
One concern I have is that configuring an agent to only accept certain builds is difficult - our team has hundreds of build configurations, and we create new ones all the time. To limit an agent to certain builds, you have to fully specify the whitelist. So maintaining the agents such that we have full use of some agents, and the other team has full use of theirs will be a pain. On the other hand, just using one pool of agents means now you have arguments over priority and starvation, etc.
Does anyone have any experience of this? Is it a workable solution? How do you configure agents to reserve them for a particular team? How do you configure the server so that each team only sees their own projects, build configurations and agents? Basically what we'd want is complete separation of the projects, just using the same TeamCity server and agents.
As a gut feeling it doesn't look like a good idea...
edit: As an aside, does Hudson do this better? The ivory tower architects want us to change from TeamCity to Hudson because other people are using Hudson. If I tell them this sharing TeamCity won't work, the Hudson camp will probably use it as a stick to beat us with. Joy.
Not sure what version of TeamCity you're using but the newly released TeamCity v7.0 now has a new Agent Pool feature that provides a much easier way to distribute agents. It may be of interested to you, check out the What's New section or the Agent Pools docs for more info.
I had a similar issue with our two departments starting to share the same TeamCity instance to save the expenses of additional licenses. I must admit we didn't really have any issues apart from our agents were now twice as busy.
I enabled Per-project permissions on the Global Settings page and created 2 user groups, one for 'us', and the other for 'them'. You can then configure each group's roles accordingly. If a group does not have the Project Viewer role for a project then it does not appear for them - a great way to only display necessary projects to the group; but there are plenty of other role options to use.
I have never used Hudson so can't compare unfortunately. I should really try it out but as I've always got on so well with TC I've never had a reason too.
You can make builds on run on certain agent, from the build configuration of each build in the agent requirements section, thereby limiting any build configuration to certain agents.
For example if your agent for one team is teamcity1 you can specify:
system.agent.name does not equal teamcity1
So it will never run on that agent.
That way you can at least copy build configurations and they will run on seperate agents without the fiddle agent configuration.
The other team can create a new Teamcity server, and it will have its own new set of free build configurations and agents.
We don't do this any more, but we used to split our agents into pseudo-pools so we could reserve some for compilations and others for automated tests (because automated test jobs can swamp the grid). We added a "can_run_tests" property to the test agents, and made those builds require that property as an agent condition. It worked great, and it's the sort of thing you can bake into the AMI for a set of cloud agents.
What we do now is to make the compilation and test builds require on different AMIs, which does essentially the same thing.
I am referred to Hudson today.
I have heard about continuous integration before, but I have no idea what the heck is a ci-server.
Hudson is really easy to install in Ubuntu and in several minutes I managed to set up an instance of it.
But I don't quite understand the workflow of a ci-server, or how am I supposed to use it?
Please tell me if you have experience about ci, thanks in advance.
Edit:
I am currently using Mercurial as my SCM, and I wonder what is the right way to use it with Hudson.
I have installed the Mercurial Plugin of Hudson, and I create a new job with a local repository. When I commit in the repository the Hudson job is built with the latest version of my source code.
If what I used is a remote repository, what's the workflow like?
Is it something like the following?
Set up a Hudson job with the repository
Developer makes a local clone of the repository
Developer commit and push changes
The remote repository update with the incoming changeset
Run a Hudson build
There may be something I misunderstanded at all, please help me point it out.
Continuous Integration is the process of "integrating software" continuously i.e. as frequently as possible (ultimately after each set of changes) to avoid any big-bang integration and all subsequent problems by getting immediate feedback.
To implement Continuous Integration, you first need to automate the build of your software (where build means of course compiling sources, packaging them, but also compiling tests, running the tests, running quality checks, etc, anything that will help to get feedback on the health of your code). Then you need to trigger the build on the latest version of the sources on a particular event (a change in the repository, a temporal event), to generate reports and to send notifications upon failure (by mail, twitter, etc).
And this is precisely the responsibility of a CI engine: offering trigger mechanisms, being able to get the latest version of the sources, running the build, generating and publishing reports, sending notifications. CI engines do implement this.
And because running a build is CPU and Disk intensive, CI engines usually run on a dedicated machine (or even a farm of machines if you want to build lots of projects).
Back to your question now. Once you've got Hudson running, configure it (Manage Hudson > Configure System): setup the JDK, build tools, etc. Then setup an Hudson Job and follow the steps: configure the location of the source repository, the build tool, the trigger, a notification channel and you're done (you can do more complex things but that's a start).
For more details on the setup, check:
The official Use Hudson guide for more details. << START HERE
Continuous Integration with Hudson - Tutorial.
Spot defects early with Continuous Integration.
Martin Fowler's overview of continuous integration is one of the canonical references. In my opinion, using automation to make sure your code base is healthy is one of the most useful things that you can set up.
Update Sorry that I didn't have much time earlier to expand on my reply. #Pascal_Thivent is right that in order to effectively use CI, you need to be able to automate your builds, tests, etc. CI is actually a good forcing function for this. For me, it's one of those little warning flags if I start to think that it would be too painful to put a build into Hudson. It means that something is not quite right.
What I like about Hudson is that it's flexible enough to accommodate different workflows. We use it for both builds / unit tests and releases. And it eliminates a lot of the worry about certain release procedures only working in one person's environment.
What I don't like about Hudson is that it is occasionally unstable when new builds break plugins. I've had a couple of upgrades (2 out of 10 or so) go bad because of incompatibilities. I do two things now:
I never upgrade my team's Hudson server to the latest and greatest right away. I generally only upgrade when there are significant new features, or bug fixes.
I now have a basic Hudson instance set up with all my plugins on a virtual machine with some dummy builds that I fire up to test out any new upgrades before doing it on the public server.