I am new to Jenkins and know how to create Jobs and add servers for JAR deployment.
I need to create deployment job using Jenkins which takes a JAR file and deploys it of 50-100 servers.
These servers are categorized in 6 categories. there will be different process run on each server but same JAR will be used.
Please suggest what is the best approach to create JOB for this.
As of now, the servers are less(6-7), I have added each server to Jenkins and using command execution over ssh for process execution. But for 50 servers this is not the possibility.
Jenkins is a great tool for managing builds and dependencies, but it is not a great tool for Configuration Management. If you're deploying to more than 2 targets (and especially if different targets have different configurations), I would highly recommend investing the time to learn a configuration management tool.
I can personally recommend Puppet and Ansible. In particular, Ansible works over an SSH connection to the target (which it sounds like you have) and requires only a base Python install.
Related
My organization's website is a Django app running on front end webservers + a few background processing servers in AWS.
We're currently using Ansible for both :
system configuration (from a bare OS image)
frequent manually-triggered code deployments.
The same Ansible playbook is able to provision either a local Vagrant dev VM, or a production EC2 instance from scratch.
We now want to implement autoscaling in EC2, and that requires some changes towards a "treat servers as cattle, not pets" philosophy.
The first prerequisite was to move from a statically managed Ansible inventory to a dynamic, EC2 API-based one, done.
The next big question is how to deploy in this new world where throwaway instances come up & down in the middle of the night. The options I can think of are :
Bake a new fully-deployed AMI for each deploy, create a new AS Launch config and update the AS group with that. Sounds very, very cumbersome, but also very reliable because of the clean slate approach, and will ensure that any system changes the code requires will be here. Also, no additional steps needed on instance bootup, so up & running more quickly.
Use a base AMI that doesn't change very often, automatically get the latest app code from git upon bootup, start webserver. Once it's up just do manual deploys as needed, like before. But what if the new code depends on a change in the system config (new package, permissions, etc) ? Looks like you have to start taking care of dependencies between code versions and system/AMI versions, whereas the "just do a full ansible run" approach was more integrated and more reliable. Is it more than just a potential headache in practice ?
Use Docker ? I have a strong hunch it can be useful, but I'm not sure yet how it would fit our picture. We're a relatively self-contained Django front-end app with just RabbitMQ + memcache as services, which we're never going to run on the same host anyway. So what benefits are there in building a Docker image using Ansible that contains system packages + latest code, rather than having Ansible just do it directly on an EC2 instance ?
How do you do it ? Any insights / best practices ?
Thanks !
This question is very opinion based. But just to give you my take, I would just go with prebaking the AMIs with Ansible and then use CloudFormation to deploy your stacks with Autoscaling, Monitoring and your pre-baked AMIs. The advantage of this is that if you have most of the application stack pre-baked into the AMI autoscaling UP will happen faster.
Docker is another approach but in my opinion it adds an extra layer in your application that you may not need if you are already using EC2. Docker can be really useful if you say want to containerize in a single server. Maybe you have some extra capacity in a server and Docker will allow you to run that extra application on the same server without interfering with existing ones.
Having said that some people find Docker useful not in the sort of way to optimize the resources in a single server but rather in a sort of way that it allows you to pre-bake your applications in containers. So when you do deploy a new version or new code all you have to do is copy/replicate these docker containers across your servers, then stop the old container versions and start the new container versions.
My two cents.
A hybrid solution may give you the desired result. Store the head docker image in S3, prebake the AMI with a simple fetch and run script on start (or pass it into a stock AMI with user-data). Version control by moving the head image to your latest stable version, you could probably also implement test stacks of new versions by making the fetch script smart enough to identify which docker version to fetch based on instance tags which are configurable at instance launch.
You can also use AWS CodeDeploy with AutoScaling and your build server. We use CodeDeploy plugin for Jenkins.
This setup allows you to:
perform your build in Jenkins
upload to S3 bucket
deploy to all the EC2s one by one which are part of the assigned AWS Auto-Scaling group.
All that with a push of a button!
Here is the AWS tutorial: Deploy an Application to an Auto Scaling Group Using AWS CodeDeploy
Three questions regarding deployment of modules to Spring XD container:
For certain sources and sinks it's necessary to say to which container a module should be deployed. Let's say we have a lot of containers on different machines, and we want to establish a stream reading a log file from one machine. The source module of type tail has to be deployed to the container running on the machine with the log file. How can you do that?
You may want to restrict the execution of modules to a group of containers. Let's say we have some powerful machines for our batch processing with containers on it, and we have other machines where our container runs parallel to some other processes only for ingesting data (log files etc.). Is that possible?
If we have a custom module, is it possible to add the module xml and the jars just to certain containers, so that those modules are just executed there? Or is it necessary that we have the same module definitions on all containers?
Thanks!
You bring up excellent points, we have been doing some design work around these issues, in particular #1 and #2 and will have some functionality here in our next milestone release in about 1 month time.
In terms of #3, the model for resolving the jars that are loaded in the containers requires the local file system or a shared file system to resolve the classpath. This is also something that has come up in our prototypes of using Spring XD on the CloudFoundry PaaS and we want to provide a more dynamic/at runtime ability to located and load new modules. No estimate on when that will be address.
Thanks for questions!
Cheers,
Mark
We are migrating from CruiseControl.NET to Jenkins just to be in sync with a partner so we don't have two different CI scripts. We are trying to setup Jenkins to do something similar to what we had CruiseControl doing which was have a centralized server invoke projects (jobs in jenkins) on remote build machines.
We have multiple build machines associated to a single project so when we build the project from the centralized CI server it would invoke the projects on the remote CI servers. The remote CI servers would pull the version from the centralized CI server project.
In CruiseCruise control we setup a project that would do a forceBuild on the remote projects. The projects on the build machines used a remoteProjectLabeller to retrieve the version number so they were always in sync.
To retrieve the master build number:
<labeller type="remoteProjectLabeller">
<project>MainProject</project>
<serverUri>tcp://central-server:21234/CruiseManager.rem</serverUri>
</labeller>
To invoke the remote projects:
<forcebuild>
<project>RemoteBuildMachineA</project>
<serverUri>tcp://remote-server:21234/CruiseManager.rem</serverUri>
<integrationStatus>Success</integrationStatus>
</forcebuild>
So far in jenkins i've setup a secondary server as a slave using the java web start but I don't know how I would have the master jenkins invoke the projects setup on the slaves.
Can I setup Jenkins to invoke projects (jobs) on slaves?
Can I make the slaves pull the version number from the master?
EDIT -
Let me add some more info.
The master, and remote build machine slaves are all running Windows.
We had the central master CruiseControl kick off the remote projects at the same time so they ran concurrently and would like to have the same thing with jenkins if possible.
Jenkins has the concept of build agents, which could perhaps fit your scenario better - there's a master that triggers the build and slaves that perform it. A build can be then restricted to some categories of slaves only (e.g. if it depends on a specific software, not present on all agents). All data is managed centrally by the master, which I believe is what you are trying to achieve.
In Jenkins it is not possible to trigger a build on a slave, i.e. where a build runs is not controlled by the one who triggers it. It is controlled by the settings of the job itself. Each job has a setting called "Restrict where this job can run".
In your case you would probably have two jobs: A and B. A would be restricted to run on "master" and B would be configured to run on "slavename". Then all that is left to do is for A to trigger B.
But you had further constraints: You want A and B check out the same version from version control and you want A and B to run in parallel. There are many ways to accomplish that but the easiest is probably to define a multi-configuration job.
There is no way to turn an existing free-style job into a multi-configuration job, so you will have to make a new job.
Choose New job
Choose Build new multi-configuration project. Add a name.
Under Configuration Matrix, open the "Add axis" drop down.
Choose Slaves
Check master and the slave
Add the SCM information and build step(s)
When the job runs, it runs on both the master and the slave. Jenkins makes sure they build from the same source version.
From the /jenkins/computer url, you can add, remove, and reconfigured "nodes" which are either local or remote "build agents".
The Jobs can then be constrained to run on particular build agents, or follow various rules to select the appropriate build agent out of the available agents.
I was thinking about Jenkins too much like CruiseControl where the job is defined on the remote machine. So in Jenkins the remote projects are defined on the master and delegated to a remote machine via an agent.
I used the Java Web Start agent installed as a windows service on the remote machines. To have specific jobs run on specific remote machines I defined each remote node with a unique label in its slave configuration. To bind specific jobs to specific slaves I used the slave's label in each job configuration ("Restrict where this project can be run").
To trigger the jobs with a single master job I created a free style job that only is set to "Build other projects" and provided a comma separated list or project names. This job builds the downstream jobs in parallel.
I'm still looking for a way to send a master build number to the downstream jobs to keep them in sync always. (This is used to version DLLs and such.)
My company has thousands of server instances running application code - some instances run databases, others are serving web apps, still others run APIs or Hadoop jobs. All servers run Linux.
In this cloud, developers typically want to do one of two things to an instance:
Upgrade the version of the application running on that instance. Typically this involves a) tagging the code in the relevant subversion repository, b) building an RPM from that tag, and c) installing that RPM on the relevant application server. Note that this operation would touch four instances: the SVN server, the build host (where the build occurs), the YUM host (where the RPM is stored), and the instance running the application.
Today, a rollout of a new application version might be to 500 instances.
Run an arbitrary script on the instance. The script can be written in any language provided the interpreter exists on that instance. E.g. The UI developer wants to run his "check_memory.php" script which does x, y, z on the 10 UI instances and then restarts the webserver if some conditions are met.
What tools should I look at to help build this system? I've seen Celery and Resque and delayed_job, but they seem like they're built for moving through a lot of tasks. This system is under much less load - maybe on a big day a thousand hundred upgrade jobs might run, and a couple hundred executions of arbitrary scripts. Also, they don't support tasks written in any language.
How should the central "job processor" communicate with the instances? SSH, message queues (which one), something else?
Thank you for your help.
NOTE: this cloud is proprietary, so EC2 tools are not an option.
I can think of two approaches:
Set up password-less SSH on the servers, have a file that contains the list of all machines in the cluster, and run your scripts directly using SSH. For example: ssh user#foo.com "ls -la". This is the same approach used by Hadoop's cluster startup and shutdown scripts. If you want to assign tasks dynamically, you can pick nodes at random.
Use something like Torque or Sun Grid Engine to manage your cluster.
The package installation can be wrapped inside a script, so you just need to solve the second problem, and use that solution to solve the first one :)
Is there a way to have my hudson slaves used by multiple hudson masters?
A bit of background info:
My build guy has set-up separate hudson masters to do the deployment and testing of our solution into different test environments. My tests are run on hudson slaves (I have 4 slaves). These slaves are associated to one specific hudson master. I want the slaves to be available for use by any of the hudson masters.
I believe the build guy chooses to use multiple hudsom masters to manage the number of jobs on each master. His set-up for one environment has 8 view tabs therefore 5 environments would mean 40 tabs. Unfortunately as is common, the solution to one problem creates another.
Yes, you can add the slaves to both Hudson masters. The problem is that each master will not be aware of the resource utilization by the other master, so you'll have to figure out some mechanism for that, such as reducing the number of executors.
Even better would be to combine the two Hudson masters into a single Hudson instance. Your question doesn't explain the motivation for having two masters.
As I cannot comment above I'll try an answer.
I think you can have several independent slaves on same machine, each attaching and discussing with its unique master. I also think that different slaves on same machine sharing same home directory is not supported, not working. And of course if they are completely independent, as Michael Donohue said above, there is a workload sharing issue to resolve.
v1.366 added support for Windows slaves running as a Win32 service to serve multiple masters
see http://hudson-ci.org/changelog.html
Hudson jobs can also be parameterized, with a default value used for scheduled jobs and a web page offered for parameter input on manually triggered jobs. That can work in some situations to reduce need for multiple jobs.
Or try the nested view plugin if number of tabs an issue and can't reduce number of jobs