Currently we are running some of our small ruby scripts through CI machines like teamcity. But the problem is that teamcity is only free to a certain point and we are reaching that cap. The thing I like about teamcity is the fact that I can define how to run the scripts in it and then have the logs shown in each of the "build" processes so if something goes wrong or I want to verify something I don't have log onto the server and inspect individual files.
The problem is that I need to be able to run the same process at about 4x the capacity I am now, which means I need about 4 times the build agents which leaving the free licensing. Now obviously I could just spin up more teamcity servers but that then becomes a pain.
So my question is, what is another way that I could just basically setup cron processes on linux machines (i have a lot "freely") but then give myself the exposure and ease of access of logs similarly that I gain from teamcity. Obviously I know that setting up cron processes aren't hard but I really want to avoid having to log onto the machine to check and see if my automated processes are running correctly or struggling.
Thanks in advance!
p.s. I also have access to windows machines if there is an easier way to do it there.
Install an open-source CI server like Jenkins if you want to host it yourself. You can also run it on your own machine, though it's usually better to have it always on in the cloud.
Related
I'm trying to learn ML on GCP. Some of the Qwiklabs and Tutorials start with Cloud Shell to setup things like env variables and install Python packages, while others start by opening an SSH terminal into a VM to do those preliminary steps.
I can't really tell the difference between the two approaches, other than the fact that in the second case a VM needs to be provisioned first. Presumably, when you use Cloud Shell some sort of VM instance is being provisioned for you behind the scenes anyway.
So how are the two approaches different?
Cloud Shell is a product that is designed to give a large number of preconfigured tools that are kept updated, as well as being quick to start, accessable from the UI, and free. Basically, its a quick way to get an interactive shell. You can learn more about this environment from its documentation.
There are also limits to Cloud Shell -- you can only use it for 60 hours a week, if you go idle your session is terminated, and there is only 5GB of storage. It is also only an f1-micro instance, IIRC. So while it is provisioned for you (and free!), it isn't really useful for anything other than an interactive shell.
On the other hand, SSHing into a VM places you directly in a terminal on that VM, much like you would on any specific host -- you only have whatever tools that the image installed onto that VM provides (and many VMs come pretty bare bones, it depends on the image). But, you're now in a terminal on the host that is likely executing the code you want to work with, and it has as much CPU and RAM as you provisioned in that instance.
As far as guides pointing you to one or the other -- thats really up to them, but I suspect they'd point client / tool type work to the cloud shell (since its easy and a reasonably standard environment, which can even be scripted with tutorials), while they'd probably point how to install necessary software for use in production to a "real" VM.
I'm trying to introduce continuous integration in an old project, and we've got quite specific situation - it's possible to put the CI server only on our test server that runs on CentOS. The server has quite a lot of unused RAM and CPU capability.
However, we need to run Ant builds on Windows (this also used to be how the project did packaging before), however it turned out that not the same output (after binary compare) is produced by just using Unix versions of Java and Ant.
I drew up a diagram of how in my mind it could work, but I'm really wondering whether that is even possible (with already given tools).
The black part is implemented, I'm curious whether the red part could be possible. Could the Jenkins slave communicate with master on different OS?
It should be possible. I have a feeling you will need to play with your network settings. But if before you start changing anything see if you can start a headless slave by following these directions: https://wiki.jenkins-ci.org/display/JENKINS/Step+by+step+guide+to+set+up+master+and+slave+machine
Using VirtualBox for CentOS, it will possible to run a Windows VM on your CentOS host.
I'm not sure you need Docker to launch your Jenkins slave.
It maybe better to use a standard JNLP Windows service to connect your Windows slave to Dockerised Jenkins master.
If the master is not able to view the Windows node using this method, you may have to tweak your network configuration on the Windows VM.
But I'm not sure it's necessary.
My team is developing a desktop application (mixed C++/Tcl) that is used in a client-server setup. Currently it is Windows-only, but soon we will need to port it to Linux. CruiseControl.NET builds it every night from the source code in SVN and packages it into NSIS installer, but we have no automated tests to run.
It is nearly impossible to add any unit tests, but integration testing of the application is easy, because it is already heavily script-based.
The main task is to install the app into 3 PCs, configure it (that involves copying some files around), run it, monitor a possible crash, wait till integration testing is done, collect a summary, send emails. It could be done with a bunch of custom PowerShell scripts, but
In future we will want to add more features and more testing, and
what used to be a simple script soon blows up (as usual), so I want
to minimize custom scripting, and if I need to script something, I
prefer bash/cygwin (I am not familiar with Python or Ruby).
I want a web dashboard that will report current progress, and if
something failed - show logs
I need some supervisor that will monitor the app under test and
report if it hangs or crashes
we will need to test it also on Linux
ideally I would like to orchestrate some test steps between the PCs
(e.g. run test X on PC1 and test Y on PC2 in parallel, wait till they
both finish, then run test Z on PC1, while monitoring that nothing
crashes on PC2 etc)
So, I am looking for a COTS tool/set of tools that will help me to do it and don't have a steep learning curve. Ideally, for free, but if it is really good and has fair pricing, my company may purchase a license.
The process should be triggered from CruiseControl.NET when the NSIS installer is ready, and then perform everything described above. Basically, it should allow at least remote installation of software, running custom scripts and have a web dashboard.
Apparently, SCCM tools like Chef could be used, but so far neither of them supports a Windows server, only nodes. I would like to avoid setting up a Linux VM just for that, although I can do it, if I have no other choice. Also, Chef seems to be a bit overkill - good for 10k machines, but I have only 3... maybe 5 in future. And I am particularly curious about chances to orchestrate a distributed test.
Most of the similar questions here on StackOverflow and in internets are about web apps, Java containers, Maven etc, and there are just so many tools and plugins for these tools to evaluate.
Thanks in advance.
Install ccnet on your test machines. Have those ccnet projects listen to a file that gets edited when a new installer is ready. Have the test machines install that new installer and run tests. There you go. ccnet sends emails so there's your basic reporting.
Have the test results get reported into a database via web services using gSOAP(that's what we did). For linux you can run java cruisecontrol if you must. Write a gSOAP enabled test controller program to report the test results from the test machines. A little c++ app will do. Then write a website(we use ASP.NET) to query the database(Postgresql) and show results. Have the test machines auto update themselves via SVN to get the latest changes to the configuration. Use Nant. Nant is far superior to just using ccnet to run tasks. Nant works through ccnet. Use XML, XSL and CSS with ccnet to make test emails have the information you want(new passes, new failures, SVN differences to code bases, etc...)
Our latest development is putting a big TV in the kitchen with a summary of test results so people can know more readily what they broke!
The first thing I'd get working is a test machine listening for the new installer, installing it, running some basic tests and emailing the results back. Put the ccnet and nant configuration in version control and get that auto updating on the test machine so you don't have to log into every test machine and do an update every time you make a change.
This is hugely broad and pretty close to opinion based. Chef can handle steps like deploying the application to the test machines but it isn't a GUI test framework so you would need something else to handle that. Jenkins supports distributing tests to windows hosts so that seems like a good choice on that side of things but it isn't that great at multi-node tests or orchestration between them. I suspect you'll need to write most of this yourself given the requirements.
I would like to run Teamcity (with a build agent) in a Linux VM to handle our none-.net projects. But in the same breath I'd like to have a BuildAgent setup on a Windows server to handle all of the .net projects.
I can't think of any reasons why this wouldn't work but has anyone any experience and any ideas about the problems I might encounter before I spend too much real time on this?
Ta
It's fully supported. TeamCity also knows which agents to route builds to.
This is a very normal scenario and many project I know do this without any problems. Just make sure that for the builds' Agent Requirements, you properly direct the appropriate job to the appropriate agent. One criterion can be that agent.os.name should contain Windows or Linux etc.
I was wondering if anybody out there is doing their software builds using something like Amazon's EC2. I was thinking about trying to move our builds into that environment. Right now our builds are serial but only because we don't have enough computers to run all the components in parallel. Using EC2 we could create 50 or so computers, run them for a few minutes in massive parallel and then send the build results back to our site. Once we're done we could shutdown or destroy the machines. This would save us a bunch of time since the bottle neck is really the builds and not the size of the results.
Is anybody else doing this? Can you offer any advice?
My company runs our build system on EC2; we have a much smaller setup than the one you're talking about, but we have a build controller instance running Hudson which kicks off builds on a separate, clean instance and then distributes the build artifacts to our repository server (which also happens to be on EC2)
Using a cloud solution is ideal for what you're describing, since you can spin up the build servers only when you need them and be confident of building from a fixed baseline each time. The only downside I can think of is the build time; an EC2 instance can take up to 10 minutes to start up, so you either have to add that on to your total build time or keep the build servers running continuously.