I use the following post-commit hook for svn:
"path\to\ruby.exe" "path\to\redmine\script\runner" "Repository.fetch_changesets; Rails.logger.flush" -e production
It works correctly, but it takes about 1-2 minutes.
I also thought that a lot of time is required for first commit, but successive commit takes the same amount of time.
Is it possible to improve such behavior?
I know about slow behavior of Ruby on Windows, about 3 times, but in my case it is much more longer.
Configuration is following: Windows Vista, redmine 1.1.1, Ruby 1.8.7 with RubyGems 1.8.7, all packages installed and testing is performed on the same PC.
The problem is that script/runner starts up a new Rails process from scratch each time it's run, which will make your commit pause. So 3 commits == 3 startups and 3 shutdowns.
There are a few things you can do to improve this:
Run the script/runner process in the background so it doesn't slow down your commit. On Linux you can do this by adding an & at the end of the command but I don't remember how to do it on Windows
Instead of fetching changesets on each commit you can run it regularly through cron or a scheduled task. The rake task redmine:fetch_changesets is built for this purpose, it will iterate through each project and run fetch_changesets for you.
The command you are running goes through every project and fetches the changes. If you know the project identifier you can change the query so it only gets the changes for the project you are working on:
script\runner "Project.find('your-project').try(:repository).try(:fetch_changesets); Rails.logger.flush"
Replace 'your-project' with the project identifier (found in most of the urls in Redmine). The try parts are used to make sure you don't get an empty record.
Use the web service to fetch changesets instead of script/runner. The web service will use an existing Ruby process so it should already be loaded and the only slow down will be while Redmine downloads and processes the changes. This can also be used with the first option (i.e. background a web service request). Docs: http://www.redmine.org/projects/redmine/wiki/HowTo_setup_automatic_refresh_of_repositories_in_Redmine_on_commit
Personally, I just run a cronjob every hour (#2). Having the Repository view current isn't that important to me. Hope this gives you some ideas.
Related
I have a situation where I need to deploy ruby code to lab devices. These machines are slow and very locked down (hard to get files to). They get re-imaged fairly often, but I am not allowed to bake the install into the image.
I've solved the problem by using the same open ports for Git code distribution to distribute my install files.
I have a long install process boiled down to 3 batch files, but if I could reduce it to one, it would make life a lot easier (not to babysit a lot of installs via VNC). I'm only mentioning all this for reference.
The problem:
I can't install git and then do a pull from command line without opening a new cmd prompt - I think it pertains to environment variables, but am not 100% sure.
I get "git is not recognized" blah blah blah if I don't break out at this point and start the next batch. Same deal when I install ruby and don't break out before starting the DevKit install.
Powershell is not an option (I'm only allowed to install Git, Ruby, and the support (DevKit, C++ redistributables, .Net Client 4) and some of the machines do not have it.
I did a version where I scripted reboots and moving the batches into startup progressively - it works, but some of the other machines are tied together in a way that makes rebooting an issue (please don't make me explain - it's complicated lol).
Is there a way to merge my 3 batch files and execute all steps without reboots?
Edit: I have tried starting the next batch from the first (Start... Call...), and even creating a scheduled task to execute the next step. Can't swear I haven't made any mistakes, but they all seem to inherit the initial conditions and don't recognize the "git" command unless a new cmd prompt is opened.
I have three build machines with Jenkins v1.473 installed.
Let's call them machine A, B and C. All are running Windows 7 x64.
Machine B and C jenkins installation was carried over from machine A. I simply copied the folder over and everything was imported and works fine.
We initially had a Windows command line job that would call svn revert and then, svn update on a bunch of folders. After that, we would build our project. The average running time was about 10 mins. SVN version of the command line tools is 1.7 on all 3 machines (with tortoisesvn)
Since we were using command line svn commands, we didn't have access to $CHANGES when sending the job completion email, so we switched to the Jenkins svn plugin.
The switch was made on machine B and C first and after confirming it was working fine, we applied the same changes to machine A. The plugin version is 1.50.
Check-out Strategy is set to "Use 'svn update' as much as possible, with 'svn revert' before update".
Repository browser is set to Auto.
Under Jenkins->Configuration, Subversion Workspace Version is set to 1.7.
The other fields are left as-is (Exclusion revprop name is empty, Validate repository URLs up to the first variable name is unchecked, Update default Subversion credentials cache after successful authentication is checked)
Now, onto my issue.
The running time on machine B and C stayed about the same: about 10 mins.
However, on machine A, running time has more than doubled: it now averages 25 mins.
Looking at the job log, the revert part seems to be the culprit.
Is there a reason switching from svn command line to the plugin would make it run slower ? On top of that, on one particular machine only ?
After digging and testing several jobs across all machines, here's what I found out:
command line tools are fairly faster than their plugin counterpart, most likely due to the speed of the programs themselves (compiled C binary vs SVNKIT which is written in Java and running on top of a JVM)
machine A is also fairly slower than B and C, despite having similar specs, most likely because of other applications and services running at the same time (B and C are near-fresh installs, while A has been running for a longer time, with more applications installed etc)
In the end, I just split the clean/revert part from the update:
Clean/Revert job is carried in command line (I/O heavy due to folder traversal I believe) so it is as fast as it can be
Update (and update only, no revert done within the plugin) is then done with the plugin (fairly fast, less than a minute usually for several dozen files) which allows me to still have access to the CHANGES variable.
With this, I somehow managed to keep the building time on machine A nearly unchanged (around 10 mins).
Machines B and C however, have seen their building times greatly improved (from 10 mins to 5 mins in general)
I have some Issues regarding Jenkins and running a Powershell Script within. Long Story short: the Script takes 8x longe execution time then running it manually (takes just a few minutes) on the Server(Slave).
Im wondering why?
In the script are functions which which invoke commands like & msbuild.exe or & svn commit. I found out that the script hangs up in those Lines where before metioned commands are executed. The result is, that Jenkins time out because the Script take that long. I could alter the Timeout threshold in the Jenkins Job Configuration but i dont think this is the solution for the problem
There are no error ouputs or any information why it takes that long and i do not have any further Idea for the reason. Maybe one of you could tell me, how Jenkins invokes internaly those commands.
This is what Jenkins does (Windows batch plugin):
powershell -File %WORKSPACE%\ScriptHead\DeployOrRelease.ps1
I've created my own Powershell CI Service before I found that Jenkins supports it's own such plugin. But in my implementation and in my current jobs configs we follow sample segregation principle rule: more is better better. I found that my CI Service works better when is separated in different steps (also in case of error it's a lot easy for a root cause analyse). The Single responsibility principle is also helpful here. So as in Jenkins we have pre- & post-, build and email steps as separate script. About
msbuild.exe
As far as I remember in my case there were issues related with the operations in FileSystem paths. So when script was divided/separated in different functions we had better performance (additional checks of params).
Use "divide and conquer" technique. You have two choices: modify your script so that will display what is doing and how much it takes for every step. Second option is to make smaller scripts to perform actions like:
get the code source,
compile/build the application,
run the test,
create a package,
send the package,
archive the logs
send notification.
The most problematic is usually the first step: To get the source code from GIT or SVN or Mercurial or whatever you have as version control system. Make sure this step is not embeded into your script.
During the job run, Jenkins capture the output and use AJAX to display the result in your browser. In the script make sure you flush standard output for every step or several steps. Some languages cache standard output so you can see the results only at the end.
Also you can create log files that can be helpful to archive and verify activity status for older runs. From my experience using Jenkins with more then 10 steps requires you to create a specialized application that can run multiple steps like "robot framework".
I'm trying to create a C++ CI environment by using CDash.
I've got CDash running on my computer and I can send some results to it from the CDash clients, by running the ctest manually.
I'm a bit lost on how to setup a client to automatically compile and test the code when the source code changes in the version control system (subversion), or at specific times.
I have the Mastering CMake book, but it doesn't seem to say much on that topic.
Is there any way to do the continuous build without hacking around with scheduled tasks / cron?
Is there any good example that would be useful to check out?
Can I somehow order to run a build on some site from the Dashboard? I kinda remember seeing this somewhere but I'm unable to find it now.
Is CDash any good for CI environments? (use comments to answer this one)
CDas#Home might be the solution. Generally I have my continuous machines run a script on a nightly cron job that polls the repository every couple of minutes for ~24 hours.
I've been using nosetests for the last few months to run my Python unit tests.
It definitely does the job but it is not great for giving a visual view of what tests are working or breaking.
I've used several other GUI based unit test frameworks that provide a visual snap shot of the state of your unit tests as well as providing drill down features to get to detailed error messages.
Nosetests dumps most of its information to the console leaving it the developer to sift through the detail.
Any recommendations?
You can use rednose plugin to color up your console. The visual feedback is much better with it.
I've used Trac + Bitten for continuous integration, it was fairly complex setup and required substantial amount of time to RTFM, set up and then maintain everything but I could get nice visual reports with failed tests and error messages and graphs for failed tests, pylint problems and code coverage over time.
Bitten is a continuous integration plugin for Trac. It has the master-slave architecture. Bitten master is integrated with and runs together with Trac. Bitten slave can be run on any system that communicate with master. It would regularly poll master for build tasks. If there is a pending task (somebody has commited something recently), master will send "build recipe" similar to ant's build.xml to slave, slave would follow the recipe and send back results. Recipe can contain instructions like "check out code from that repository", "execute this shell script", "run nosetests in this directory".
The build reports and statistics then show up in Trac.
I know this question was asked 3 years ago, but I'm currently developing a GUI to make nosetests a little easier to work with on a project I'm involved in.
Our project uses PyQt which made it really simple to start with this GUI as it provides all you need to create interfaces. I've not been working with Python for long but its fairly easy to get to grips with so if you know what you're doing it'll be perfect providing you have the time.
You can convert .UI files created in the PyQt Designer to python scripts with:
pyuic4 -x interface.ui -o interface.py
And you can get a few good tutorials to get a feel for PyQt here. Hope that helps someone :)
I like to open a second terminal, next to my editor, in which I just run a loop which re-runs nosetests (or any test command, e.g. plain old unittests) every time any file changes. Then you can keep focus in your editor window, while seeing test output update every time you hit 'save' in your editor.
I'm not sure what the OP means by 'drill down', but personally all I need from the test output is the failure traceback, which of course is displayed whenever a test fails.
This is especially effective when your code and tests are well-written, so that the vast majority of your tests only take milliseconds to run. I might run these fast unit tests in a loop as described above while I edit or debug, and then run any longer-running tests manually at the end, just before I commit.
You can re run tests manually using Bash 'watch' (but this just runs them every X seconds. Which is fine, but it isn't quite snappy enough to keep me happy.)
Alternatively I wrote a quick python package 'rerun', which polls for filesystem changes and then reruns the command you give it. Polling for changes isn't ideal, but it was easy to write, is completely cross-platform, is fairly snappy if you tell it to poll every 0.25 seconds, doesn't cause me any noticeable lag or system load even with large projects (e.g. Python source tree), and works even in complicated cases (see below.)
https://pypi.python.org/pypi/rerun/
A third alternative is to use a more general-purpose 'wait on filesystem changes' program like 'watchdog', but this seemed heavyweight for my needs, and solutions like this which listen for filesystem events sometimes don't work as I expected (e.g. if Vim saves a file by saving a tmp somewhere else and then moving it into place, the events that happen sometimes aren't the ones you expect.) Hence 'rerun'.