I'm trying to setup an automated "build" server for my rails projects using Hudson CI. SO far it's able to run specs and do metrics on the code but I have 2 different projects dependent on 2 different versions of ruby. So i'm trying to use RVM to run multiple copies of ruby then switch back and forth in a pre-build step.
I found a couple posts like this one that try and explain how to make this work, but I'm not running a startup script for hudson, it starts on boot which is how it worked out of the box when i installed it via the debian instructions.
The problem seems to be that even though hudson runs under the "hudson" account and that account has rvm installed (and working) when it tries to run a shell based prebuild step to call rvm switch 1.8.7 it fails with the error "rvm: command not found"
Not sure what I'm doing wrong. Hudson is using SH as its shell but i also tried using bash. no luck.
Has anyone gotten this working before in this setup?
edit the "/etc/init.d/hudson" (!) and change the line:
SU=/bin/su
... change to:
SU="/bin/su -"
... and add rvm setup in the /home/hudson/.profile
I had the same symptoms as you.
After a couple of hours of headbanging, check your $HOME environment variable for Hudson (viewable at http://yourserver/hudson/systemInfo).
Under Ubuntu, the Tomcat 6 start script doesn't set $HOME. Somehow, Hudson inherited my $HOME instead!
I added HOME=$CATALINA_HOME to the /etc/init.d/tomcat6 script just under the rest of the ENV declarations, and now it all works. Very annoying issue, to be sure.
Related
We started seeing some strange errors in our logs that normally appear when ruby isn't compiled properly with OpenSSL. But it's inconcistent...
We're getting errors like:
RuntimeError: Unsupported digest algorithm (SHA256). (also with other digests, like sha1). example error trace
Faraday::SSLError (SSL_CTX_new: (null)) example error trace
We managed to reproduce it when starting unicorn using service unicorn start or systemctl start unicorn. But only with some requests... Not all of them. Some requests that use OpenSSL under the hood do work. Others don't.
However, when we start unicorn with /etc/init.d/unicorn start, everything works without a hitch. (to clarify, systemd starts the same /etc/init.d script)
We tried debugging ENV vars, user permissions, file/dir ownership, recompile ruby, bootstrap a new server from scratch... Nothing seems to help.
In case this helps:
unicorn init.d script
unicorn.rb
What are we missing? What can we try that we haven't thought of?
UPDATE 1
output of some debug commands, e.g. OpenSSL, ruby etc
PATH is being set inside the init.d script
unicorn is being executed via su into www-data user
The same problem happens when we use this unicorn.service file in /etc/systemd/system
We're running Ubuntu 16.04 on Gcloud
Ruby was not installed via apt (explicitly removed, in case platform came pre-installed) and compiled from scratch. We're currently running 2.3.4 and tried also 2.3.6. Compiled either manually or using ruby-build. No rbenv, nor RVM.
We install libssl-dev via apt (we're running apt-get install -y autoconf bison build-essential libssl-dev libyaml-dev libreadline6-dev zlib1g-dev libncurses5-dev libffi-dev libgdbm3 libgdbm-dev before building ruby)
UPDATE 2
We're using a scripted/repeatable build process for the VM (using fabric), and this problem is consistent on multiple VMs we bootstrapped on GCloud. We then tried a VM on DigitalOcean with the same bootstrap scripts, and the problem doesn't seem to appear there.
In both cases we picked Ubuntu 16.04 64bit base image, but obviously there are some differences with kernel versions, base installed packages etc...
UPDATE 3
The problem simply vanished. See my answer below.
#gingerlime I had a similar situation with our Jenkins on GCP, we're using ChefDK 3.1.0 (ruby embeed 2.5.1p57) -- tried other also, over a Jenkins that was running over systemd (Ubuntu 16.04) and upstart (Ubuntu 14.04) -- we tried on both versions, right now running over 16.04 in 4.15.0-1023-gcp kernel version, running a few jobs with kitchen-docker and this problem always emerge in a few situations.
I digged into and found that this only happens when the Etc.getlogin class gets called (for me here), this doesn't return any error, it return the correct info, the correct type of the class (String), but once it gets a call, the Unsupported digest algorithm gets raised.
If I start the process manually by root or jenkins user, this problem doesn't happen. I tried to implement the Etc.getlogin in several different ways, like using ENV['USER'], a fixed String, or other classes from Etc, like getpwuid, simulating the return class and values from Etc.getlogin, and the error doesn't get raised.
I'm not sure if this is some bug related to the ruby version and the custom kernel that GCP instances uses, but it happens in a similar situation like yours, and for me, the Etc.getlogin was the problem. Right now, I fixed by using a custom configuration that doesn't gets the call from this function, and it's working normally.
One option is that this isn't an issue of sysVinit vs systemd at all, but you just haven't triggered the issue with your sysVinit script yet.
When you run your svsVinit script through the systemctl command it's going through a compatibility layer, and there may be a problem there. Your problem would be simplified both yourself and for us if you reproduced the issue directly with a systemd service file and shared that file.
You mentioned debugging ENV, but didn't mention exactly what you checked in the ENV. This is definitely one place where systemd could make a difference. As seen in man systemd.exec, systemd sets $PATH in the environment to a fixed value:
/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
If this is not exactly the same as when run directly as an sysVinit script, that could be an issue.
I would also check for all your copies of SSL on the system. Do you have more than one? Where? Do you have more than copy of the ruby openssl module loaded?
locate -r lib/.*libssl.*so
Also see the answer to the FAQ: Why do things behave differently under systemd?
(also posted on this github issue)
It looks like the problem just vanished. We were testing and reproducing it consistently, across several Compute Engine instances on Google Cloud. Under certain conditions (unicorn / puma started by systemd, etc), it was completely reproducible both with our own rails app, and with a plain vanilla rails app we've set for testing purposes. It was reproducible across several ruby versions as well (we tested 2.3.4, 2.3.6 and 2.5.0).
Suddenly, all instances that were consistently failing started working without exhibiting these problems. Like it never existed. We didn't even reboot some of those instances, and we saw no evidence of any unattended upgrades taking place... We also had one snapshot of a system that had this problem, and that we can reliably reproduce on. Creating an instance from this snapshot stopped exhibiting it as well from that specific point in time a few hours ago.
We're totally confused as to what might have cause it, and what might have made it disappear... However, without being able to reproduce it now, I guess there's no point leaving this issue open, so will close it. Chalk it up to Deus ex machina I suppose. (perhaps the google support gods, but they haven't reported anything back to us yet)
I'm currently working on a build pipeline that uses Jenkins and GitLab to trigger builds for the project. Basically, the build is triggered when someone pushes to the repository. Also, some Ruby scripts are executed as part of the build process. These scripts run some checks on the projects and perform some fixes, like synchronizing an Xcode project with added and deleted files from the source directory - in this case they are not the same.
I'm using several tools to configure the pipeline. The builds run on a machine that is physically located on the build slave. Jenkins is deployed to an AWS machine. For this reason, I used pritunl to connect the two on a virtual network. I can use local IPs to communicate between the machines and SSH is working fine both ways.
When I push to the remote the build starts correctly on the slave, but it fails to complete. However, if I manually access using SSH through the terminal, the build performs fine. This is the output I get from Jenkins:
/System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- xcodeproj (LoadError)
from /System/Library/Frameworks/Ruby.framework/Versions/2.0/usr/lib/ruby/2.0.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /Users/jenkins/workspace/Core/platform/ios/scripts/pbxsync.rb:58:in `<main>'
As you can see, it fails to require Xcodeproj, causing the build to fail. Still, this only happens if the build is triggered by Jenkins, not manually.
This makes me think that Jenkins is using some different installation of Ruby, or at least a different environment. Basically what I need is to install gems for the same Ruby environment that Jenkins is using, but I don't know which one that is. Any ideas?
Jenkins has a console that runs Groovy scripts on the remote slave. I've been playing with it a bit, but not many conclusions so far. Maybe that helps.
This may be important; this is the shebang I'm using for the Ruby scripts: #!/usr/bin/env ruby
On the terminal, I'm using the same user as Jenkins is to access the slave machine. It's called "jenkins".
One thing I forgot to mention is that the output is telling me the right version: /Users/jenkins/.rvm/rubies/ruby-2.4.0. At least that's the path it's indicating it's trying to load the gem from. So I tried the following:
: /Users/jenkins/.rvm/rubies/ruby-2.4.0/bin/ruby
require 'xcodeproj'
Then I press ctrl+D and get no output - that installation of ruby is finding the gem properly.
If you are using Jenkins Slave plugin to communicate between Jenkins Master and Jenkins Slave, every command that u specify will be run in non-interactive shell. That means that Jenkins will only have access to system ruby in your case.
So if you want to install something that needs to be installed you have to do it in system ruby. You are using rvm so: rvm use system and you can install gem to system ruby.
If you want to use different Ruby version than system ruby you need to add RVM to $PATH for non-interactive shell. Here is basic setup that should help: https://rvm.io/rvm/basics
I finally managed it. As #Cosaquee indicated in another response, it's important to distinguish between interactive and non-interactive shells. The main reason for this is because, depending on how you call SSH, it makes a difference. As the man page indicates:
If command is specified, it is executed on the remote host instead of
a login shell.
This is meaningful, because the Launch Command for the node I have set for Jenkins is this one:
ssh jenkins#x.x.x.x java -jar ~/bin/slave.jar
In the meanwhile, I was logging in with the standard ssh jenkins#x.x.x.x from the terminal, which starts a login shell. It makes sense that I was getting different results because the two shells load different initial scripts. Basically, if you use ssh jenkins#x.x.x.x to login into the machine ~/.bash_profile is loaded, while if you specify a command, such as ssh jenkins#x.x.x.x whatever, then ~/.bashrc is loaded instead. As such, I added this line to ~/.bashrc:
[[ -s "$HOME/.rvm/scripts/rvm" ]] && . "$HOME/.rvm/scripts/rvm"
Without it I got:
RVM is not a function, selecting rubies with 'rvm use ...' will not
work.
The advantage was that I could now use RVM from the same environment Jenkins was using. The rest is easy:
ssh jenkins#x.x.x.x rvm --default use 2.3
And:
ssh jenkins#x.x.x.x
rvm --default use 2.3
And both are now using the same version of ruby.
I'm pretty sure I've tracked the issue down to Node.js not seeing Sass, but I have no clue why...
If I push from my laptop using:
git push lamp somebranch:master, the server remotely checks it out fine, runs npm install without error, and starts processing the gruntfile, but then aborts with "remote: Warning: spawn ENOENT Use --force to continue."
However, (after I push from my laptop like above) I can ssh in, cd into my hooks directory and run ./post-receive and it finishes "Done, without errors." I also tried running grunt in the website's root and it also completed without error.
Any ideas as to what might be going on? I'm completely stumped. Should I set paths to the sass gem in the hook? I scrapped down my gruntfile to use the same target locally as well as on the server to rule out the gruntfile. It compiles fine locally, compiles fine on the server, but fails only when using git push lamp somebranch:master.
Some may wonder why I just didn't compile locally and dump the css into the web root from the devel box... perhaps I should. This time though, I really wanted Push-to-deploy all the way through, compiles and all. For anyone attempting the same thing and running into the same problem, this should help.
First off, it probably wouldn't hurt to scrub the system of any versions of ruby and sass that were installed via the distro's package manager. Then I scrubbed any remnants of previous tinkering with rvm implode and removed traces from .bashrc, etc. Next I ran \curl -sSL https://get.rvm.io | bash -s stable --ruby --auto-dotfiles and pressed ctrl-c to fix any errors first. Once the install script was happy, I let it download and install as normal. I did not have to use rvm install n.n.n,rvm use n.n.n, or rvm use n.n.n --default as 2.2.1 was pulled in like I wanted anyway and seemed fine. After rvm had setup ruby, I then ran gem install sass
Now, the end-all-be-all... using PermitUserEnvironment, like had been mentioned here: How to use sshd-config permituserenvironment option was the way to go. I saw that there were security concerns with that method, but it was the only thing that worked and I won't be trying to run limited shells. It is normal behavior for SSH to not allow the env vars when not using a login shell. I assumed, however, that the git hooks had full access to the user's normal vars (with ruby paths, etc.) and that assumption was incorrect. Add PermitUserEnvironment yes to the server's /etc/ssh/sshd_config or the like and restart the ssh daemon. As the user on the server, I ran env and copied that into .ssh/environment and cleaned up what wasn't needed. After that, I did my git push from the devel box and it found and ran the sass compiler just fine.
I'm improving the continuos integration of a project. And we decided to take an extra step and start using cocoapods. All the rvm installation is legacy and indeed I have a lot of troubles installing ruby 2.2.0. The thing is that, when I test my build script using terminal it works fine, but when I try to run them without opening a terminal window (called from applescript, jenkins or another ruby script). The command is not found.
Already tried adding the path to .rvm/scripts to the PATH variable in both .bashrc and .bash_profile
Have you try to reconnect the server after you installed the cocoapods? sometimes it doesn't see the new vars till it disconnected and reconnected.
Also make suer that the vars that you see through the terminal are available for jenkins user. you can check that through the slave "Script Console"
If it still don't work, try to set the path in the "execute shell", just before you run the pod install.
This is how it works for me:
echo "Running pod install"
cd ${WORKSPACE}
export LANG=en_US.UTF-8
pod install
I'm trying to set up CI on my local machine running on Mac. To do so I use Xubuntu virtual machine, Jenkins, and some simple selenium tests. tests on github
I get fresh install of Xubuntu, where I install Jenkins using official manual.
In Jenkins I installed some plugins(git, ruby, rake, rbenv).
In job config I use rbenv wrapper(2.1.0) with ignorance of os versions, also I use this gemlist:
bundler,rake,rspec,selenium-webdriver,capybara
and running that job with
rspec spec
And when I run this job I recieve something like that for every test:
Selenium::WebDriver::Error::WebDriverError:
unable to obtain stable firefox connection in 60 seconds (127.0.0.1:7055)
full output is here
it looks like jenkins user have no access to display to run/see firefox.
Anyone know how to make it work?
We ran into this at work recently and actually opted for Capybara and set the driver to poltergeist. This seemed better than trying to figure out how to run FF on our VM's.
That said, we were able to get a small test suite running by following the instructions here
Answer was not so simple as I think.
The problem is that jenkins service has no access to displays (when it installed via native package). Thats why when I try to start Firefox it's throw me a error. try:
$ sudo su - jenkins && firefox
So it's need access to display to start browser successfully.
This is how I done it:
first of all I used answer form here where I changed to my local user.
Then I installed xvfb plugin to Jenkins, and in my build job preset display to '0' - which is my actual user display. with that option all my tests would run 'headless' but on actual display.
This could be not the best way to solve my problem, but it definitely works for me.