I have a build job on jenkins that is building my project and after it is done, it opens an ssh shell script on a remote server and transfers files and then stop and starts a daemon.
When I stop and start the daemon from the command line on a RHEL server, it executes just fine. When the job executes in jenkins, there are no errors.
The daemon stops fine and it starts fine. But shortly after starting, the daemon dies suddenly.
sudo service daemonName stop
# transfer files.
sudo service daemonName start
I'm sure that the problem isn't pathing
Does anyone know what could be special about the way Jenkins is executing the ssh shell script that would cause the daemon start to not fully complete?
The problem:
When executing a build through jenkins, the command to start the daemon process was clearly successfully executing, yet after the build job was done, the daemon would suddenly quit.
The solution:
I thought for this whole time that it was jenkins killing the daemon. So I tried many different incarnations and permutations of disabling the ProcessTree module that goes through and cleans up zombie child processes. I tried fooling it by resetting the BUILD_ID environment variable. Nothing worked.
Thanks to this thread I found out that that solution only works for child processes executed on the BUILD machine. I.E. not applicable to my problem.
More searching led me here: Run a persistent process via ssh
The solution? Nohup.
So now the build successfully restarts the daemon by executing the following:
sudo nohup service daemonname start
Jenkins watches for processes spawned by the job and kill them to avoid zombie processes.
See https://wiki.jenkins-ci.org/display/JENKINS/ProcessTreeKiller
The workaround is to override the BUILD_ID environment variable:
BUILD_ID=dontKillMe
Related
Horizon runs fine but only recently, after a deploy, supervisor and queue workers do not start back up again with Horizon GUI showing "Inactive"
To get them running again I can:
restart the daemon worker from within forge
restart the supervisor /etc/init.d/supervisor restart
My deploy script has php artisan horizon:terminate within it. I have also tried reset/purge and a combination thereof.
When I run terminate in the command with an inactive horizon, it seems to do nothing. When I run the same command with horizon active, it shuts it down but the daemon does not reboot supervisor.
The daemon runs without any errors throughout all of this.
Should terminate take down and bring up the service or is it the daemon itself?
Running horizon:terminate will kill the daemon, when the daemon is killed supervisor will realize this and boot up a new daemon. You can clearly see this if you monitor your server with htop while running terminate command.
If a long running job is running, it will run the current job until it finishes. Terminate in general is to reboot the process, to be certain the new code is loaded into horizon, this should be done after the last step in envoyer or similar deployment tool.
This seems like there is something wrong in your setup. Does the horizon process run before you call terminate, again check htop?. Or what happens when the command is called manually?
I am trying to find solution for the jenkins nodes to automatically connect to master in non headleass mode.
Here is what I have done so far
I have written batch script which calls the powershell script. Powershell script has series of commands to start the jenkins slave agent using jnlp.
This batch script is configured in Task Scheduler to run at the startup. The jenkins slave node is connected to master as soon as node starts up.
But execution of GUI test is not visible in the node when the test starts.
When I manually run this batch script from command line, it works fine i.e non headless mode and I can see the execution of GUI test.
I've read few articles about this kind of setup and none worked.
I also wrote a small piece of powershell script to allow a Windows Service to Interact with Desktop. This script runs after the batch script to start jenkins slave agent.
Task scheduler job is configured to Run as Administrator and I login as administrator to see the execution.
Since this batch script is running via Task scheduler I am guessing its running in a different session.
How do I solve it.
Try to put your script into shell:startup.
On a Windows 10 machine with the Jenkins slave under the user you need, press Windows+R (Run...) and type shell:startup and put your script into opened folder. It will run once the user logged in.
This is how you'll see console and any windows it spawns.
How can I check if the Windows setup process is complete from a script? I have a Powershell script that runs on new builds which sometimes gets kicked off before the Windows installation is complete, but needs to wait until setup is complete to execute.
List all processes and ensure that WinDeploy.exe is not among them. If WinDeploy.exe is running, then Windows' setup is not yet complete:
Get-Process -Name windeploy
Note that when WinDeploy.exe stops executing, you will need to ensure that the system is not rebooting before continuing, and wait until it comes back up before continuing execution. For our environment, it was easier to use the VMWare API from our deployment system to wait until WinDeploy.exe was finished executing and also check that the guest was in the running state before attempting to execute the provisioning script.
If this is something that you have to run in a local script for whatever reason, consider using Powershell workflows which can be used across reboots.
I've made a (docker) container for ddclient.
The problem is that I'm having problems in running that service in the foreground so that the docker container keeps running.
I've managed to keep the docker running by adding a bashat the end of the script but this is hackish, since the actual process it should be whatching is the ddclient.
Another way I found was to tail -f the log file, but if the service stops, the container will keep running instead of stoping.
Q: So is there any (easy) way to keep a service running in the foreground?
The problem with the process (any process) running in a container is signal management: you need to make sure the SIGKILL and other signals are properly communicated to the right process(es) in order to successfully stop/remove a container (and not leave zombie processes: see "PID 1 zombie reaping issue")
One option is at least to make your service at least write in a log file
ENTRYPOINT ["/bin/sh" "-c" ]
CMD yourProcess > log
That should keep it in foreground, as suggested in "How do I bring a daemon process to foreground?".
For a service, try and use as a base image phusion/baseimage-docker which manages other services properly.
I am trying to get Jenkins to start a virtual machine on a Jenkins slave. The VM itself will then act as a Jenkins slave.
In order to do so I need to boot the VM and keep it running, even after the Jenkins job terminates. I have tried to create a freestyle project which runs a batch script on the slave and checks if the VM is running:
"C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe" -T ws start "D:\VM\MyVM.vmx"
"C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe" list
The second command shows me that the VM is actually up and running, but apparently it directly shuts down again since I can't see the node that corresponds to the VM as online.
The Jenkins Slave agent is installed as a Windows service on the VM's host and logs in as a domain user.
If I switch the first command to
"C:\Program Files (x86)\VMware\VMware Workstation\vmware.exe" -x "D:\VM\MyVM.vmx"
the VM powers on, the node gets connected to Jenkins. This is because somehow the batch script gets stuck after this command and does not terminate, so the VM remains powered on. However, if I log on the host with the same user the Jenkins service uses, I cannot see the VM running.
Ironically, I can in fact power OFF any virtual machine that I have started locally on the host from Jenkins by creating a project with the batch command
"C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe" -T ws stop "D:\VM\MyVM.vmx" soft
So, to summarize:
I want to create a Jenkins job that powers on a VM so I can use it as a slave agent. The VM has to remain powered on even after the job is done, I will shut it down with a different job as needed.
But only the shutdown job is working as intended.
try to start your VM with START command:
START "C:\Program Files (x86)\VMware\VMware Workstation\vmrun.exe" -T ws start "D:\VM\MyVM.vmx"
After playing around with VMs and Jenkins today I learned that vmrun works perfectly if the Jenkins slave does not run as a Windows service but is launched via the Java Webstart application.
Besides, one can prevent processes from getting killed by altering the BUILD_ID env. variable since Jenkins is using this variable to track the processes the build launched. So by changing the value of BUILD_ID before spawning processes they won't get killed after the Job finishes.