I currently have a problem with a shell step in a workflow script hanging. The step appears to complete but the workflow doesn't move on, the Jenkins java process also begins to consume large amounts of CPU.
Jenkins is running on an OS X box and the sh step is a call to xbuild to build a Xamarin app.
def shell = "xbuild /p:Configuration=${buildConfig} /t:Build ${_solution.getPath()}"
sh("${shell} >> ${_logFile.getPath()}")
The contents of the log file suggest that xbuild completed succesfully but the workflow never moves on from the sh step.
Could anyone suggest a strategy to find out what is causing it to hang?
This turned out to be caused by a tight loop I had that was executing after the sh step completed.
My recommendation to anyone else experiencing a problem like this would be to make good use of logging to the console output so you can see exactly where the build is stuck.
Related
I am trying to become a Jenkins user through this command - sudo su -s /bin/bash jenkins.
When I do run it, this happens in my shell: bash-4.2$ Killed.
I have been trying to debug it by identifying the memory allocation used on the server (we have plenty), reading debug lines of processes being killed, and the jenkins log but there's nothing that leads me towards a solution.
How do I start to debug this problem and what could be going on? There aren't any obvious errors being thrown my way.
I've discovered that when I restart the server, I can log in as the jenkins user and the jenkins builds work. However, shortly thereafter, I cannot log in and the builds no longer work.
Visual Studio has a relatively new way of doing Linux builds, using the local WSL system. This platform toolset is called WSL_1_0
What in a remote build was called "Remote Pre-Build Event" is now called "Wsl Pre-Build Event" (in the UI..still Remote whatever in msbuild, apparently) and it works basically the same, but with execution occurring in WSL on the local machine rather than on a linux machine.
Aaaand...any script output seems to be swallowed (vs. running on a remote machine where it was displayed in the build window):
1>Target RemotePreBuildEvent:
1> Invoking 'echo "me : stuff error 567 HELLO"', working directory: '/mnt/s/buildroot/CoreTech/ThirdPartyTools/Encryption/Rijndael/Project/Linux'
1>Target ClCompile:
1> Starting remote build
If I replace this with a script, it is executed..but the output is swallowed.
I've tried redirecting to stderr both within the script and within the command and it seems to change nothing. I've also adjusted Tools->Options->Project and Solutions->Build and Run output verbosity as high as I'm willing to (Normal) with no changes.
Anyone have any ideas on solving this?
I've also adjusted Tools->Options->Project and Solutions->Build and
Run output verbosity as high as I'm willing to (Normal) with no
changes.
Normally it's recommended to change the Verbosity to Detailed and higher to see more details. And this is the only option I know in latest VS2019 16.3.7 can control the behavior of output log.
So you can change it to Detailed to check if there's more info to be displayed in output window.(For some reason,my configuration fails for the wsl.exe, so I can't test it myself whether this option can work)
And if the echo info is still not shown, I guess this scenario is not be supported for now since it's quite a new feature. Just post your feedback in DC as the Give Us Feedback in this document suggests. The team there would look and consider about supporting that. May it makes some help :)
Getting below error while trying to run shell script
+ /home/pqsharma/symlinkBuild.sh 19.07
sh: line 1: 21887 Terminated sleep 3
With Jenkinsfile:
node ('linux')
{
stage('creating symlink')
{stdout = sh(script:'/home/pqsharma/symlinkBuild.sh 19.07 ', returnStdout: true)
}
}
This is followed by JENKINS 55308: "intermittent "terminated" messages using sh in Pipelines"
Jenkins master runs from a Docker image based on jenkins/jenkins:2.138.2-alpine with specific plugins baked into the image by /usr/local/bin/install-plugins.sh
The message originates in durable-task-plugin, which must be a dependency of one of the plugins.txt plugins.
Check if this is the case for you.
Caused by JENKINS 55867: "sh step termination is never detected if the wrapper process is killed"
When you execute a shell step, Jenkins runs a wrapper shell process that's responsible for saving the exit code of your script. If this process is killed, then Jenkins never discovers that your script has terminated, and the step hangs forever.
This seems to have been introduced after v1.22 of durable-task-plugin
Diagnostic:
The sleep 3 is part of the execution of a shell step.
A background process touches a specific file on the agent every 3 seconds, and the Jenkins master checks the timestamp on that file as a proxy to know whether the script is still running or not.
It seems based on the reports here that something is causing that process to be killed on some systems, but I don't have any ideas about what it could be offhand.
Possible cause:
The bug is not just in the durable-task-plugin, although the symptoms come from there. It is introduced when you upgrade workflow-job. I have managed to pinpoint it down to a specific version.
Upgrading workflow-job to 2.27 or later triggers the bug. (2.26 does not exist.)
So try and downgrade your workflow-job plugin to 2.25
I just installed Jenkins 2.46.2 on a Windows 2012 Server \o/. It runs as a system service.
I created a job that execute a windows batch (.bat) script to build a code project. This batch results in executing 2 mingw32-make.exe commands to clean and then build a full binary from source code.
Executing the batch manually on the machine, located on the same filesystem (same workspace as used by the Jenkins' job, local disk - not network disk), the clean-build takes ~50 seconds.
But when executed by Jenkins, the job takes more than 20x more time longer! (~19 minutes). It terminates succesfully with the same behavior as executed manually in cmd.exe.
I changed the launch arguments for the jvm in the jenkins.xml file with "-Xmx1024m -XX:MaxPermSize=512m" options as I have read in the documentation to improve performance. But it does not fix anything :-(
Also when I monitors the CPU/disk/RAM usages they all stay very very low while building, so I deduce that brute performances of the machine are not in cause.
Whether I invoke the batch with call statement in the Jenkins job build step or not does not change anything : the job always last 19 minutes.
Can anybody help me to investigate why so slowness ?
Thanks in advance :)
I had a similar problem. I noticed that .bat files with echo Hello World ran fast and with no problem.
But once I tried to launch any grep.exe from a batch script, it took 24 seconds (in my case) to run even with no input files. If launched manually it finishes in no time.
I used grep.exe version 2.5.4 from MSys 1.0 distribution.
The solution in my case was rather unexpected - I updated grep to version 2.24, and now, being launched from Jenkins, it takes less than one second to process over 1 MB log file.
For a couple of day investigation, I finally find the cause.
In my case, it is the reason of Jenkins agent.
When I install Jenkins agent as a windows service in the slave agent, the consuming time is so huge,but when I try to start Jenkins agent via windows command line, the consuming time is as normal as executing the batch file manually.
My env:
master: CentOS7
slave agent: win 7
And I also test this case in a slave agent of win 10 for comparison.
The time executing via Jenkins is approximately the same as executing the batch file manually on the agent machine.
So I guess this is the compatibility issue between win 7 and Jenkins.
But for that the Jenkins official said that Jenkins not support win 7 anymore (Microsoft does not support Windows 7), we temporarily put it aside.
Anyway we find a way to conquer this. Hope this will help you for similar scenario.
I am new to Hudson with PSEXEC, i am using hudson in my computer, i want to run batch file on remote computer from hudson build.
I used PSEXEC to run batch file on remote computer,when i executed from command promt it working successfully.But same i did from Hudson build its hanging..it's not doing anything.so please give any suggestions is there any other way we can handle this.
I want to do this quikly...urgent
Anyones help is appreciable
thanks in advance.
Could it be that psexec is waiting for a user entry? In this case hudson will hang.
Does Hudson runs with your user credentials? if not lunch a command Prompt with the credentials Hudson is running with and see what happens if you call psexec. If hudson runs on the local system account, you can't start a command prompt with it. Than try to run Hudson with your credentials and see what happens.
BTW, did you check Hudsons log files (just in case that there might be something)?
I had the same problem where PSEXEC simply hung.
The solution i found (after trawling for AGES) was to add the parameter -accepteula to the PSEXEC called. Hudson stopped hanging after that.
Very frustrating.
maybe it is too late, but I thought I will post it for others who have the same problem.
Psexec has problem running from any CI server, doesn't matter what CI server you have, is it Hudson, Teamcity, ...).
Psexec hangs when run from the build server.
It doesn't hang for all commands, but with some commands , psexec will hang.
The problem resides with how psexec will read the output of the remote running command.
The ultimate solution is to use other tools than psexec.
You can check this blog of this solution.
But if you need a simpler solution, and you don't care about the output of the command, I came up with a simple solution that suppress the output of the psexec.
You can write a small c# program that will run the psexec command, suppressing its output.
I blogged about that here
I thought I would respond to this since I was having the same hanging problem with TeamCity and psExec. Since I believe it is actually an issue with Java executing psexec, it should apply in the case of Hudson as well.
While I ran into the accepteula issue was early on, you only need to agree once and it sets a registry setting. As user327759 indicates the -accepteula switch in the script would cover your basis the first time and for ever after.
If this doesn't fix your problem, which it didn't in my case, you need to add the "-i" command switch. ex: psexec -i \ServerName "C:\folder\file.bat"
For context regarding this solution, I have included my backstory:
I was able to execute this PsExec command just fine via a MsBuild task, via a .Net wrapper and via straight command prompt usage, but TeamCity would just hang on the execution. It seems that Java is expecting some response on standard in, when it invokes the psexec exe. On the remote server I would see the psexecsvc process fire up and register in the services console appropriately, but then it would stop. After reading many posts such as this one, and struggling for far longer than I would care to admit, I eventually tried -i option, despite not needing this in any other execution scenario outside of TeamCity. The psexec documentation indicates that -i will "Run the program so that it interacts with the desktop of the specified session on the remote system. If no session is specified the process runs in the console session." It is still unclear to me why this makes all the difference for TeamCity.