Parallel pipeline with 2 inline stages - jenkins-pipeline

Playing with Jenkins pipeline from https://www.jenkins.io/doc/pipeline/examples/#parallel-multiple-nodes
Simple two parallel steps (OK)
I made a first test pipeline this way:
pipeline {
stages {
stage('Build') {
steps {
script {
def labels = ['precise', 'trusty'] // labels for Jenkins node types we will build on
def builders = [:]
for (x in labels) {
def label = x // Need to bind the label variable before the closure - can't do 'for (label in labels)'
// Create a map to pass in to the 'parallel' step so we can fire all the builds at once
builders[label] = {
node('JenkinsNode') {
sh script: 'echo build', label: 'Build on $env.NODE_NAME'
}
}
}
parallel builders
}
}
}
}
}
It resulted in the following expected diagram in Blue Ocean view:
Simple two parallel steps with two sub steps each (KO)
Attempt#1
Then I tried to split each parallel step in two inline stages (to simulate build and tests for example)
pipeline {
stages {
stage('Build') {
steps {
script {
def labels = ['precise', 'trusty'] // labels for Jenkins node types we will build on
def builders = [:]
for (x in labels) {
def label = x // Need to bind the label variable before the closure - can't do 'for (label in labels)'
// Create a map to pass in to the 'parallel' step so we can fire all the builds at once
builders[label] = {
node('JenkinsNode') {
stage("build") {
sh script: 'echo build', label: 'Build on $env.NODE_NAME'
}
stage("test") {
sh script: 'echo run unit tests', label: 'Run unit tests on $env.NODE_NAME'
}
}
}
}
parallel builders
}
}
}
}
}
The Jenkins logs show both build and test stages are run for each parallel step, but the Blue Ocean view only states build stage:
I would expect something like:
I'm not very clear about the boundaries between declarative and scripted pipelines, but I suspect a misunderstanding around this.
Attempt#2
Following a suggestion in comments, I slightly changed the code to have sub-stages unique names (build1, test1, build2, test2) and it does not change the diagram. I still have build steps only.
Here are the Jenkins logs in this case:
Question: Is the pipeline invalid (leading to only "build" sub-steps instead of build + test sub-steps) or is it a limitation of Blue Ocean (1.25.3)?

When combining declarative and scripted syntax things become a bit tricky.
In this specific combination case, to make it work like you expect you must create an encapsulating stage for each parallel execution code that has the same name as the parallel branch.
This will cause the blue ocean to display the inner stages as requested.
pipeline {
agent any
stages {
stage('Build') {
steps {
script {
def labels = ['precise', 'trusty']
def builders = [:]
for (x in labels) {
def label = x
builders[label] = {
stage(label) { // Encapsulating stage with same name as parallel branch
node('JenkinsNode') {
stage("build") {
sh script: 'echo build', label: 'Build on $env.NODE_NAME'
}
stage("test") {
sh script: 'echo run unit tests', label: 'Run unit tests on $env.NODE_NAME'
}
}
}
}
}
parallel builders
}
}
}
}
}
Or in a more Groovy way:
pipeline {
agent any
stages {
stage('Build') {
steps {
script {
def labels = ['precise', 'trusty']
def builders = labels.collectEntries {
["${it}" : {
stage(it) { // Encapsulating stage with same name as parallel branch
node('JenkinsNode') {
stage("build") {
sh script: 'echo build', label: 'Build on $env.NODE_NAME'
}
stage("test") {
sh script: 'echo run unit tests', label: 'Run unit tests on $env.NODE_NAME'
}
}
}
}]
}
parallel builders
}
}
}
}
}
The result:

Related

What is correct way to write a condition using when in Jenkinsfile?

I am trying to write when statement in the single stage block in Jenkinsfile. I have tried to write as below. I know it's not the correct way to write. It's a declarative pipeline script. The pipeline expects only a single when block. How can I combine both of my when blocks and write as a single when.
stages{
stage('Approve Dev Deployment') {
agent { label 'docker-kitchensink-slave' }
when {
anyOf {
expression {
return (env.GIT_BRANCH.equals('master') || env.GIT_BRANCH.startsWith('hotfix-'))
}
}
}
when {
expression {
input message: 'Deploy test?'
return true
}
beforeAgent true
}
steps{
approveDeployment()
}
}
}
Write function with the conditions outside the pipeline scope and use the function as a condition.
def checkcondition(){
your_condition
}
stages{
stage('Approve Dev Deployment') {
agent { label 'docker-kitchensink-slave' }
when { checkcondition() }
steps{
approveDeployment()
}
}
}

How to resolve "please provide compiled classes with sonar.java.binaries property"?

I tried all the possible solutions posted by other people but still I am struggling to resolve this issue. I believe in my case, it has got to do something with the agents. I will post 2 codes, the one code works but the other doesn't. Both the codes call the same groovy methods but still the second code snippet doesn't work.
The below code 1 works fine and executed the pipeline successfully:
pipeline{
agent { label 'docker-kitchensink-slave' }
stages{
stage('Checkout') {
steps{
checkout scm
}
}
//Build and Unit Tests
stage('Build and Unit Tests') {
steps{
script{
if (buildType.buildSystem == 'npm'){
buildNpm(configuration)
} else {
build(configuration)
}
}
}
}
// SonarQube Analysis
stage('SonarQube analysis') {
steps{
script{
if (buildType.buildSystem != 'npm'){
sonarQubeGating(configuration)
}
}
}
}
// Build Docker Image and Push to Artifactory
stage('Build Docker Image and Push to Artifactory') {
steps{
artifactoryImagePush(configuration)
}
}
// Approve DEV Deployment
stage('Approve Dev Deployment') {
agent none
when {
anyOf {
expression {
return (env.GIT_BRANCH.equals('master') || env.GIT_BRANCH.startsWith('hotfix-'))
}
}
}
steps{
approveDeployment()
}
}
}
}
The below code 2 doesn't work:
pipeline{
agent none
stages{
stage('Checkout') {
agent { label 'docker-kitchensink-slave' }
steps{
checkout scm
}
}
//Build and Unit Tests
stage('Build and Unit Tests') {
agent { label 'docker-kitchensink-slave' }
steps{
script{
if (buildType.buildSystem == 'npm'){
buildNpm(configuration)
} else {
build(configuration)
}
}
}
}
// SonarQube Analysis
stage('SonarQube analysis') {
agent { label 'docker-kitchensink-slave' }
steps{
script{
if (buildType.buildSystem != 'npm'){
sonarQubeGating(configuration)
}
}
}
}
// Build Docker Image and Push to Artifactory
stage('Build Docker Image and Push to Artifactory') {
agent { label 'docker-kitchensink-slave' }
steps{
unstash 'artifacts'
unstash 'artifacts'
artifactoryImagePush(configuration)
}
}
// Approve DEV Deployment
stage('Approve Dev Deployment') {
agent none
when {
anyOf {
expression {
return (env.GIT_BRANCH.equals('master') || env.GIT_BRANCH.startsWith('hotfix-'))
}
}
}
steps{
approveDeployment()
}
}
}
}
I get the error as below:
[ERROR] Failed to execute goal org.sonarsource.scanner.maven:sonar-maven-plugin:3.6.0.1398:sonar (default-cli) on project xyz-service: Your project contains .java files, please provide compiled classes with sonar.java.binaries property, or exclude them from the analysis with sonar.exclusions property. -> [Help 1]
Below is my sonar code:
void call(Map optionParams = [:]) {
script {
try {
String jacocoPath = optionParams.get('buildSystem').equals('gradle') ?
'build/JacocoReport/test/jacocoTestReport.xml' : 'target/site/jacoco/jacoco.xml'
glSonarMavenScan gitUserCredentialsId: 'sonar-key', javaVersionForSonar: '11.0', mavenVersion: '3.5.4',
additionalProps: ['sonar.coverage.jacoco.xmlReportPaths' : jacocoPath]
} catch (Exception e) {
echo "The following Sonar exception thrown ${e}"
//Stop build here, unless 'requireSonar' is set to False (String or Boolean)
if (!optionParams.get('requireSonar').toString().equalsIgnoreCase('false')) {
throw e
}
}
}
}
I'm a little confused about what you're trying to achieve here. You are showing the code that works. Are you trying to understand WHY the first block works, compared to the second, or are you just trying to get it working? If the latter, clearly you are already done.
If the former, I'm only familiar with scripted pipeline, not declarative pipeline, but it seems possible to me that if there is more than one build node that satisfies that label, then each of those "agent" lines could potentially select a build node, and each one could potentially select a different one. If the build step executes a different node than the sonarqube scan is run on, you will find yourself in a workspace without any compiled classes.

Scripted Jenkinsfile executes scheduling on wrong nodes

The system/setup
Here is my setup:
1 machine running Jenkins which also provides the built-in node (former master). It has 4 workers assigned to it
1 FreeBSD machine as remote SSH node with the name FreeBSD 13.0 x86_64. It has 1 worker
1 Ubuntu machine as remote SSH node with the name Ubuntu 20.04 x86_64. It has 1 worker
I would like to have a sequence of Checkout, Build, Test and Upload steps executed serially on each of those FreeBSD 13.0 x86_64 and Ubuntu 20.04 x86_64 nodes. However, I would like each of those nodes to work independently on their list of serial tasks. Something like this:
For this, I have created the following scripted Jenkinsfile
String[] nodesNames = [
'FreeBSD 13.0 x86_64',
'Ubuntu 20.04 x86_64'
]
Map builders = [:]
for (nodeName in nodesNames) {
builders[nodeName] = {
node(nodeName) {
stage('Checkout') {
sh 'sleep 5'
}
stage('Build') {
sh 'sleep 10'
}
stage('Test') {
sh 'sleep 15'
}
stage('Upload') {
sh 'sleep 20'
}
}
}
}
parallel builders
The problem
Jenkins will need an executor to execute the Jenkinsfile itself, which is scheduling the real work. Again, not the work, but the scheduling of the work. And it picks one up from the nodes like this:
Running on Ubuntu 20.04 x86_64 in /home/jenkins/workspace/test
That is the problem: it picks up the wrong node. That node has one executor, and it should never run the scheduling. Scheduling should be done on the built-in node. This will later result in the following message:
Still waiting to schedule task
Waiting for next available executor on ‘Ubuntu 20.04 x86_64’
The outcome
Because one of the 1-worker-node is doing both scheduling and actual work, we will end up with a seemengly "parallel" execution but in fact, is as serial as it can be. Here is a picture taken in the middle of the whole process. Notice how the FreeBSD machine is alternatively doing work then scheduling. As it happens, is giving work to itself. When the work for itself is finished, it will start giving work to Ubuntu.
The solution?
How can one tell Jenkins to execute the Jenkins file itself (the scheduling part) on the built-in node (the former master) and not use a precious worker from the actual remote nodes?
The non-maintainable solution
(Update after initial question and as a response for #MaratC)
We can use declarative syntax. However, it has a major/crippling flaw: imagine one needs to add another machine. It will basically repeat a lot of code. After the 4th-5th machine, it becomes unmaintainable.
pipeline {
agent { node('built-in') }
stages {
stage('Build all') {
parallel {
stage('FreeBSD') {
agent { node('FreeBSD 13.0 x86_64') }
stages {
stage('Checkout') {
steps { sh 'sleep 5' }
}
stage('Build') {
steps { sh 'sleep 10' }
}
stage('Test') {
steps { sh 'sleep 15' }
}
stage('Upload') {
steps { sh 'sleep 20' }
}
}
}
stage('Ubuntu') {
agent { node('Ubuntu 20.04 x86_64') }
stages {
stage('Checkout') {
steps { sh 'sleep 5' }
}
stage('Build') {
steps { sh 'sleep 10' }
}
stage('Test') {
steps { sh 'sleep 15' }
}
stage('Upload') {
steps { sh 'sleep 20' }
}
}
}
}
}
}
}
First of all, you can combine declarative and scripted syntax (note I didn't check the following code):
pipeline {
agent { node('built-in') }
stages {
stage('Build all') {
script {
def myBuilders = getParallelBuilders()
parallel myBuilders
}
}
}
}
def getParallelBuilders() {
String[] nodesNames = [
'FreeBSD 13.0 x86_64',
'Ubuntu 20.04 x86_64'
]
Map builders = [:].asSynchronized() // don't ask
for (nodeName in nodesNames) {
def final_name = nodeName // don't ask
builders[final_name] = {
node(final_name) {
stage('Checkout') {
sh 'sleep 5'
}
stage('Build') {
sh 'sleep 10'
}
stage('Test') {
sh 'sleep 15'
}
stage('Upload') {
sh 'sleep 20'
}
}
}
return builders
}
But I think that your problem may be solved faster by disallowing the planning code to run on Ubuntu and FreeBSD nodes, by configuring these nodes to only run what is planned to run on the labels (and not just everything). This is achieved by selecting "Only build jobs with label expressions matching this node" in the node configuration screen.
The solution to original scripted Jenkinsfile question
String[] nodesNames = [
'FreeBSD 13.0 x86_64',
'Ubuntu 20.04 x86_64'
]
def tasks = [:]
def createTask(tasks, title, nodeName) {
tasks[title] = {
stage (title) {
node(nodeName) {
stage('Checkout') {
sh 'hostname; sleep 1'
}
stage('Build') {
sh 'hostname; sleep 2'
}
stage('Test') {
sh 'hostname; sleep 3'
}
stage('Upload') {
sh 'hostname; sleep 4'
}
}
}
}
}
for (nodeName in nodesNames) {
createTask(tasks, nodeName, nodeName)
}
node('built-in') {
parallel tasks
}
It builds everything as expected:
Solution proposed by #MaratC:
(Thank you for the suggestion)
It was slight/minor changed to make it work, and it looks like this:
pipeline {
agent { node('built-in') }
stages {
stage('Build all') {
steps {
script {
def myBuilders = getParallelBuilders()
parallel myBuilders
}
}
}
}
}
def getParallelBuilders() {
String[] nodesNames = [
'FreeBSD 13.0 x86_64',
'Ubuntu 20.04 x86_64'
]
//org.jenkinsci.plugins.scriptsecurity.sandbox.RejectedAccessException: //
// Scripts not permitted to use staticMethod org.codehaus.groovy.runtime.DefaultGroovyMethods asSynchronized java.util.Map
Map builders = [:]//.asSynchronized() // don't ask:
for (nodeName in nodesNames) {
def final_name = nodeName // don't ask
builders[final_name] = {
node(final_name) {
stage('Checkout') {
sh 'hostname; sleep 1'
}
stage('Build') {
sh 'hostname; sleep 2'
}
stage('Test') {
sh 'hostname; sleep 3'
}
stage('Upload') {
sh 'hostname; sleep 4'
}
}
}
}
return builders
}
And the behavior:
Notice that the Checkout step is executed, and the only step is listed in the graph. However, notice the blue ongoing task below. Something is off.

Jenkins pipelines with parallel and different containers

So I am already running Jenkins pipelines with parallel base on the example from: Is it possible to create parallel Jenkins Declarative Pipeline stages in a loop?
I want to run each job in different isolated container, the agent name should be the same to all of them. Tried a few options all of them ended up withe errors, I think I need to use both declarative and scripted but not sure how.
Things I tired:
def generateTerraformStage(env) {
return {
agent { label 'local_terraform' }
stage("stage: Terraform ${TERRAFORM_ACTION} ${env}") {
echo "${env}"
sleep 30
}
}
}
stage('parallel stages') {
agent { label 'local_terraform' }
steps {
script {
parallel parallelStagesMapEnvironment
}
}
}
One of the errors I got during testing:
"java.lang.NoSuchMethodError: No such DSL method 'agent' found among steps" and "java.lang.IllegalArgumentException: Expected named arguments but got org.jenkinsci.plugins.workflow.cps.CpsClosure2#560f3533"
Dynamic parallel stages could be created only by using Scripted Pipelines. The API built-it Declarative Pipeline is not available (like agent, options, when etc.).
I don't see any information that you really need dynamic stages (e.g. based on the value returned by a 3rd-party service), so I prepared two solutions:
dynamic parallel stages - stages are generated based on something
static parallel stages - you know all stages (the when block could be used to disable these which are not needed - e.g. passed in parameters)
pipeline {
// ...
stages {
stage('dynamic parallel stages') {
steps {
script {
// params.ENVS == ['envA', 'envB', 'envC']
def values = params.ENVS.split(',')
def stages = [:]
for (def value in values) {
stages[value] = generateTerraformStage(value)
}
parallel stages
}
}
}
stage('static parallel stages') {
parallel {
stage('envA') {
agent { label 'local_terraform' }
when {
expression { return params.ENVS.split(',').contains('envA') }
}
steps {
terraformStageLogic 'envA'
}
}
stage('envB') {
agent { label 'local_terraform' }
when {
expression { return params.ENVS.split(',').contains('envB') }
}
steps {
terraformStageLogic 'envB'
}
}
stage('envC') {
agent { label 'local_terraform' }
when {
expression { return params.ENVS.split(',').contains('envC') }
}
steps {
terraformStageLogic 'envC'
}
}
// ...
}
}
}
}
Closure<Void> generateTerraformStage(env) {
return {
node('local_terraform') {
stage("stage: Terraform ${TERRAFORM_ACTION} ${env}") {
echo "${env}"
sleep 30
}
}
}
}
void terraformStageLogic(env) {
echo "${env}"
sleep 30
}
When you don't use the workspace in the stage responsible for generating or executing other stages (dynamic parallel stages and static parallel stages) then you don't need to allocate any node to it (waste of resources).

How to write a dynamic declarative pipeline that contains sequential job inside parallel job

I'm trying to write a declarative pipeline code that accepts a map and create a pipeline. I can able to achieve sequential stages or parallel stages but facing problems while making a pipeline that contains sequential stages inside parallel stages.
The input data would be Map. Each list in the map should run parallel and the items inside the list corresponding to each key should run in sequentially.
example data : [1:[11,12], 2:[21,22], 3:[31,32]]
The output should be of image. Could someone give some idea?
Below is the code i have tried.
def stageData = [1:[11,12], 2:[21,22], 3:[31,32]];
def getDeployStages1(stageData){
Map deployStages = [:]
stageData.each{ key, stgValue ->
List stgs = []
stgValue.each{ value ->
deployStages.put("${value}", {
echo "${value}"
})
}
}
return deployStages;
}
def getDeployStages2(stageData){
Map deployStages = [:]
stageData.each{ key, stgValue ->
List stgs = []
stgValue.each{ value ->
stgs.add(stage("${value}"){
echo "${value}"
})
}
deployStages.put("${key}", stgs)
}
return deployStages;
}
pipeline {
agent any
stages {
stage ("deploy1") {
steps {
script {
parallel getDeployStages1(stageData)
}
}
}
stage ("deploy2") {
steps {
script {
parallel getDeployStages2(stageData)
}
}
}
}
}
According to this documentation you can nest the stages in this way
pipeline {
agent none
stages {
stage("build and deploy on Windows and Linux") {
parallel {
stage("windows") {
agent {
label "windows"
}
stages {
stage("build") {
steps {
bat "run-build.bat"
}
}
stage("deploy") {
when {
branch "master"
}
steps {
bat "run-deploy.bat"
}
}
}
}
stage("linux") {
agent {
label "linux"
}
stages {
stage("build") {
steps {
sh "./run-build.sh"
}
}
stage("deploy") {
when {
branch "master"
}
steps {
sh "./run-deploy.sh"
}
}
}
}
}
}
}
}
This should result in the following flow
To apply this in your case, you can simplify your functions to return just elements that need to be sequential (just the values).
pipeline {
agent any
stages {
stage ("parallel") {
parallel {
stage ("deploy1") {
stages {
def list = getDeployStages1(stageData)
for (int i=0; i < list.size(); i++) {
stage(i) {
echo("${list[i]}")
}
}
}
stage ("deploy2") {
stages {
//similar
}
}
}
}
}

Resources