How to write a dynamic declarative pipeline that contains sequential job inside parallel job - jenkins-pipeline

I'm trying to write a declarative pipeline code that accepts a map and create a pipeline. I can able to achieve sequential stages or parallel stages but facing problems while making a pipeline that contains sequential stages inside parallel stages.
The input data would be Map. Each list in the map should run parallel and the items inside the list corresponding to each key should run in sequentially.
example data : [1:[11,12], 2:[21,22], 3:[31,32]]
The output should be of image. Could someone give some idea?
Below is the code i have tried.
def stageData = [1:[11,12], 2:[21,22], 3:[31,32]];
def getDeployStages1(stageData){
Map deployStages = [:]
stageData.each{ key, stgValue ->
List stgs = []
stgValue.each{ value ->
deployStages.put("${value}", {
echo "${value}"
})
}
}
return deployStages;
}
def getDeployStages2(stageData){
Map deployStages = [:]
stageData.each{ key, stgValue ->
List stgs = []
stgValue.each{ value ->
stgs.add(stage("${value}"){
echo "${value}"
})
}
deployStages.put("${key}", stgs)
}
return deployStages;
}
pipeline {
agent any
stages {
stage ("deploy1") {
steps {
script {
parallel getDeployStages1(stageData)
}
}
}
stage ("deploy2") {
steps {
script {
parallel getDeployStages2(stageData)
}
}
}
}
}

According to this documentation you can nest the stages in this way
pipeline {
agent none
stages {
stage("build and deploy on Windows and Linux") {
parallel {
stage("windows") {
agent {
label "windows"
}
stages {
stage("build") {
steps {
bat "run-build.bat"
}
}
stage("deploy") {
when {
branch "master"
}
steps {
bat "run-deploy.bat"
}
}
}
}
stage("linux") {
agent {
label "linux"
}
stages {
stage("build") {
steps {
sh "./run-build.sh"
}
}
stage("deploy") {
when {
branch "master"
}
steps {
sh "./run-deploy.sh"
}
}
}
}
}
}
}
}
This should result in the following flow
To apply this in your case, you can simplify your functions to return just elements that need to be sequential (just the values).
pipeline {
agent any
stages {
stage ("parallel") {
parallel {
stage ("deploy1") {
stages {
def list = getDeployStages1(stageData)
for (int i=0; i < list.size(); i++) {
stage(i) {
echo("${list[i]}")
}
}
}
stage ("deploy2") {
stages {
//similar
}
}
}
}
}

Related

Parallel pipeline with 2 inline stages

Playing with Jenkins pipeline from https://www.jenkins.io/doc/pipeline/examples/#parallel-multiple-nodes
Simple two parallel steps (OK)
I made a first test pipeline this way:
pipeline {
stages {
stage('Build') {
steps {
script {
def labels = ['precise', 'trusty'] // labels for Jenkins node types we will build on
def builders = [:]
for (x in labels) {
def label = x // Need to bind the label variable before the closure - can't do 'for (label in labels)'
// Create a map to pass in to the 'parallel' step so we can fire all the builds at once
builders[label] = {
node('JenkinsNode') {
sh script: 'echo build', label: 'Build on $env.NODE_NAME'
}
}
}
parallel builders
}
}
}
}
}
It resulted in the following expected diagram in Blue Ocean view:
Simple two parallel steps with two sub steps each (KO)
Attempt#1
Then I tried to split each parallel step in two inline stages (to simulate build and tests for example)
pipeline {
stages {
stage('Build') {
steps {
script {
def labels = ['precise', 'trusty'] // labels for Jenkins node types we will build on
def builders = [:]
for (x in labels) {
def label = x // Need to bind the label variable before the closure - can't do 'for (label in labels)'
// Create a map to pass in to the 'parallel' step so we can fire all the builds at once
builders[label] = {
node('JenkinsNode') {
stage("build") {
sh script: 'echo build', label: 'Build on $env.NODE_NAME'
}
stage("test") {
sh script: 'echo run unit tests', label: 'Run unit tests on $env.NODE_NAME'
}
}
}
}
parallel builders
}
}
}
}
}
The Jenkins logs show both build and test stages are run for each parallel step, but the Blue Ocean view only states build stage:
I would expect something like:
I'm not very clear about the boundaries between declarative and scripted pipelines, but I suspect a misunderstanding around this.
Attempt#2
Following a suggestion in comments, I slightly changed the code to have sub-stages unique names (build1, test1, build2, test2) and it does not change the diagram. I still have build steps only.
Here are the Jenkins logs in this case:
Question: Is the pipeline invalid (leading to only "build" sub-steps instead of build + test sub-steps) or is it a limitation of Blue Ocean (1.25.3)?
When combining declarative and scripted syntax things become a bit tricky.
In this specific combination case, to make it work like you expect you must create an encapsulating stage for each parallel execution code that has the same name as the parallel branch.
This will cause the blue ocean to display the inner stages as requested.
pipeline {
agent any
stages {
stage('Build') {
steps {
script {
def labels = ['precise', 'trusty']
def builders = [:]
for (x in labels) {
def label = x
builders[label] = {
stage(label) { // Encapsulating stage with same name as parallel branch
node('JenkinsNode') {
stage("build") {
sh script: 'echo build', label: 'Build on $env.NODE_NAME'
}
stage("test") {
sh script: 'echo run unit tests', label: 'Run unit tests on $env.NODE_NAME'
}
}
}
}
}
parallel builders
}
}
}
}
}
Or in a more Groovy way:
pipeline {
agent any
stages {
stage('Build') {
steps {
script {
def labels = ['precise', 'trusty']
def builders = labels.collectEntries {
["${it}" : {
stage(it) { // Encapsulating stage with same name as parallel branch
node('JenkinsNode') {
stage("build") {
sh script: 'echo build', label: 'Build on $env.NODE_NAME'
}
stage("test") {
sh script: 'echo run unit tests', label: 'Run unit tests on $env.NODE_NAME'
}
}
}
}]
}
parallel builders
}
}
}
}
}
The result:

What is correct way to write a condition using when in Jenkinsfile?

I am trying to write when statement in the single stage block in Jenkinsfile. I have tried to write as below. I know it's not the correct way to write. It's a declarative pipeline script. The pipeline expects only a single when block. How can I combine both of my when blocks and write as a single when.
stages{
stage('Approve Dev Deployment') {
agent { label 'docker-kitchensink-slave' }
when {
anyOf {
expression {
return (env.GIT_BRANCH.equals('master') || env.GIT_BRANCH.startsWith('hotfix-'))
}
}
}
when {
expression {
input message: 'Deploy test?'
return true
}
beforeAgent true
}
steps{
approveDeployment()
}
}
}
Write function with the conditions outside the pipeline scope and use the function as a condition.
def checkcondition(){
your_condition
}
stages{
stage('Approve Dev Deployment') {
agent { label 'docker-kitchensink-slave' }
when { checkcondition() }
steps{
approveDeployment()
}
}
}

Jenkins pipelines with parallel and different containers

So I am already running Jenkins pipelines with parallel base on the example from: Is it possible to create parallel Jenkins Declarative Pipeline stages in a loop?
I want to run each job in different isolated container, the agent name should be the same to all of them. Tried a few options all of them ended up withe errors, I think I need to use both declarative and scripted but not sure how.
Things I tired:
def generateTerraformStage(env) {
return {
agent { label 'local_terraform' }
stage("stage: Terraform ${TERRAFORM_ACTION} ${env}") {
echo "${env}"
sleep 30
}
}
}
stage('parallel stages') {
agent { label 'local_terraform' }
steps {
script {
parallel parallelStagesMapEnvironment
}
}
}
One of the errors I got during testing:
"java.lang.NoSuchMethodError: No such DSL method 'agent' found among steps" and "java.lang.IllegalArgumentException: Expected named arguments but got org.jenkinsci.plugins.workflow.cps.CpsClosure2#560f3533"
Dynamic parallel stages could be created only by using Scripted Pipelines. The API built-it Declarative Pipeline is not available (like agent, options, when etc.).
I don't see any information that you really need dynamic stages (e.g. based on the value returned by a 3rd-party service), so I prepared two solutions:
dynamic parallel stages - stages are generated based on something
static parallel stages - you know all stages (the when block could be used to disable these which are not needed - e.g. passed in parameters)
pipeline {
// ...
stages {
stage('dynamic parallel stages') {
steps {
script {
// params.ENVS == ['envA', 'envB', 'envC']
def values = params.ENVS.split(',')
def stages = [:]
for (def value in values) {
stages[value] = generateTerraformStage(value)
}
parallel stages
}
}
}
stage('static parallel stages') {
parallel {
stage('envA') {
agent { label 'local_terraform' }
when {
expression { return params.ENVS.split(',').contains('envA') }
}
steps {
terraformStageLogic 'envA'
}
}
stage('envB') {
agent { label 'local_terraform' }
when {
expression { return params.ENVS.split(',').contains('envB') }
}
steps {
terraformStageLogic 'envB'
}
}
stage('envC') {
agent { label 'local_terraform' }
when {
expression { return params.ENVS.split(',').contains('envC') }
}
steps {
terraformStageLogic 'envC'
}
}
// ...
}
}
}
}
Closure<Void> generateTerraformStage(env) {
return {
node('local_terraform') {
stage("stage: Terraform ${TERRAFORM_ACTION} ${env}") {
echo "${env}"
sleep 30
}
}
}
}
void terraformStageLogic(env) {
echo "${env}"
sleep 30
}
When you don't use the workspace in the stage responsible for generating or executing other stages (dynamic parallel stages and static parallel stages) then you don't need to allocate any node to it (waste of resources).

Jenkins pipeline RejectedAccessException error

I am trying to setup a jenkins pipeline script that sends out an email when there is a job that have been running for more than 24 hours.
// Long running jobs
pipeline {
agent any
environment {
EMAIL_ALERT_TO = "address"
EMAIL_ALERT_CC = "address"
}
stages {
stage('def methods') {
steps {
script {
Jenkins.instance.getAllItems(Job).each(){ job -> job.isBuildable()
if (job.isBuilding()){
def myBuild= job.getLastBuild()
def runningSince= groovy.time.TimeCategory.minus( new Date(), myBuild.getTime() )
echo "myBuild = ${myBuild}"
echo "runningSince = ${runningSince}"
env.myBuild = myBuild
env.runningSince = runningSince
}
}
}
}
}
}
post {
// Email out the results
always {
script {
if (runningSince.hours >= 1){
mail to: "${env.EMAIL_ALERT_CC}",
cc: "${env.EMAIL_ALERT_CC}",
subject: "Long Running Jobs",
body: "Build: ${myBuild} ---- Has Been Running for ${runningSince.hours} hours:${runningSince.minutes} minutes"
}
}
}
}
}
I am seeing RejectedAccessException which appears to be related to arrays/list.
This is what I believe you are looking for
https://issues.jenkins-ci.org/browse/JENKINS-54952?page=com.atlassian.jira.plugin.system.issuetabpanels%3Achangehistory-tabpanel

Jenkins pipeline stage inside another stage based on if condition

I am trying to write a stages inside another stage based on if condition. I am not able to come up with a solution. Can anyone guide on this
stages {
stage('Example') {
steps {
script {
if(!(fileExists("c:/test.txt")))
{
echo "Inside if"
stage('1') {
echo "stage1"
}
stage('2') {
echo "stage2"
}
}
else
{
stage('else stage') {
echo "else stage1"
}
}
}
}
}
}
This worked for me.
when {
expression
{
return !(fileExists("c:/test.txt"))
}
}

Resources