jenkins pipeline fails without some explanatory exception - amazon-ec2

I have a pipeline where among other parameters like machine name, client it is supposed to get latest ami from aws for that branch for example and then put it the clien.json which terraform would use to create the machine but als I want to enable the user to be able to provide a value for the parameter and when there is no value for that parameter to have the value picked from latest ami in develop for example:
#!/usr/bin/env groovy
pipeline {
agent { label 'new' }
parameters {
string(name: 'AMI_ID', defaultValue: '', description: '[Mandatory]')
}
stages {
stage('Retrieve latest AMI.') {
when {
expression { ${AMI_ID} == '' }
}
steps {
script {
AMI_ID = sh(script: "aws ec2 describe-images --region region1 --owners 123456 --filters \"Name=tag:type,Values=develop\" --query 'sort_by(Images, &CreationDate)[-1].ImageId' | jq -r '.'", returnStdout: true).trim()
echo "AMI retrieved: " + $ { AMI_ID }
}
}
}
stage("Updating client data") {
environment {
TERRAHELP_KEY = credentials('some-key')
}
steps {
dir("data/clients/") {
clientJson = readJSON file: "${CLIENT}.json"
clientJson.put("client_ec2_eda_ami_id", ${ AMI_ID })
writeJSON file: "${CLIENT}.json", json: clientJson, pretty: 4
echo "Following data will be applied:"
sh "cat ${CLIENT}.json"
}
}
}
}
}

Found the root cause was that I was passing a shell script variable to groovy where I say:
clientJson.put("client_ec2_eda_ami_id", ${ AMI_ID })
Instead I should have passed the AMI_ID to a groovy variable above and say like:
clientJson.put("client_ec2_eda_ami_id", currentAmi)
Peace of the code with the aws query dimmed:
stage('Retrieve latest AMI.') {
}
steps {
script {
currentAmi = params.AMI_ID
if (currentAmi.isEmpty())
currentAmi = sh(script: "aws ec2 query blah blah")
echo "Ami retrieved is: ${currentAmi}"
}
}
}
stage("Updating client data") {
environment {
TERRAHELP_KEY = credentials('some-key')
}
steps {
dir("data/clients/") {
clientJson = readJSON file: "${CLIENT}.json"
clientJson.put("client_ec2_eda_ami_id", currentAmi)
writeJSON file: "${CLIENT}.json", json: clientJson, pretty: 4
echo "Following data will be applied:"
sh "cat ${CLIENT}.json"
}
}
}
}
}

Related

Jenkins declarative when condition to check if a variable is NULL

I want to skip Build stage if AMI already exists using declarative syntax.
stage('Build') {
environment {
AMI = sh(returnStdout: true, script: 'aws ec2 describe-images').trim()
}
when {
expression { AMI = null }
}
steps {
sh 'packer build base.json -machine-readable'
}
}
But when I'm running this pipeline I get groovy.lang.MissingPropertyException: No such property: AMI for class: groovy.lang.Binding
At the same time scripted pipeline works perfectly fine
stage('Build') {
steps {
script {
env.AMI = sh(returnStdout: true, script: 'aws ec2 describe-images').trim()
if (env.AMI == '') {
sh 'packer build base.json -machine-readable'
}
}
}
}
}
I'd really love to switch to the declarative pipelines just stuck with this condition. Any help is really appreciated. Thanks
I tried a lot things without any luck
when {
expression {
return AMI.isEmpty()
}
}
when {
not {
expression {
AMI == ''
}
}
when {
not {
expression { env.AMI }
}
}
Nothing works. I suspect it is somehow related to env variable association through sh
You can do something like this.
pipeline {
agent any
stages {
stage('Build') {
when {
expression {
return isAMIAvailable()
}
}
steps {
sh 'packer build base.json -machine-readable'
}
}
}
}
def isAMIAvailable() {
AMI = sh(returnStdout: true, script: 'aws ec2 describe-images').trim()
return AMI == null
}
With #ycr help I was able to build it!
Just in case here is the whole thing
pipeline {
agent any
environment {
ENV = 'dev'
}
stages {
stage('Build') {
when { not { expression { return isAMIAvailable() } } }
steps {
sh 'packer build base.json -machine-readable'
}
}
}
}
def isAMIAvailable() {
AMI = sh(returnStdout: true, script: "aws ec2 describe-images --owners self --filters 'Name=name,Values=base-${ENV}-1' --query 'Images[*].[Name]' --output text").trim()
if (AMI == '') {
return AMI == null
}
return AMI
}

Jenkins dynamic Pipeline not showing all stages in BlueOcean

I have a Jenkins pipeline script with the key pieces shown below. Everything seems to be executing in the background but is not showing on BlueOcean.
Am I doing something wrong or it is some kind of bug on the UI?
def projects = [
[name: 'API', folder: 'api-tests', repo: 'url'],
[name: 'UI',folder: 'selenium-tests', repo: 'url']
]
// Create List of build stages to suit
def jobs = [:]
pipeline {
agent {
kubernetes {
cloud "url"
yaml """contents"""
}
}
stages {
stage('Downstream Testing') {
steps {
script{
// Set up List<Map<String,Closure>> describing the builds
projects.each { project ->
print "Adding stages for ${project.folder}"
jobs[project.name] = {
stage("Checkout Project") {
...
}
stage("Smoke Tests") {
dir("${project.folder}") {
container('maven') {
// Run the maven build
sh """mvn package test"""
}
}
}
}
}
// Run all Nth step for all Projects in Parallel.
parallel jobs
}
}
}
}
}
Yes, It seems there is a bug in the Blueocean UI.
The scenario works fine for the scripted pipeline but the same fail in declarative pipeline
scripted pipeline
def j=[:]
node {
[
[name: "api", folder: "api"],
[name: "ui", folder: "ui"]
].each { m ->
j[m.name] = {
stage('a') {
echo "A"
}
stage('b') {
echo "B"
}
}
}
parallel j
}
declarative pipeline
pipeline {
agent any;
stages {
stage("parallel") {
steps {
script {
def j=[:]
[
[name: "api", folder: "api"],
[name: "ui", folder: "ui"]
].each { m ->
j[m.name] = {
stage('a') {
echo "A"
}
stage('b') {
echo "B"
}
}
}
parallel j
}
}
}
}
}

How to pass paramaters to remote script in a Jenkins pipeline

I am working on a declarative jenkins pipeline where I am trying to login to remote host and execute shell script. Below is the sample snippet. I want to know How to pass a parameter to my script.sh script.
#!/bin/bash
echo "argument $1"
below is the pipeline script
hosts = [“x.x.x”, “x.x.x”]
pipeline {
agent { node { label 'docker' } }
parameters {
choice(name: 'stageparam', choices: ['build', 'deploy'], description: ‘xyz’)
string(name: 'Username', defaultValue: ‘abc’, description: 'enter username')
}
stages {
stage('Setup') {
steps {
script {
pom = getPom(effective: false)
}
}
}
stage('Deploy') {
steps {
script {
def targetServers = null
if (stageparam == "deploy") {
targetServers = hosts
}
targetServers.each { server ->
echo "Server : ${server}"
def remote = [:]
remote.name = ‘server’
remote.host = server
remote.user = Username
def pass = passwordParameter description: "Enter password for user ${remote.user} "
remote.password = pass
remote.allowAnyHosts = true
stage('Remote SSH') {
sshPut remote: remote, from: ‘./script.sh', into: '.'
sshScript remote: remote, script: "doc.sh ${Username}"
}
}
}
}
}
}
}
getting below error while executing the script
/home/jenkins/workspace/script.sh Username does not exist.
I have written a simple pipeline script to show the usage of variable in bash script.
Pipeline script:
pipeline {
agent any
parameters {
choice(name: 'stageparam', choices: ['build', 'deploy'], description: 'enter stageparam')
string(name: 'USERNAME', defaultValue: 'abc', description: 'enter username')
}
stages {
stage('Git Pull')
{
steps {
checkout([$class: 'GitSCM', branches: [[name: '*/master']], doGenerateSubmoduleConfigurations: false, extensions: [], submoduleCfg: [], userRemoteConfigs: [[credentialsId: 'gitlab-test', url: 'http://localhost:8076/test-upgrade.git']]])
}
}
stage('Run the python script') {
steps {
sh 'chmod 777 test.py'
sh "python test.py ${env.WORKSPACE}"
}
}
stage('Run the bash script') {
steps {
sh 'chmod 777 run.sh'
sh "./run.sh ${env.USERNAME}"
}
}
}
}
Output:

Terraform remote state in Jenkins

am working on a declarative Jenkins pipeline for Terraform deployments. I want to have the terraform state stored remotely in my s3 I get an error
terraform apply -input=false ow.plan
[31mFailed to load Terraform configuration or plan: open ow.plan: no such file or directory
Any suggestions?
Here's my code:
pipeline {
agent any
tools {
"org.jenkinsci.plugins.terraform.TerraformInstallation" "terraform"
}
parameters {
choice(name: 'action', choices: 'create\ndestroy', description: 'Create/update or destroy the eks cluster.')
string(name: 'cluster', defaultValue : 'demo', description: "EKS cluster name;eg demo creates cluster named eks-demo.")
string(name: 'vpc_network', defaultValue : '10.0', description: "First 2 octets of vpc network; eg 10.0")
string(name: 'num_subnets', defaultValue : '3', description: "Number of vpc subnets/AZs.")
string(name: 'instance_type', defaultValue : 'm4.large', description: "k8s worker node instance type.")
string(name: 'num_workers', defaultValue : '3', description: "k8s number of worker instances.")
string(name: 'api_ingress_cidrs', defaultValue : '0.0.0.0/0', description: "k8s api ingress cidrs; space delimited list.")
string(name: 'jenkinsfile', defaultValue : '', description: "Jenkins credential that provides the AWS access key and secret.")
string(name: 'region', defaultValue : 'eu-west-1', description: "AWS region.")
}
environment {
TF_HOME = tool('terraform')
TF_IN_AUTOMATION = "true"
PATH = "$TF_HOME:$PATH"
AWS_ACCESS_KEY_ID = credentials('AWS_ACCESS_KEY_ID')
AWS_SECRET_ACCESS_KEY = credentials('AWS_SECRET_ACCESS_KEY')
TF_VAR_access_key = credentials('AWS_ACCESS_KEY_ID')
TF_VAR_secret_key = credentials('AWS_SECRET_ACCESS_KEY')
}
stages {
stage('Setup') {
steps {
script {
currentBuild.displayName = "#" + env.BUILD_NUMBER + " " + params.action + " eks-" + params.cluster
plan = params.cluster + '.plan'
}
}
}
stage('TF Plan') {
when {
expression { params.action == 'create' }
}
steps {
dir('infra/terraform/eks'){
script {
withCredentials([[$class: 'AmazonWebServicesCredentialsBinding',
credentialsId: awsCredentialsId,
accessKeyVariable: 'AWS_ACCESS_KEY_ID',
secretKeyVariable: 'AWS_SECRET_ACCESS_KEY']]) {
// Format cidrs into a list array
def ips = '["' + params.api_ingress_cidrs.replaceAll(/\s+/,'\",\"') + '"]'
sh """
terraform init
terraform workspace new ${params.cluster} || true
terraform workspace select ${params.cluster}
terraform plan \
-var cluster-name=${params.cluster} \
-var vpc-network=${params.vpc_network} \
-var vpc-subnets=${params.num_subnets} \
-var inst-type=${params.instance_type} \
-var num-workers=${params.num_workers} \
-var 'api-ingress-ips=${ips}' \
-out ${plan}
"""
}
}
}
}
}
stage('TF Apply') {
when {
expression { params.action == 'create' }
}
steps {
script {
input "Create/update Terraform stack eks-${params.cluster} in aws?"
withCredentials([[$class: 'AmazonWebServicesCredentialsBinding',
credentialsId: awsCredentialsId,
accessKeyVariable: 'AWS_ACCESS_KEY_ID',
secretKeyVariable: 'AWS_SECRET_ACCESS_KEY']]) {
sh """
terraform apply -input=false ${plan}
"""
}
}
}
}
stage('TF Destroy') {
when {
expression { params.action == 'destroy' }
}
steps {
script {
input "Destroy Terraform stack eks-${params.cluster} in aws?"
withCredentials([[$class: 'AmazonWebServicesCredentialsBinding',
credentialsId: awsCredentialsId,
accessKeyVariable: 'AWS_ACCESS_KEY_ID',
secretKeyVariable: 'AWS_SECRET_ACCESS_KEY']]) {
sh """
terraform workspace select ${params.cluster}
terraform destroy -auto-approve
"""
}
}
}
}
}
}

Stages based on a list on Jenkinsfile

I have something like this:
def projects = [
"foo",
"bar",
"foobar",
"etc",
]
pipeline {
agent any
stages {
stage('lint') {
parallel {
stage('lint: foo') {
steps {
sh "lint foo"
}
}
stage('lint: bar') {
steps {
sh "lint bar"
}
}
stage('lint: foobar') {
steps {
sh "lint foobar"
}
}
}
}
}
}
I didn't really want to repeat myself there, is there a way I can "generate" the stage code for each project?
I was able to do something similar, without the parallel part with this:
pipeline {
agent any
stages {
stage('initialize') {
steps {
script {
for (int i = 0; i < projects.size(); i++) {
stage("lint: ${projects[i]}") {
sh "lint ${project}"
}
}
}
}
}
}
}
Any help appreciated.
For scripted pipeline you may use something like:
def projects = [
"foo",
"bar",
"foobar",
"etc",
]
Map branches = [:]
projects.each { String project ->
branches[project] = {
stage(project) {
node {
// I assume we need to checkout something, e.g. using git or checkout step...
// git ...
sh "lint ${project}"
}
}
}
}
parallel branches

Resources