How to re-use pre process jenkins/groovy in each test - shell

Im using the following code to run our voter , currently I’ve one target which is called Run Tests
Which use exactly the same steps as the last (lint) , currently I duplicate it which I think is not a good solution ,
Is there is nice way to avoid this duplication and done it only once as per-requisite process ?
I need all the steps until the cd to the project
The only difference is one target I run
go test ...
and the second
go lint
All steps before are equal
#!/usr/bin/env groovy
try {
parallel(
'Run Tests': {
node {
//————————Here we start
checkout scm
def dockerImage = 'docker.company:50001/crt/deg:0.1.3-09’
setupPipelineEnvironment script: this,
measureDuration(script: this, measurementName: 'build') {
executeDocker(dockerImage: dockerImage, dockerWorkspace: '/go/src') {
sh """
mkdir -p /go/src/github.com/ftr/myGoProj
cp -R $WORKSPACE/* /go/src/github.com/ftr/MyGoProj
cd /go/src/github.com/ftr/MyGoProj
//————————Here we finish and TEST
go test -v ./...
"""
}
}
}
},
‘Lint’: {
node {
//————————Here we start
checkout scm
def dockerImage = 'docker.company:50001/crt/deg:0.1.3-09’
setupPipelineEnvironment script: this,
measureDuration(script: this, measurementName: 'build') {
executeDocker(dockerImage: dockerImage, dockerWorkspace: '/go/src') {
sh """
mkdir -p /go/src/github.com/ftr/myGoProj
cp -R $WORKSPACE/* /go/src/github.com/ftr/MyGoProj
cd /go/src/github.com/ftr/MyGoProj
//————————Here we finish and LINT
go lint
"""
}
}
)
}
}
}

You can use function and pass Go arguments:
try {
parallel(
'Run Tests': {
node {
checkout scm
runTestsInDocker('test -v ./...')
}
},
'Lint': {
node {
checkout scm
runTestsInDocker('lint')
}
}
)
}
def runTestsInDocker(goArgs) {
def dockerImage = 'docker.company:50001/crt/deg:0.1.3-09'
setupPipelineEnvironment script: this,
measureDuration(script: this, measurementName: 'build') {
executeDocker(dockerImage: dockerImage, dockerWorkspace: '/go/src') {
sh """
mkdir -p /go/src/github.com/ftr/myGoProj
cp -R $WORKSPACE/* /go/src/github.com/ftr/MyGoProj
cd /go/src/github.com/ftr/MyGoProj
go ${goArgs}
"""
}
}
}
Update
If some actions can be separated out of runTestsInDocker they probably should be.
For example setupPipelineEnvironment step. I don't know exact logic but maybe it can be run once before running test.
node {
stage('setup') {
setupPipelineEnvironment script: this
}
stage ('Tests') {
parallel(
'Run Tests': {
node {
checkout scm
runTestsInDocker('test -v ./...')
}
},
'Lint': {
node {
checkout scm
runTestsInDocker('lint')
}
}
)
}
}
def runTestsInDocker(goArgs) {
def dockerImage = 'docker.company:50001/crt/deg:0.1.3-09'
measureDuration(script: this, measurementName: 'build') {
executeDocker(dockerImage: dockerImage, dockerWorkspace: '/go/src') {
sh """
mkdir -p /go/src/github.com/ftr/myGoProj
cp -R $WORKSPACE/* /go/src/github.com/ftr/MyGoProj
cd /go/src/github.com/ftr/MyGoProj
go ${goArgs}
"""
}
}
}
Note
If you are running parallel on remote agents you must remember that setup performed on master may be not aviailable on remote slave.

Related

report folder does not exist error with htmlpublisher

I am trying to write a jenkins pipeline script for one of my playwright test. Below is one simple code which i have done so far.
pipeline {
agent any
stages {
stage('Run Playwright Test') {
steps {
runTest()
}
}
stage('Publish Report'){
steps {
script {
sh 'ls -lrta'
//print REPORT_FILES
}
publishHTML([
allowMissing: false,
alwaysLinkToLastBuild: true,
keepAll: true,
//reportDir: '.',
reportDir : "./TestReport2",
reportFiles: 'index.html',
reportName: "Html Reports",
reportTitles: 'Report title'
])
}
}
}
}
def runTest() {
node('MYNODE') {
docker.image('image details').inside('--user root'){
git branch: 'mybranchName',credentialsId: 'ID, url: 'url'
catchError() {
sh """
cd WebTests
npm install
npx playwright test --project=CHROME_TEST --grep #hello
"""
}
sh "cp -R WebTests/TestReport TestReport2"
sh 'cd TestReport2; ls -lrta'
}
When I use the above code, the test executed successfully however i am seeing an error while trying to publish the report.
Below is the error :
Specified HTML directory '/bld/workspace//TestReport2' does not exist.
observation: when i put a ls -ltr after the runTest code i could not see the TestReport2 folder even if it was copied successfully.
Another thing i tried is when i put the code to publish the HTML as part of the runTest() it worked fine and i am able to see the reports generated. Something is going on with the TestReport2 folder when the block of code for runTest() is completed.
Does anyone have an eye on what is the root cause. Any suggestion will be appreciated

Run a set of linux commands using Jenkinsfile in Jenkins

I had to create a jenkins job to automate certain tasks that will perform certain operations like Updating the public site, Changing public version to latest public release, Updating Software on public site and Restarting Server these include certain operations such as copy files to a tmp folder, log in to a an on-prem server, go to the folder and unzip the file etc.
I have created the jenkinsfile as follows:
pipeline {
options {
skipDefaultCheckout()
timestamps()
}
parameters {
string(name: 'filename', defaultValue: 'abc', description: 'Enter the file name that needs to be copied')
string(database: 'database', defaultValue: 'abc', description: 'Enter the database that needs to be created')
choice(name: 'Run', choices: '', description: 'Data migration')
}
agent {
node { label 'aws && build && linux && ubuntu' }
}
triggers {
pollSCM('H/5 * * * *')
}
stages {
stage('Clean & Clone') {
steps {
cleanWs()
checkout scm
}
}
stage('Updating the public site'){
steps{
sh "scp ./${filename}.zip <user>#<server name>:/tmp"
sh "ssh <user>#<server name>"
sh "cp ./tmp/${filename}.zip ./projects/xyz/xyz-site/"
sh "cd ./projects/xyz/xyz-site/ "
sh "unzip ./${filename}.zip"
sh "cp -R ./${filename}/* ./"
}
stage('Changing public version to latest public release') {
steps {
sh "scp ./${filename}.sql.gz <user>#<server name>:/tmp"
sh "ssh <user>#<server name>"
sh "mysql -u root -p<PASSWORD>"
sh "show databases;"
sh "create database ${params.database};"
sh "GRANT ALL PRIVILEGES ON <newdb>.* TO 'ixxyz'#'localhost' WITH GRANT OPTION;"
sh "exit;"
sh "zcat tmp/${filename}.sql.gz | mysql -u root -p<PASSWORD> <newdb>"
sh "db.default.url="jdbc:mysql://localhost:3306/<newdb>""
sh "ps aux|grep monitor.sh|awk '{print "kill "$2}' |bash"
}
}
stage('Updating Software on public site') {
steps {
sh "scp <user>#<server>:/tmp/abc<version>_empty_h2.zip"
sh "ssh <user>#<server name>"
sh "su <user>"
sh "mv tmp/<version>_empty_h2.zip ./xyz/projects/xyz"
sh "cd xyz/projects/xyz"
sh "cp latest/conf/local.conf <version>_empty_h2/conf/"
}
}
stage('Restarting Server') {
steps {
sh "rm latest/RUNNING_PID"
sh "bash reload.sh"
sh "nohup bash monitor.sh &"
}
}
}
}
Is there a way I can dynamically obtain the zip filename in the root folder? I used ${filename}.zip , but it doesn't seem to work.
Also, is there a better way to perform these operations using jenkins? Any help is much appreciated.
You could write all your steps in one shell script for each stage and execute under one stage.
Regarding filename.zipeither you can take this as a parameter and pass this value to your stages. OR You can also use find command as a shell command or shell script to find .zip files in a current directory. find <dir> -iname \*.zip find . -iname \*.zip .
Example:
pipeline {
options {
skipDefaultCheckout()
timestamps()
}
parameters {
string(name: 'filename', defaultValue: 'abc', description: 'Enter the file name that needs to be copied')
choice(name: 'Run', choices: '', description: 'Data migration')
}
stage('Updating the public site'){
steps{
sh "scp ./${params.filename}.zip <user>#<server name>:/tmp"
...
}
}
}
For executing script at a certain location based on your question , you could use dir with path where your scripts are placed.
OR you can also give the path directly sh label: 'execute script', script:"C:\\Data\\build.sh"
stage('Your stage name'){
steps{
script {
// Give path where your scripts are placed
dir ("C:\\Data") {
sh label: 'execute script', script:"build.sh <Your Arguments> "
...
}
}
}
}

Getting a jenkinsfile stage to run on master branch and tag

I tried to get a stage to run when user push to master branch or push a tag to origin. This is the when condition that i set in the stage:
when {
expression {
branch 'master' || tag 'v*'
}
}
It throws an error that No such field found: field org.jenkinsci.plugins.structs.describable.UninstantiatedDescribable v*. Is there another way to do a conditional expression in such a way that it works?
I have a Jenkinsfile to automate the process to build a docker image and push it to the docker registry. In this example the docker image is build when someone push a tag in master branch:
stage('Master Docker Prepare') {
when {
expression {
return (env.BRANCH_NAME=="master" && env.GIT_TAG!=null && env.GIT_TAG != '')
}
}
steps {
echo 'Deploying'
script {
dockerImage = docker.build DOCKER_REGISTRY_PUT+"/"+DOCKER_IMAGE + ":" + env.GIT_TAG
}
script {
docker.withRegistry( 'http://'+DOCKER_REGISTRY_PUT, DOCKER_REGISTRY_CREDENTIAL ) {
dockerImage.push()
}
}
}
}
I think you can make something similar.
Use anyOf inside when as shown below
when {
anyOf {
branch 'master'
expression { tag =~ 'v*' }
}
}

Jenkins declarative pipeline get environment variable at post stage

I am getting runtime value in build stage stage which I stored in an environment variable . I saved that to env.cfg file under WORKSPACE .
Now I am trying to get that value in post pipeline step to be used in email communication. I tried load method but it did not work
Any help ?
post {
always {
echo $SNAPSHOT / /this always comes null
}
}
This is the way you can access an environment variable across the pipeline
pipeline {
agent any;
environment {
MESSAGE="Hello World"
}
stages {
stage('one') {
steps {
echo "${env.MESSAGE}"
sh "echo $MESSAGE"
script {
print env.MESSAGE
}
}
}
}
post {
success {
echo "${env.MESSAGE}"
script {
print env.MESSAGE
}
}
failure {
echo "${env.MESSAGE}"
script {
print env.MESSAGE
}
}
}
}
but as per your scenario let say I have a file called .env with the content below in the current Jenkins job WORKSPACE and I want to read and make this env variable in the pipeline.
.env
export SNAPSHOT=1.0.0
export MESSAGE='Hello World'
export MESSAGE_FROM_ENV_FILE='Hello From .env file'
your pipeline should look like
scripted pipeline
node {
stage('one') {
sh """
source $WORKSPACE/.env
echo \$SNAPSHOT
echo \$MESSAGE
echo \$MESSAGE_FROM_ENV_FILE
"""
}
}
declarative pipeline
pipeline {
agent any;
stages {
stage('build') {
steps {
sh """
source $WORKSPACE/.env
echo \$SNAPSHOT
echo \$MESSAGE
echo \$MESSAGE_FROM_ENV_FILE
"""
}
}
}
post {
success {
sh """
source $WORKSPACE/.env
echo \$SNAPSHOT
echo \$MESSAGE
echo \$MESSAGE_FROM_ENV_FILE
"""
}
}
}
You need a global variable:
SNAPSHOT = ""
println "SNAPSHOT is ${SNAPSHOT}"
pipeline {
agent any
stages {
stage('Build') {
steps {
script {
println "SNAPSHOT is ${SNAPSHOT}"
SNAPSHOT = "Modified"
println "SNAPSHOT is now ${SNAPSHOT}"
}
}
}
}
post {
always {
echo "${SNAPSHOT}"
}
}
}

Jenkins/ MacOS - dial unix /var/run/docker.sock: connect:permission denied

i am new to using jenkins and docker. Currently I ran into an error where my jenkinsfile doesnt have permission to docker.sock. Is there a way to fix this? Dried out of ideas
things i've tried:
-sudo usermod -aG docker $USER //usermod not found
-sudo setfacl --modify user:******:rw /var/run/docker.sock //setfacl not found
-chmod 777 /var/run/docker.sock //still receiving this error after reboot
-chown -R jenkins:jenkins /var/run/docker.sock //changing ownership of '/var/run/docker.sock': Operation not permitted
error image:
def gv
pipeline {
agent any
environment {
CI = 'true'
VERSION = "$BUILD_NUMBER"
PROJECT = "foodcore"
IMAGE = "$PROJECT:$VERSION"
}
tools {
nodejs "node"
'org.jenkinsci.plugins.docker.commons.tools.DockerTool' 'docker'
}
parameters {
choice(name: 'VERSION', choices: ['1.1.0', '1.2.0', '1.3.0'], description: '')
booleanParam(name: 'executeTests', defaultValue: true, description: '')
}
stages {
stage("init") {
steps {
script {
gv = load "script.groovy"
CODE_CHANGES = gv.getGitChanges()
}
}
}
stage("build frontend") {
steps {
dir("client") {
sh 'npm install'
echo 'building client'
}
}
}
stage("build backend") {
steps {
dir("server") {
sh 'npm install'
echo 'building server...'
}
}
}
stage("build docker image") {
steps {
sh 'docker build -t $IMAGE .'
}
}
// stage("deploy") {
// steps {
// script {
// docker.withRegistry(ECURL, ECRCRED) {
// docker.image(IMAGE).push()
// }
// }
// }
// }
}
// post {
// always {
// sh "docker rmi $IMAGE | true"
// }
// }
}
docker.sock permissions will be lost if you restart system or docker service.
To make it persistence setup a cron to change ownership after each reboot
#reboot chmod 777 /var/run/docker.sock
and When you restart the docker, make sure to run the below command
chmod 777 /var/run/docker.sock
Or you can put a cron for it also, which will execute in each every 5 minutes.

Resources