Jenkins : Loop over a JSON List inside a shell script - shell

{
"TEST_SCRIPTS":["test_1.py","test_2.py"],
"TEST_SCRIPTS1":"test_1.py;test_2.py"
}
This json file, I load in my Jenkins pipeline using :
def load_config(){
def config = readJSON file "./test.json"
return config
}
Now, I need a loop in shell script which can execute each python files defined in TEST_SCRIPTS & TEST_SCRIPTS1.
stage('Test') {
steps {
script{
config = load_config()
sh """
conda env create -n test_env_py37 -f conda.yaml
conda activate test_env_py37
// Below loop is not working. This env is huge, and mendatory for below code to run
for test_script in ${config.TEST_SCRIPTS};
do
python "\$test_script"
done
for test_script in ${config.TEST_SCRIPTS1};
do
python "\$test_script"
done
"""
}
}
}

You can use a groovy approach instead of a shell one, and do all the parsing logic using groovy functionality.
Something like:
stage('Test') {
steps {
script {
def config = readJSON file "./test.json"
def testScripts = config.TEST_SCRIPTS.collect { "python \"$it\""}.join("\n")
def testScripts1 = config.TEST_SCRIPTS1.split(';').collect { "python \"$it\""}.join("\n")
sh """
conda env create -n test_env_py37 -f conda.yaml
conda activate test_env_py37
${testScripts}
${testScripts1}
"""
}
}
}

Related

Access string variable from bash in jenkinsfile groovy script

I'm building several android apps in a docker image using gradle and a bash script. The script is triggered by jenkins, which runs the docker image.
In the bash script I gather information about the successes of the builds. I want to pass that information to the groovy script of the jenkinsfile.
I tried to create a txt file in the docker container, but the groovy script in the jenkinsfile can not find that file.
This is the groovy script of my jenkinsfile:
script {
try {
sh script:'''
#!/bin/bash
./jenkins.sh
'''
} catch(e){
currentBuild.result = "FAILURE"
} finally {
String buildResults = null
try {
def pathToBuildResults="[...]/buildResults.txt"
buildResults = readFile "${pathToBuildResults}"
} catch(e) {
buildResults = "error receiving build results. Error: " + e.toString()
}
}
}
In my jenkins.sh bash script I do the following:
[...]
buildResults+=" $appName: Build Failed!" //this is done for several apps
echo "$buildResults" | cat > $pathToBuildResults //this works I checked, if the file is created
[...]
The file is created, but groovy cannot find it. I think the reason is, that the jenkins script does not run inside the docker container.
How can I access the string buildResults of the bash script in my groovy jenkins script?
One option that you have in order to avoid the need to read the results file is to modify your jenkins.sh script to print the results to the output instead of writing them to a file and then use the sh step to capture that output and use it instead of the file.
Something like:
script {
try {
String buildResults = sh returnStdout: true, script:'''
#!/bin/bash
./jenkins.sh
'''
// You now have the output of jenkins.sh inside the buildResults parameter
} catch(e){
currentBuild.result = "FAILURE"
}
}
This way you are avoiding the need to handle the output files and directly get the results you need, which you can then parse and use however you need.

Execute shell script from Jenkins Pipeline

I am trying to execute shell script from my jenkins pipeline. I have provided absolute and relative path in the shell command and still I am facing No such file or directory error while building the pipeline.
This is simple script but yet not working.
Try 1:
stage ( 'Executing shell script' ) {
steps {
sh '/home/patching/shell_script.sh'
}
}
Try 2:
stage ( 'Executing shell script' ) {
steps {
sh './shell_script.sh'
}
}
Try 3:
stage ( 'Executing shell script' ) {
steps {
dir ('/home/patching/shell_script.sh){
sh './shell_script.sh'
}
}
I really don't know what is really wrong with the script. Could some one help me on this?
I got the issue why it wasn't able to find the file that I wanted to run. It was running on different slave.
Can you try this
pipeline {
agent any
stages {
stage('Hello World') {
steps {
script {
sh '''
/home/ubuntu/First.sh
'''
}
}
}
}
}

How to assign a shell command output as value to envrionment variable?

Below is my pipeline snippet and I am trying to assign RSTATE variable a value at run time. This value is basically stored in a text file but we need to grep and cut it. So a shell command output should be its value.
pipeline
{
agent any
environment
{
RSTATE = 'R4C'
ISO_REV = 'TA'
BuildSource = '18'
}
stages
{
stage('get Rstate')
{
echo env.RSTATE
}
}
}
I am trying to assign RSTATE value like:
RSTATE = sh ( script: 'grep RSTATE /proj/MM/scm/com/iv_build/mm18_1/rstate/next_rstate.txt
|cut -d "=" -f2', returnStdout: true).trim()
But this is not working.
I also tried to run a shell script but that also not works. Only hard coded value is working. Please suggest.
I tested and worksm you need to validate if your script return the value you want.
pipeline
{
agent any
environment
{
RSTATE = 'R4C'
ISO_REV = 'TA'
BuildSource = '18'
}
stages
{
stage('get Rstate')
{
steps {
script {
def RSTATE2 = sh ( script: 'echo \${RSTATE}', returnStdout: true).trim()
echo env.RSTATE
echo RSTATE2
}
}
}
}
}

Env variable value got reset to original even after assigning the pom version number in jenkins script

I have a scenario where i have to read the maven pom versions for different components and assign the version to docker image(TAG). But after i read the pom, assigned it to some global variable it will reset to original value in groovy jenkins script. Below is the sample. HMAP_VERSION value will 1.2.1 but when it is used in the line: sh "docker login -u ${ART_USERNAME} -p ${ART_PASSWORD} test.com" the value will be UNINITIALISED.
Can somebody tell me what might have gone wrong? This will work with single maven file which is read in env block as below:
environment {
CLOADER_VERSION = readMavenPom().getVersion()
}
Below is the sample of what im tring to do.
#! groovy
environment {
HMAP_VERSION = "UNINITIALISED"
CLOADER_VERSION = "UNINITIALISED"
}
stages {
stage('Build Cloader') {
steps {
checkout([$class: 'GitSCM' "rest is removed")
dir('isa-casloader') {
script {
CLOADER_VERSION = readMavenPom().getVersion()
}
container('build') {
sh '/opt/apache-maven/bin/mvn -s settings.xml -B clean install -DskipTests=true'
}
}
}
}
stage ('Build Casloader Docker Image') {
steps {
dir('isa-casloader') {
container('tools') {
echo("CLOADER_VERSION=${CLOADER_VERSION}")
withCredentials() {
sh "docker login -u ${ART_USERNAME} -p ${ART_PASSWORD} testing.com"
sh 'docker build -t testing.com:${CLOADER_VERSION} .'
sh 'docker push testing.com:${CLOADER_VERSION}'
}
}
}
}
}
stage ('Build Heat Map Docker Image') {
steps {
checkout([$class: 'GitSCM', "rest is commented"])
dir('apps') {
container('tools') {
script {
def pom = readMavenPom file: 'pom-docker.xml'
HMAP_VERSION = pom.version
}
echo("HMAP_VERSION=${HMAP_VERSION}")
withCredentials() {
sh "docker login -u ${ART_USERNAME} -p ${ART_PASSWORD} test.com"
sh 'docker build -t test.com:${HMAP_VERSION} .'
sh 'docker push test.com:${HMAP_VERSION}'
}}}}}}}
By my read of your code, you're mixing environment variables with variables within the Groovy context.
These lines create environment variables, which are accessible in the shell as $HMAP_VERSION and $CLOADER_VERSION:
environment {
HMAP_VERSION = "UNINITIALISED"
CLOADER_VERSION = "UNINITIALISED"
}
However, you're populating a Groovy variable here:
script {
CLOADER_VERSION = readMavenPom().getVersion()
}
To instead populate the environment variable, you'd want to use env.CLOADER_VERSION instead.
This changes what context the variables are evaluated in when you're calling out to shell using the sh directive:
1-> sh "docker login -u ${ART_USERNAME} -p ${ART_PASSWORD} testing.com"
2-> sh 'docker build -t testing.com:${CLOADER_VERSION} .'
3-> sh 'docker push testing.com:${CLOADER_VERSION}'
In line number 1 above, the command is quoted using a double quotes (") which means that the variables ART_USERNAME and ART_PASSWORD are evaluating in the context of the Groovy script.
However, in lines 2 and 3 the commands are quoted using a single quote (') which means that those variables are being evaluated by the shell (likely /bin/sh) and therefore using the values from the environment.
The easiest fix would be to ensure that values you want exposed in the shell are always accessed using the env. prefix in the Groovy context:
// set environment for CLOADER_VERSION
env.CLOADER_VERSION = readMavenPom().getVersion()
// print value of environment variable CLOADER_VERSION
echo("CLOADER_VERSION=${env.CLOADER_VERSION}")
// set environment for HMAP_VERSION
env.HMAP_VERSION = pom.version
// print value of environment variable HMAP_VERSION
echo("HMAP_VERSION=${env.HMAP_VERSION}")
Cheers.
Thanks for the response. My issue got resolved. In docker context as shown below,
withCredentials() {
sh "docker login -u ${ART_USERNAME} -p ${ART_PASSWORD} testing.com"
sh 'docker build -t testing.com:${CLOADER_VERSION} .'
sh 'docker push testing.com:${CLOADER_VERSION}'
}
Login command is proper which is inside double quotes, but the next statements were in single quotes. So variables latest value was not getting resolved. When i change the statements to be inside double quotes, it worked!!
Below is the proper command:
withCredentials() {
sh "docker login -u ${ART_USERNAME} -p ${ART_PASSWORD} testing.com"
sh "docker build -t testing.com:${CLOADER_VERSION} ."
sh "docker push testing.com:${CLOADER_VERSION}"
}
Thanks you.

Using parameters within a shell command in Jenkinsfile for Jenkins pipeline

I want to use defined parameters in Jenkinsfile in several shell commands, but I get an exception. In my example I want to execute a simple docker command. The parameter defines the path to docker executable.
This is my very short Jenkinsfile:
pipeline {
agent any
parameters {
string(defaultValue: '/Applications/Docker.app/Contents/Resources/bin/docker', description: '', name: 'docker')
}
stages {
stage('Test') {
steps {
sh 'sudo ${params.docker} ps -a'
}
}
}
}
And I get the following exception:
[e2e-web-tests_master-U4G4QJHPUACAEACYSISPVBCMQBR2LS5EZRVEKG47I2XHRI54NCCQ] Running shell script
/Users/Shared/Jenkins/Home/workspace/e2e-web-tests_master-U4G4QJHPUACAEACYSISPVBCMQBR2LS5EZRVEKG47I2XHRI54NCCQ#tmp/durable-e394f175/script.sh: line 2: ${params.docker}: bad substitution
When I change the Jenkinsfile without using the paramter inside the shell command it passes successfully:
pipeline {
agent any
parameters {
string(defaultValue: '/Applications/Docker.app/Contents/Resources/bin/docker', description: '', name: 'docker')
}
stages {
stage('Test') {
steps {
sh 'sudo /Applications/Docker.app/Contents/Resources/bin/docker ps -a'
}
}
}
}
So, how can I use parameters inside a shell command in Jenkinsfile? I tried string and text as parameter types.
The issue you have is that single quotes are a standard java String.
Double quotes are a templatable String, which will either return a GString if it is templated, or else a standard Java String.
So it you use double quotes:
stages {
stage('Test') {
steps {
sh "sudo ${params.docker} ps -a"
}
}
}
then params.docker will replace the ${params.docker} inside the 'sh' script in the pipeline.
If you want to put " inside the "sudo ${params.docker} ps -a" it doesn't work like bash (which is confusing) you use java style escaping, so "sudo \"${params.docker}\" ps -a"

Resources