I have test$001 as a value in Jenkins secret text credentials. Later in pipeline script i'm accessing that value and writing it to yaml file like mentioned below, which is used as K8S configmap.
Problem is with the Dollar sign in the value.
environment {
TEST_CRED=credentials('TEST_CRED')
}
script.sh
cat << EOF > test.yaml
...
data:
TEST: ${TEST_CRED}
EOF
Expected: test$001
Printed: test$$001 (Note extra dollar sign being inserted automatically)
I tried all possibilities to escape this dollar sign, nothing worked.
TEST_01: '${TEST_CRED}'
TEST_02: ${TEST_CRED}
TEST_03: '$${TEST_CRED}'
TEST_04: $${TEST_CRED}
TEST_05: "$${TEST_CRED}"
TEST_08: $TEST_CRED
When storing value in Jenkins secret text credentials, escape the dollar sign. So, test$001 should actually be stored as test\$001.
Following works for me:
pipeline {
agent any
environment {
MYTEST_CRED=credentials('TEST_CRED')
}
stages {
stage('Special Char') {
steps {
sh """
cat << EOF > test.yaml
Name: test-config
Namespace: default
data:
TEST: ${MYTEST_CRED}
EOF
"""
}
}
}
}
Output:
This is an example when I'm passing a not escaped string to the Jenkins job via parameters. And things are not going my way.
// Original and expected value. Works fine with pure groovy
echo env.SECRET_VALUE
test#U$3r
// But this variable in shell is getting messed up
// sh("\$ENV") and sh('$ENV') are using value of shell env variale
sh("echo \$SECRET_VALUE")
test#U$$3r
sh('echo $SECRET_VALUE')
test#U$$3r
// sh("$ENV") and sh("${ENV}") are using value of groovy variables passed to the shell
sh("echo $SECRET_VALUE")
test#Ur
sh("echo ${SECRET_VALUE}")
test#Ur
Let's try to fix it
env.ESCAPED_SECRET_VALUE = env.SECRET_VALUE.replaceAll(/(!|"|#|#|\$|%|&|\\/|\(|\)|=|\?)/, /\\$0/)
// groovy variable is becoming a bit broken
echo env.ESCAPED_SECRET_VALUE
test\#U\$3r
// shell env variable is still broken
sh("echo \$ESCAPED_SECRET_VALUE")
test\#U\$$3r
sh('echo $ESCAPED_SECRET_VALUE')
test\#U\$$3r
// But, if we will pass groovy env variable to the shell - it looks good
sh("echo $ESCAPED_SECRET_VALUE")
test#U$3r
sh("echo ${ESCAPED_SECRET_VALUE}")
test#U$3r
If You are using command straight in the sh(script:""), then just pass groovy ESCAPED variable. If You need to invoke shell script file, then try to pass value of this groovy ESCAPED variable as input argument into it
Example:
sh("./my_super_script.sh $ESCAPED_SECRET_VALUE")
# my_super_script.sh
#!/bin/bash
SECRET_VALUE=$1
echo $SECRET_VALUE
I did a setup as per your requirement and got the desired results.
The setup is shown below with the screenshots,
Setup Jenkins secret text credential
Setup Binding in the Jenkins job
Configuring the build to create the test.yaml
Content of test.yaml
$ cat test.yaml
...
data:
TEST: test$001
Related
How to get the output of kubectl describe deployment nginx | grep Image in an environment variable?
My code:
stage('Deployment'){
script {
sh """
export KUBECONFIG=/tmp/kubeconfig
kubectl describe deployment nginx | grep Image"""
}
}
In this situation, you can access the environment variables in the pipeline scope within the env object, and assign values to its members to initialize new environment variables. You can also utilize the optional returnStdout parameter to the sh step method to return the stdout of the method, and therefore assign it to a Groovy variable (because it is within the script block in the pipeline).
script {
env.IMAGE = sh(script: 'export KUBECONFIG=/tmp/kubeconfig && kubectl describe deployment nginx | grep Image', returnStdout: true).trim()
}
Note you would also want to place the KUBECONFIG environment variable within the environment directive at the pipeline scope instead (unless the kubeconfig will be different in different scopes):
pipeline {
environment { KUBECONFIG = '/tmp/kubeconfig' }
}
You can use the syntax:
someVariable = sh(returnStdout: true, script: some_script).trim()
I have a nextflow script that creates a variable from a text file, and I need to pass the value of that variable to a command line order (which is a bioconda package). Those two processes happen inside the "script" part. I have tried to call the variable using the '$' symbol without any results, I think because using that symbol in the script part of a nextflow script is for calling variables defined in the input part.
To make myself clearer, here is a code sample of what I'm trying to achieve:
params.gz_file = '/path/to/file.gz'
params.fa_file = '/path/to/file.fa'
params.output_dir = '/path/to/outdir'
input_file = file(params.gz_file)
fasta_file = file(params.fa_file)
process foo {
//publishDir "${params.output_dir}", mode: 'copy',
input:
path file from input_file
path fasta from fasta_file
output:
file ("*.html")
script:
"""
echo 123 > number.txt
parameter=`cat number.txt`
create_report $file $fasta --flanking $parameter
"""
}
By doig this the error I recieve is:
Error executing process > 'foo'
Caused by:
Unknown variable 'parameter' -- Make sure it is not misspelt and defined somewhere in the script before using it
Is there any way to call the variable parameter inside the script without Nextflow interpreting it as an input file? Thanks in advance!
The documentation re the script block is useful here:
Since Nextflow uses the same Bash syntax for variable substitutions in
strings, you need to manage them carefully depending on if you want to
evaluate a variable in the Nextflow context - or - in the Bash
environment execution.
One solution is to escape your shell (Bash) variables by prefixing them with a back-slash (\) character, like in the following example:
process foo {
script:
"""
echo 123 > number.txt
parameter="\$(cat number.txt)"
echo "\${parameter}"
"""
}
Another solution is to instead use a shell block, where dollar ($) variables are managed by your shell (Bash interpreter), while exclamation mark (!) variables are handled by Nextflow. For example:
process bar {
echo true
input:
val greeting from 'Hello', 'Hola', 'Bonjour'
shell:
'''
echo 123 > number.txt
parameter="$(cat number.txt)"
echo "!{greeting} parameter ${parameter}"
'''
}
declare "parameter" in the top 'params' section.
params.parameter="1234"
(..)
script:
"""
(...)
create_report $file $fasta --flanking ${params.parameter}
(...)
"""
(...)
and call "nextflow run" with "--parameter 87678"
I am trying to execute set of commands from jenkinsfile.
The problem is, when I try to assign the value of stdout to a variable it is not working.
I tried different combinations of double quotes and single quotes, but so far no luck.
Here I executed the script with latest version of jenkinsfile as well as old version. Putting shell commands inside """ """ is not allowing to create new variable and giving error like client_name command does not exist.
String nodeLabel = env.PrimaryNode ? env.PrimaryNode : "slave1"
echo "Running on node [${nodeLabel}]"
node("${nodeLabel}"){
sh "p4 print -q -o config.yml //c/test/gradle/hk/config.yml"
def config = readYaml file: 'devops-config.yml'
def out = sh (script:"client_name=${config.BasicVars.p4_client}; " +
'echo "client name: $client_name"' +
" cmd_output = p4 clients -e $client_name" +
' echo "out variable: $cmd_output"',returnStdout: true)
}
I want to assign the stdout from the command p4 clients -e $client_name to variable cmd_output.
But when I execute the code the error that is thrown is:
NoSuchPropertyException: client_name is not defined at line cmd_output = p4 clients -e $client_name
What am I missing here?
Your problem here is that all the $ are interpreted by jenkins when the string is in double quotes. So the first 2 times there's no problem since the first variable comes from jenkins and the second time it's a single quote string.
The the third variable is in a double quote string, therefore jenkins tries to replace the variable with its value but it can't find it since it's generated only when the shell script is executed.
The solution is to escape the $ in $client_name (or define client_name in an environment block).
I rewrote the block:
String nodeLabel = env.PrimaryNode ? env.PrimaryNode : "slave1"
echo "Running on node [${nodeLabel}]"
node("${nodeLabel}"){
sh "p4 print -q -o config.yml //c/test/gradle/hk/config.yml"
def config = readYaml file: 'devops-config.yml'
def out = sh (script: """
client_name=${config.BasicVars.p4_client}
echo "client name: \$client_name"
cmd_output = p4 clients -e \$client_name
echo "out variable: \$cmd_output"
""", returnStdout: true)
}
I am trying to access an env variable in Jenkins pipeline and want to use it in a Shell Script executing in the same pipeline but a differnt step,
pipeline {
agent any
tools {
maven 'M2'
}
environment {
stable_revision = sh(script: 'curl -H "Authorization: Basic $base64encoded" "https://xyz.sad" | jq -r "name"', returnStdout: true)
}
stages {
stage('Initial-Checks') {
steps {
echo "Stable Revision: ${env.stable_revision}" //displays the value
bat "sh && sh undeploy.sh"
}}
...
}}
This is a sample Shell script, it has many lines, but I have an issue in only accessing the above stable_revision variable,
#!/bin/bash
echo xyz = ${stable_revision} #### this gives the right value xyz = 22
echo xyz2 = ${stable_revision}/d ### here after the value the /d is written in new line
For example, let's say the stable_revision value is 22, then in the SH script echo I am getting the value as,
xyz2 = 22
/d
I want the value to be xyz2 = 22/d
You can use .trim() to strip off a trailing newline.
environment {
stable_revision = sh(script: 'curl -H "Authorization: Basic $base64encoded" "https://xyz.sad" | jq -r "name"', returnStdout: true).trim()
}
returnStdout (optional):
If checked, standard output from the task is returned as the step value as a String, rather than being printed
to the build log. (Standard error, if any, will still be printed to
the log.) You will often want to call .trim() on the result to strip
off a trailing newline.
https://jenkins.io/doc/pipeline/steps/workflow-durable-task-step/#sh-shell-script
If you use bash instead of sh for your commands, you can benefit from Bash's built-in string transformations
Here it trims all trailing characters from the [:space:] class witch includes actual spaces and newlines.
echo "xyz2 = ${stable_revision%%[[:space:]]}/d"
If $stable_revision is always an integer, you can force the shell to use it like an integer with:
echo "xyz2 = $((stable_revision))/d"
If you are sure that $stable_revision contains no space, you can force the shell to trim all spaces by using it like a table element:
sr=($stable_revision); echo "xyz2 = ${sr[0]}/d"
You can also use the automatic trimming of a sub-shell returned value, that would trim any leading, trailing and duplicate spaces in-between:
echo "xyz2 = $(echo ${stable_revision})/d"`
I'm trying to build a Jenkins Pipeline for which a parameter is
optional:
parameters {
string(
name:'foo',
defaultValue:'',
description:'foo is foo'
)
}
My purpose is calling a shell script and providing foo as argument:
stages {
stage('something') {
sh "some-script.sh '${params.foo}'"
}
}
The shell script will do the Right Thing™ if the provided value is the empty
string.
Unfortunately I can't just get an empty string. If the user does not provide
a value for foo, Jenkins will set it to null, and I will get null
(as string) inside my command.
I found this related question but the only answer is not really helpful.
Any suggestion?
OP here realized a wrapper script can be helpful… I ironically called it junkins-cmd and I call it like this:
stages {
stage('something') {
sh "junkins-cmd some-script.sh '${params.foo}'"
}
}
Code:
#!/bin/bash
helpme() {
cat <<EOF
Usage: $0 <command> [parameters to command]
This command is a wrapper for jenkins pipeline. It tries to overcome jenkins
idiotic behaviour when calling programs without polluting the remaining part
of the toolkit.
The given command is executed with the fixed version of the given
parameters. Current fixes:
- 'null' is replaced with ''
EOF
} >&2
trap helpme EXIT
command="${1:?Missing command}"; shift
trap - EXIT
typeset -a params
for p in "$#"; do
# Jenkins pipeline uses 'null' when the parameter is undefined.
[[ "$p" = 'null' ]] && p=''
params+=("$p")
done
exec $command "${params[#]}"
Beware: prams+=("$p") seems not to be portable among shells: hence this ugly script is running #!/bin/bash.