I'm trying to run the sed command in Jenkins groovy scripted pipeline file.
Parsing variable as below
def SERVICE = args.service
def RESOURCE = "Services"
regionSuffix = (action == 'failover') ? 'us-east-2' : 'us-east-1'
environment taking as an argument
The below sed command is working in the Ubuntu terminal.
sed "\|CFT_ENV_FILE|s|$|$RESOURCE/$SERVICE/$environment-$regionSuffix.yml|" docker-compose.yml > docker-compose-$SERVICE.yml
but when I apply this command via Groovy file it gives me an error:
sh '''sed "\\|CFT_ENV_FILE|s|$|"${RESOURCE}"/"${SERVICE}"/"${args.environment}"-"${regionSuffix}".yml|" docker-compose.yml > docker-compose-"${SERVICE}".yml'''
jenkins error
[2022-08-03T11:54:48.642Z] /tmp/jenkins-ac851b81/workspace/Infrastructure/failover/region-failover-test-job#tmp/durable-98781677/script.sh: line 1:
"\|CFT_ENV_FILE|s|$|"${RESOURCE}"/"${SERVICE}"/"${args.environment}"-"${regionSuffix}".yml|": bad substitution
script returned exit code 1
Your variables are in Jenkins/Groovy scope, not in shell scope.
Therefore, you need to substitute their values before passing them to shell:
Use " (double quotes) instead of ' (single quotes):
sh """sed '\\|CFT_ENV_FILE|s|$|${RESOURCE}/${SERVICE}/${args.environment}-${regionSuffix}.yml|' docker-compose.yml > docker-compose-${SERVICE}.yml"""
Please notice that you don't need quotes every time you use a variable (e.g "${RESOURCE}")
Also - consider that you probably need to escape dollar sign (when not used to groovy vars) else based on the command logic.
You can also use just one quotes sign instead of triple, the triple are used for multi-lined string
Related
I have to introduce some templating over text configuration files (yaml, xml, json) that already contain bash-like syntax variables. I need to preserve existing bash-like variables untouched but substitute my ones. List is dynamic, variables should come from environment. Something like simple processor taking "$${MY_VAR}}" pattern but ignoring $MY_VAR. Preferably pure Bash or as small number of tooling required as possible.
Pattern could be $$(VAR) or anything that can be easily separated from ${VAR} and $VAR. The key limitation - it is intended for a docker container startup procedure injecting environment variables into provided service configuration templates and this way building this configuration. So something like Java or even Perl processing is not an option.
Does anybody have a simple approach?
I was using the following bash processing for such variable substitution where original files had no variables. But now I need something one step smarter.
# process input file ($1) placing output into ($2) with shell variables substitution.
process_file() {
set -e
eval "cat <<EOF
$(<$1)
EOF
" | cat > $2
}
Obvious clean solution that is too complex for Docker file because of number of packages needed:
perl -p -i -e 's/\$\{\{([^}]+)\}\}/defined $ENV{$1} ? $ENV{$1} : $&/eg' < test.json
This filters out ${{VAR}}, even better - only set ones.
I'm using a bash script to make changes to an SQL database. One of the values i'm updating uses dollar signs. The current value being something like "$$$$$" and i need to change it to "$$$$$$$$$$". However, a a $ in a bash script is used for variables.
How can i allow this small section of my bash script to used a $ as a normal character?
function prep() {
DATE_STAMP=$(date +%m%d%Y)
log "Changing mask to 10 characters"
log "$(/opt/CPU/bin/connx-query -q "update TYPE set TYPE.MASK = '$$$$$$$$$$'")"
}
As it stands right now, its just replacing each dollar sign with some random number found earlier in my script.
Bash provides different types of quoting, each with different rules about substitution (single quote ', double quote ", here document/string <<<"string" and and $'.
The double quote (used in the log ... update) will enable variable substitution, replacing each pair of $$ with the current shell PID (looks like random number).
Few options:
Consider quoting each '$' to prevent expansion
log "$(/opt/CPU/bin/connx-query -q "update TYPE set TYPE.MASK = '\$\$\$\$\$\$\$\$\$\$'")"
Over thought my own question. I can just escape the $. '\$\$\$\$\$\$\$\$\$\$'
For the life of me, I can't figure out how to declare and use new variables inside a shell block in a groovy script.
For example, this shell block -
sh """
export earlist='abc.ear,def.ear'
echo $earlist;
"""
throws an error saying
No such property: earlist for class: GroovyUserScript
If I add a def earlist before the sh, then it throws error saying -
No signature of method: GroovyUserScript.sh() is applicable for
argument types: (org.codehaus.groovy.runtime.GStringImpl) values: [
export earlist='abc.ear,def.ear' echo ;
Can someone please help me with how to declare and then use variable inside a shell block, in a groovy script?
After consulting with senior experts at my workplace, I found the solution I was looking for.
The problem with this code -
sh """
export earlist='abc.ear,def.ear'
echo $earlist;
"""
is that when I say $earlist, the compiler looks for a groovy variable named earlist and doesn't find it. Since earlist there is a shell variable, I need to escape the $. So, the correct code is -
sh """
earlist='abc.ear,def.ear'
echo \$earlist;
"""
Bonus TIL - if I access a groovy variable inside a shell block, the access is Read-Only. I can't edit the value of the groovy variable, even temporarily within the shell block. If I do want to do that, I can assign the groovy variable to a shell variable, manipulate the shell variable value, save the modified value in a file and when the shell block ends, read the file into the original groovy variable.
Use a triple single quoted string instead, which doesn't interpolate variables:
sh '''
export earlist='abc.ear,def.ear'
echo $earlist;
'''
See here the documentation on triple single quoted strings: http://groovy-lang.org/syntax.html#_triple_single_quoted_string
Here a overview of the available string types in Groovy: http://groovy-lang.org/syntax.html#_string_summary_table
I'm using the CI Lint tester to try and figure out how to store an expected JSON result, which I later compare to a curl response. Neither of these work:
Attempt 1
---
image: ruby:2.1
script:
- EXPECT_SERVER_OUTPUT='{"message": "Hello World"}'
Fails with:
did not find expected key while parsing a block mapping at line 4 column 5
Attempt 2
---
image: ruby:2.1
script:
- EXPECT_SERVER_OUTPUT="{\"message\": \"Hello World\"}"
Fails with:
jobs:script config should be a hash
I've tried using various combinations of echo as well, without a working solution.
You could use literal block scalar1 style notation and put the variable definition and subsequent script lines on separate lines2 without worrying about quoting:
myjob:
script:
- |
EXPECT_SERVER_OUTPUT='{"message": "Hello World"}'
or you can escape the nested double quotes:
myjob:
script:
- "EXPECT_SERVER_OUTPUT='{\"message\": \"Hello World\"}'"
but you may also want to just use variables like:
myjob:
variables:
EXPECT_SERVER_OUTPUT: '{"message": "Hello World"}'
script:
- dothething.sh
Note: variables are by default expanded inside variable definitions so take care with any $ characters inside the variable value (they must be written as $$ to be literal). This feature can also be turned off.
1See this answer for an explanation of this and related notation
2See this section of the GitLab docs for more info on multi-line commands
I made it work like this:
script: |
"EXPECT_SERVER_OUTPUT='{\"message\": \"Hello World\"}'"
echo $EXPECT_SERVER_OUTPUT
I am trying to run a RTC 4.x command to add components to a workspace. The list of components have spaces in the names so they need to be surrounded by quotes. I am storing this list in a simple string variable:
COMPONENTS="\"TestComp\" \"Common Component\""
When I just echo out COMPONENTS it displays correctly, but when I use it in a scm command odd things happen to the quotes. I am running this in Jenkins so I can get some additional output, but the same thing happens when I run it on the command line so this is not a Jenkins issue.
From the console log:
+ COMPONENTS='"TestComp" "Common Component"'
+ echo '"TestComp"' '"Common' 'Component"'
"TestComp" "Common Component"
The command is trying to run the following:
+ scm workspace add-components TEST_Workspace -s Test_Stream '"TestComp"' '"Common' 'Component"'
Which produces:
Problem running 'workspace add-components':
Unmatched component ""Common".
Typically, you need to use an array to store items that may themselves contain whitespace:
components=("TestComp" "Common Component")
scm workspace add-components TEST_Workspace -s Test_Stream "${components[#]}"
Quoting an array expansion indexed with # produces a sequence of words, one per element of the array, rather than a single word.