How to use custom variables in gitlab ci/cd? - bash

I'm struggling with gitlab ci/cd variables. I see so many conflicting examples. Anyhow, what I would like to know is how to use variables outside and within scripts.
For example, in a job config, can I assign a variable in the script section with a bash command?
some-job:
variables:
SOME_VAR: ''
script:
- SOME_VAR = $(<file_with_a_line_of_text.txt)
In the above case, I'm not sure if I can do this. But I need to populate a variable with the file contents (i.e. artifact). Also, when do I use '$' in front of the variable? Some examples I see using these formats:
"SOME_VAR" #in quotes, no dollar sign
"${SOME_VAR}" #in quotes, with dollar sign and wrapped with curly braces
${SOME_VAR} #no quotes, with dollar sign and wrapped with curly braces
$SOME_VAR #i.e. without the double quotes or curly braces
SOME_VAR #i.e. without the double quotes, dollar sign, and curly braces
So many variations of usage that I can see in examples but don't really know when to use each style. And I can't find one example online of a custom variable being set in a script using a bash command.

When I'm setting variables in bash, I always do it without the spaces around the =:
VAR1="some string"
VAR2=23
VAR3=true
VAR4=$(cat /path/to/file.txt)
Let's go through these examples one at a time:
You can set a variable as a string by using quotes around the string.
You can set it to an int (probably a float too, but haven't personally used it)
You can set it to a bool
You can set it to the output of a command. The command is inside the command: $(#command).
Now let's use them:
echo $VAR1
# some string
echo "This is my variable $VAR1"
# This is my variable some string
echo "This is my variable ${VAR1}"
# This is my variable some string
echo ${VAR1}
# some string
echo "Error code ${VAR2}A"
# Error code 23A
echo "Error code $VAR2A"
# Error code --- Note: the variable $VAR2A dosn't exist
echo "Error code ${VAR2}${VAR1}"
# Error code 23some string
echo VAR1
# VAR1
echo "VAR1"
# VAR1
This illustrates the difference between the different forms, but in general, you reference a variable's value with $+variable-name. Doing "SOME_VAR" or SOME_VAR just prints out the string "SOME_VAR" (ie, not referencing a variable at all).
The difference between $SOME_VAR and ${SOME_VAR} is that the latter lets you use it when there is other content directly before or after the variable without erroring.

How to use custom variables in gitlab ci/cd?
Normally like in any other shell.
But note that gitlab-ci.yml is a yaml file and yaml has special parsings. Because of that in script: ex. - echo bla is the same as - 'echo bla', because in yaml the content of script: is an array of strings that are later spitted by shell.
how to use variables outside and within scripts.
Normally like in any other shell script.
when to use each style
"SOME_VAR" #in quotes, no dollar sign
SOME_VAR #i.e. without the double quotes, dollar sign, and curly braces
when you want to have a string SOME_VAR literally
"${SOME_VAR}"
is the same as "$SOME_VAR". When you want to have the content of SOME_VAR variable literally.
${SOME_VAR} #no quotes, with dollar sign and wrapped with curly braces
$SOME_VAR #i.e. without the double quotes or curly braces
When you want the content of SOME_VAR variable after word splitting and filename expansion. That means that SOME_VAR='*' and then echo "$SOME_VAR" will print *, but echo $SOME_VAR will print all files in current directory. You usually always want to quote expansions.
The form ${SOME_VAR} is used if concatenated with some other string, ex. $SOME_VARbla is not ${SOME_VAR}bla.
Do not use upper case variables in your scripts - prefer lower case. Prefer using upper case variables for exported variables. Be aware of clashes - COLUMN PATH USER UID are examples of already used variables.
can I assign a variable in the script section with a bash command?
Shell is space aware. var = val will execute a command named var with two arguments = and val. var=val will assign the string val to variable named var. Do:
- SOME_VAR=$(<file_with_a_line_of_text.txt)
In gitlab-ci I would prefer to use cat in case I will want to move to alpine. $(< is a bash extension.
- SOME_VAR=$(cat file_with_a_line_of_text.txt)
There doesn't seem to be any point in setting providing SOME_VAR in environment with variables: SOME_VAR.
When do I use '$' in front of the variable?
When you want to trigger variable expansion. Variable expansion substitutes variable name for the variable value.
Check your scripts with http://shellcheck.net . Read https://mywiki.wooledge.org/BashGuide a good shell introduction and https://wiki.bash-hackers.org/scripting/obsolete .

Related

bash loop with the variable name inside file name [duplicate]

In shell scripts, when do we use {} when expanding variables?
For example, I have seen the following:
var=10 # Declare variable
echo "${var}" # One use of the variable
echo "$var" # Another use of the variable
Is there a significant difference, or is it just style? Is one preferred over the other?
In this particular example, it makes no difference. However, the {} in ${} are useful if you want to expand the variable foo in the string
"${foo}bar"
since "$foobar" would instead expand the variable identified by foobar.
Curly braces are also unconditionally required when:
expanding array elements, as in ${array[42]}
using parameter expansion operations, as in ${filename%.*} (remove extension)
expanding positional parameters beyond 9: "$8 $9 ${10} ${11}"
Doing this everywhere, instead of just in potentially ambiguous cases, can be considered good programming practice. This is both for consistency and to avoid surprises like $foo_$bar.jpg, where it's not visually obvious that the underscore becomes part of the variable name.
Variables are declared and assigned without $ and without {}. You have to use
var=10
to assign. In order to read from the variable (in other words, 'expand' the variable), you must use $.
$var # use the variable
${var} # same as above
${var}bar # expand var, and append "bar" too
$varbar # same as ${varbar}, i.e expand a variable called varbar, if it exists.
This has confused me sometimes - in other languages we refer to the variable in the same way, regardless of whether it's on the left or right of an assignment. But shell-scripting is different, $var=10 doesn't do what you might think it does!
You use {} for grouping. The braces are required to dereference array elements. Example:
dir=(*) # store the contents of the directory into an array
echo "${dir[0]}" # get the first entry.
echo "$dir[0]" # incorrect
You are also able to do some text manipulation inside the braces:
STRING="./folder/subfolder/file.txt"
echo ${STRING} ${STRING%/*/*}
Result:
./folder/subfolder/file.txt ./folder
or
STRING="This is a string"
echo ${STRING// /_}
Result:
This_is_a_string
You are right in "regular variables" are not needed... But it is more helpful for the debugging and to read a script.
Curly braces are always needed for accessing array elements and carrying out brace expansion.
It's good to be not over-cautious and use {} for shell variable expansion even when there is no scope for ambiguity.
For example:
dir=log
prog=foo
path=/var/${dir}/${prog} # excessive use of {}, not needed since / can't be a part of a shell variable name
logfile=${path}/${prog}.log # same as above, . can't be a part of a shell variable name
path_copy=${path} # {} is totally unnecessary
archive=${logfile}_arch # {} is needed since _ can be a part of shell variable name
So, it is better to write the three lines as:
path=/var/$dir/$prog
logfile=$path/$prog.log
path_copy=$path
which is definitely more readable.
Since a variable name can't start with a digit, shell doesn't need {} around numbered variables (like $1, $2 etc.) unless such expansion is followed by a digit. That's too subtle and it does make to explicitly use {} in such contexts:
set app # set $1 to app
fruit=$1le # sets fruit to apple, but confusing
fruit=${1}le # sets fruit to apple, makes the intention clear
See:
Allowed characters in Linux environment variable names
The end of the variable name is usually signified by a space or newline. But what if we don't want a space or newline after printing the variable value? The curly braces tell the shell interpreter where the end of the variable name is.
Classic Example 1) - shell variable without trailing whitespace
TIME=10
# WRONG: no such variable called 'TIMEsecs'
echo "Time taken = $TIMEsecs"
# What we want is $TIME followed by "secs" with no whitespace between the two.
echo "Time taken = ${TIME}secs"
Example 2) Java classpath with versioned jars
# WRONG - no such variable LATESTVERSION_src
CLASSPATH=hibernate-$LATESTVERSION_src.zip:hibernate_$LATEST_VERSION.jar
# RIGHT
CLASSPATH=hibernate-${LATESTVERSION}_src.zip:hibernate_$LATEST_VERSION.jar
(Fred's answer already states this but his example is a bit too abstract)
Following SierraX and Peter's suggestion about text manipulation, curly brackets {} are used to pass a variable to a command, for instance:
Let's say you have a sposi.txt file containing the first line of a well-known Italian novel:
> sposi="somewhere/myfolder/sposi.txt"
> cat $sposi
Ouput: quel ramo del lago di como che volge a mezzogiorno
Now create two variables:
# Search the 2nd word found in the file that "sposi" variable points to
> word=$(cat $sposi | cut -d " " -f 2)
# This variable will replace the word
> new_word="filone"
Now substitute the word variable content with the one of new_word, inside sposi.txt file
> sed -i "s/${word}/${new_word}/g" $sposi
> cat $sposi
Ouput: quel filone del lago di como che volge a mezzogiorno
The word "ramo" has been replaced.

bash variable substitution right next to another string [duplicate]

In shell scripts, when do we use {} when expanding variables?
For example, I have seen the following:
var=10 # Declare variable
echo "${var}" # One use of the variable
echo "$var" # Another use of the variable
Is there a significant difference, or is it just style? Is one preferred over the other?
In this particular example, it makes no difference. However, the {} in ${} are useful if you want to expand the variable foo in the string
"${foo}bar"
since "$foobar" would instead expand the variable identified by foobar.
Curly braces are also unconditionally required when:
expanding array elements, as in ${array[42]}
using parameter expansion operations, as in ${filename%.*} (remove extension)
expanding positional parameters beyond 9: "$8 $9 ${10} ${11}"
Doing this everywhere, instead of just in potentially ambiguous cases, can be considered good programming practice. This is both for consistency and to avoid surprises like $foo_$bar.jpg, where it's not visually obvious that the underscore becomes part of the variable name.
Variables are declared and assigned without $ and without {}. You have to use
var=10
to assign. In order to read from the variable (in other words, 'expand' the variable), you must use $.
$var # use the variable
${var} # same as above
${var}bar # expand var, and append "bar" too
$varbar # same as ${varbar}, i.e expand a variable called varbar, if it exists.
This has confused me sometimes - in other languages we refer to the variable in the same way, regardless of whether it's on the left or right of an assignment. But shell-scripting is different, $var=10 doesn't do what you might think it does!
You use {} for grouping. The braces are required to dereference array elements. Example:
dir=(*) # store the contents of the directory into an array
echo "${dir[0]}" # get the first entry.
echo "$dir[0]" # incorrect
You are also able to do some text manipulation inside the braces:
STRING="./folder/subfolder/file.txt"
echo ${STRING} ${STRING%/*/*}
Result:
./folder/subfolder/file.txt ./folder
or
STRING="This is a string"
echo ${STRING// /_}
Result:
This_is_a_string
You are right in "regular variables" are not needed... But it is more helpful for the debugging and to read a script.
Curly braces are always needed for accessing array elements and carrying out brace expansion.
It's good to be not over-cautious and use {} for shell variable expansion even when there is no scope for ambiguity.
For example:
dir=log
prog=foo
path=/var/${dir}/${prog} # excessive use of {}, not needed since / can't be a part of a shell variable name
logfile=${path}/${prog}.log # same as above, . can't be a part of a shell variable name
path_copy=${path} # {} is totally unnecessary
archive=${logfile}_arch # {} is needed since _ can be a part of shell variable name
So, it is better to write the three lines as:
path=/var/$dir/$prog
logfile=$path/$prog.log
path_copy=$path
which is definitely more readable.
Since a variable name can't start with a digit, shell doesn't need {} around numbered variables (like $1, $2 etc.) unless such expansion is followed by a digit. That's too subtle and it does make to explicitly use {} in such contexts:
set app # set $1 to app
fruit=$1le # sets fruit to apple, but confusing
fruit=${1}le # sets fruit to apple, makes the intention clear
See:
Allowed characters in Linux environment variable names
The end of the variable name is usually signified by a space or newline. But what if we don't want a space or newline after printing the variable value? The curly braces tell the shell interpreter where the end of the variable name is.
Classic Example 1) - shell variable without trailing whitespace
TIME=10
# WRONG: no such variable called 'TIMEsecs'
echo "Time taken = $TIMEsecs"
# What we want is $TIME followed by "secs" with no whitespace between the two.
echo "Time taken = ${TIME}secs"
Example 2) Java classpath with versioned jars
# WRONG - no such variable LATESTVERSION_src
CLASSPATH=hibernate-$LATESTVERSION_src.zip:hibernate_$LATEST_VERSION.jar
# RIGHT
CLASSPATH=hibernate-${LATESTVERSION}_src.zip:hibernate_$LATEST_VERSION.jar
(Fred's answer already states this but his example is a bit too abstract)
Following SierraX and Peter's suggestion about text manipulation, curly brackets {} are used to pass a variable to a command, for instance:
Let's say you have a sposi.txt file containing the first line of a well-known Italian novel:
> sposi="somewhere/myfolder/sposi.txt"
> cat $sposi
Ouput: quel ramo del lago di como che volge a mezzogiorno
Now create two variables:
# Search the 2nd word found in the file that "sposi" variable points to
> word=$(cat $sposi | cut -d " " -f 2)
# This variable will replace the word
> new_word="filone"
Now substitute the word variable content with the one of new_word, inside sposi.txt file
> sed -i "s/${word}/${new_word}/g" $sposi
> cat $sposi
Ouput: quel filone del lago di como che volge a mezzogiorno
The word "ramo" has been replaced.

Expand ENV variable of a string, run command and store in variable?

How can I expand a variable, run that command and store output to variable?
Usually you do this
var="$(echo string)"
but I want this
envString='echo $stringToEcho'
stringToEcho="hello world"
var="$(${envString})"`
But the dollar sign inside doesn't expand. I need to run a command stored in a variable and store the output to another variable.
If your string contains content which was written to be eval-safe:
envString="echo string"
var=$(eval "$envString")
...which will work even if it includes variable references, if quoting correctly:
envString='echo "$someVar"' # use single-quotes here to avoid premature expansion!
someVar='hello world'
var=$(eval "$envString")
However, if your string contains contents generated by expanding variables without using printf %q to safely escape any variables contained, do not do this.
References:
BashFAQ #50 ("I'm trying to put a command in a variable, but the complex cases always fail!")
BashFAQ #48 ("Eval command and security issues").

using double quotes in bash export statement

Hello I'm reading a book about bash scripting and the author says to add the following to the end of my .bashrc file. export PATH=~/bin:"$PATH" in order to execute my file from the command line by typing its name. I notice however that if I put export PATH=~/bin:$PATH I can achieve the same result. So my question is what is the difference between the one with quotes and the one without quotes? thanks.
The quotes won't hurt anything, but neither are they necessary. Assignments are processed specially by the shell. From the man page:
A variable may be assigned to by a statement of the form
name=[value]
If value is not given, the variable is assigned the null string. All values undergo tilde expansion, parameter and variable
expansion, command substitution, arithmetic expansion, and
quote removal (see EXPANSION below).
Notice that word-splitting and pathname generation are not on the list in bold. These are the two types of expansion you are trying to prevent by quoting a parameter expansion, but in this context they are not performed. The same rules apply to the assignments that are passed to the export built-in command.
You must include the variable PATH inside double quotes. So that it would handle the filepaths which has spaces but without double quotes, it won't handle the filenames which has spaces in it.
I was facing the same with trying to assign a JSON string to a variable in the terminal.
Wrap it with Single Quotes or Double Quotes
Use single quotes, if you string contains double quotes and vice-versa.
$ export TEMP_ENV='I like the "London" bridge'
$ echo $TEMP_ENV
>> I like the "London" bridge
$ export TEMP_ENV="I like the 'London' bridge"
$ echo $TEMP_ENV
>> I like the 'London' bridge

Why are shell script variables declared without a preceding `$`?

I noticed that in shell script when we declare a variable, the preceding dollar sign is not needed, although when we want to access this variable later we should add a dollar sign in front of this variable name.
just like:
#!/bin/sh
VAR_1=Hello
VAR_2=Unix
echo "$VAR_1 $VAR_2"
This is different from other languages, like Perl we will always have the preceding dollar sign with the variable name, I just want to know any good reason for shell script to do it in this way, or it's just a convention...?
Shell is a different language than Perl is a different language than C++ is a different language than Python. You can add "with different rules" to each of the languages.
In shell an identifier like VAR_1 names a variable, the dollar sign is used to invoke expansion. $var is replaced with var's content; ${var:-foo} is replaced with var's content if it is set and with the word foo if the variable isn't set. Expansion works on non-variables as well, e.g. you can chain expansion like ${${var##*/}%.*} should leave only a file base name if var contains a file name with full path and extension.
In Perl the sigil in front of the variable tells Perl how to interpret the identifier: $var is a scalar, #var an array, %var a hash etc.
In Ruby the sigil in front of the varible tells Ruby its scope: var is a local variable, $var is a global one, #var is an instance variable of an object and ##var is a class variable.
In C++ we don't have sigils in front of variable names.
Etc.
In the shell, the $ sign is not part of the variable name. It just tells the shell to replace the following word with the contents of the variable with the same name, i.e. $foo means "insert the contents of the variable foo here".
This is not used when assigning to the variable because there you explicitly don't want to insert the old contents; you want to use the variable itself (in some ways this is similar to dereferencing pointers).
It's basically a syntactical convention.
DOS/.bat file syntax works the same way.
1) to create a variable, no metacharacter.
2) to "dereference" the contents of the variable, use the metacharacter.
DOS:
set VAR=123
echo %VAR%

Resources