I noticed that in shell script when we declare a variable, the preceding dollar sign is not needed, although when we want to access this variable later we should add a dollar sign in front of this variable name.
just like:
#!/bin/sh
VAR_1=Hello
VAR_2=Unix
echo "$VAR_1 $VAR_2"
This is different from other languages, like Perl we will always have the preceding dollar sign with the variable name, I just want to know any good reason for shell script to do it in this way, or it's just a convention...?
Shell is a different language than Perl is a different language than C++ is a different language than Python. You can add "with different rules" to each of the languages.
In shell an identifier like VAR_1 names a variable, the dollar sign is used to invoke expansion. $var is replaced with var's content; ${var:-foo} is replaced with var's content if it is set and with the word foo if the variable isn't set. Expansion works on non-variables as well, e.g. you can chain expansion like ${${var##*/}%.*} should leave only a file base name if var contains a file name with full path and extension.
In Perl the sigil in front of the variable tells Perl how to interpret the identifier: $var is a scalar, #var an array, %var a hash etc.
In Ruby the sigil in front of the varible tells Ruby its scope: var is a local variable, $var is a global one, #var is an instance variable of an object and ##var is a class variable.
In C++ we don't have sigils in front of variable names.
Etc.
In the shell, the $ sign is not part of the variable name. It just tells the shell to replace the following word with the contents of the variable with the same name, i.e. $foo means "insert the contents of the variable foo here".
This is not used when assigning to the variable because there you explicitly don't want to insert the old contents; you want to use the variable itself (in some ways this is similar to dereferencing pointers).
It's basically a syntactical convention.
DOS/.bat file syntax works the same way.
1) to create a variable, no metacharacter.
2) to "dereference" the contents of the variable, use the metacharacter.
DOS:
set VAR=123
echo %VAR%
Related
I'm struggling with gitlab ci/cd variables. I see so many conflicting examples. Anyhow, what I would like to know is how to use variables outside and within scripts.
For example, in a job config, can I assign a variable in the script section with a bash command?
some-job:
variables:
SOME_VAR: ''
script:
- SOME_VAR = $(<file_with_a_line_of_text.txt)
In the above case, I'm not sure if I can do this. But I need to populate a variable with the file contents (i.e. artifact). Also, when do I use '$' in front of the variable? Some examples I see using these formats:
"SOME_VAR" #in quotes, no dollar sign
"${SOME_VAR}" #in quotes, with dollar sign and wrapped with curly braces
${SOME_VAR} #no quotes, with dollar sign and wrapped with curly braces
$SOME_VAR #i.e. without the double quotes or curly braces
SOME_VAR #i.e. without the double quotes, dollar sign, and curly braces
So many variations of usage that I can see in examples but don't really know when to use each style. And I can't find one example online of a custom variable being set in a script using a bash command.
When I'm setting variables in bash, I always do it without the spaces around the =:
VAR1="some string"
VAR2=23
VAR3=true
VAR4=$(cat /path/to/file.txt)
Let's go through these examples one at a time:
You can set a variable as a string by using quotes around the string.
You can set it to an int (probably a float too, but haven't personally used it)
You can set it to a bool
You can set it to the output of a command. The command is inside the command: $(#command).
Now let's use them:
echo $VAR1
# some string
echo "This is my variable $VAR1"
# This is my variable some string
echo "This is my variable ${VAR1}"
# This is my variable some string
echo ${VAR1}
# some string
echo "Error code ${VAR2}A"
# Error code 23A
echo "Error code $VAR2A"
# Error code --- Note: the variable $VAR2A dosn't exist
echo "Error code ${VAR2}${VAR1}"
# Error code 23some string
echo VAR1
# VAR1
echo "VAR1"
# VAR1
This illustrates the difference between the different forms, but in general, you reference a variable's value with $+variable-name. Doing "SOME_VAR" or SOME_VAR just prints out the string "SOME_VAR" (ie, not referencing a variable at all).
The difference between $SOME_VAR and ${SOME_VAR} is that the latter lets you use it when there is other content directly before or after the variable without erroring.
How to use custom variables in gitlab ci/cd?
Normally like in any other shell.
But note that gitlab-ci.yml is a yaml file and yaml has special parsings. Because of that in script: ex. - echo bla is the same as - 'echo bla', because in yaml the content of script: is an array of strings that are later spitted by shell.
how to use variables outside and within scripts.
Normally like in any other shell script.
when to use each style
"SOME_VAR" #in quotes, no dollar sign
SOME_VAR #i.e. without the double quotes, dollar sign, and curly braces
when you want to have a string SOME_VAR literally
"${SOME_VAR}"
is the same as "$SOME_VAR". When you want to have the content of SOME_VAR variable literally.
${SOME_VAR} #no quotes, with dollar sign and wrapped with curly braces
$SOME_VAR #i.e. without the double quotes or curly braces
When you want the content of SOME_VAR variable after word splitting and filename expansion. That means that SOME_VAR='*' and then echo "$SOME_VAR" will print *, but echo $SOME_VAR will print all files in current directory. You usually always want to quote expansions.
The form ${SOME_VAR} is used if concatenated with some other string, ex. $SOME_VARbla is not ${SOME_VAR}bla.
Do not use upper case variables in your scripts - prefer lower case. Prefer using upper case variables for exported variables. Be aware of clashes - COLUMN PATH USER UID are examples of already used variables.
can I assign a variable in the script section with a bash command?
Shell is space aware. var = val will execute a command named var with two arguments = and val. var=val will assign the string val to variable named var. Do:
- SOME_VAR=$(<file_with_a_line_of_text.txt)
In gitlab-ci I would prefer to use cat in case I will want to move to alpine. $(< is a bash extension.
- SOME_VAR=$(cat file_with_a_line_of_text.txt)
There doesn't seem to be any point in setting providing SOME_VAR in environment with variables: SOME_VAR.
When do I use '$' in front of the variable?
When you want to trigger variable expansion. Variable expansion substitutes variable name for the variable value.
Check your scripts with http://shellcheck.net . Read https://mywiki.wooledge.org/BashGuide a good shell introduction and https://wiki.bash-hackers.org/scripting/obsolete .
I have this makefile:
echo:
echo "PASS=$(PASS)"
Which I invoke:
PASS='MYPA$$' make
Which shows me:
echo "PASS=MYPA$"
PASS=MYPA$
Somebody is evaluating $$ -> $.
Is this the shell? Not when inputting the value, since I use single-quotes, preventing the shell to evaluate it.
Maybe the shell invoked by make is doing this ...
Or is it maybe make itself?
How can I avoid it?
On make variables
It's better to think of make variables as macros, than as conventional variables (actually in some versions of make, variables are called macros). The reason is, each time a variable is referenced, it is expanded.
An example from the docs illustrates the standard recursively expanded variables behaviour:
foo = $(bar)
bar = $(ugh)
ugh = Huh?
all:
echo $(foo)
# echoes: Huh?
# `$(foo)' expands to `$(bar)' which expands to `$(ugh)' which finally expands to `Huh?'
If you are using GNU make, one way to avoid further expansion is by using the simply expanded variables:
Simply expanded variables are defined by lines using := (see section Setting Variables). The value of a simply expanded variable is scanned once and for all, expanding any references to other variables and functions, when the variable is defined. The actual value of the simply expanded variable is the result of expanding the text that you write. It does not contain any references to other variables; it contains their values as of the time this variable was defined.
Although simply expanded variables behave more like variables in most programming languages, their sole usage wouldn't solve the problem of environment variables' expansion here because even the first reference var := $(PASS) would expand $$ from the PASS environment variable.
Avoiding expansion of environment variables
We can use the shell function in make to read our environment variable in shell (and not expand it in make):
expanded := $(shell echo "$$PASS")
test:
echo 'PASS=$(expanded)'
echo "PASS=$$PASS"
The shell function will execute echo "$PASS" in shell ($$ is expanded to $ by make when function is executed), and the result (the value of your shell variable PASS) will be stored in the make variable expanded. This variable can now be freely used elsewhere in make, without ever being further expanded.
The only processing make does on the result, before substituting it into the surrounding text, is to convert each newline or carriage-return / newline pair to a single space. It also removes the trailing (carriage-return and) newline, if it's the last thing in the result.
The example above illustrates how to use the make variable expanded and the environment variable PASS in your Makefile script:
$ PASS='MYPA$$' make
echo 'PASS=MYPA$$'
PASS=MYPA$$
echo "PASS=$PASS"
PASS=MYPA$$
For each of two examples below I'll try to explain what result I expected and what I got instead. I'm hoping for you to help me understand why I was wrong.
1)
VAR1=VAR2
$VAR1=FOO
result: -bash: VAR2=FOO: command not found
In the second line, $VAR1 gets expanded to VAR2, but why does Bash interpret the resulting VAR2=FOO as a command name rather than a variable assignment?
2)
'VAR=FOO'
result: -bash: VAR=FOO: command not found
Why do the quotes make Bash treat the variable assignment as a command name?
Could you please describe, step by step, how Bash processes my two examples?
How best to indirectly assign variables is adequately answered in other Q&A entries in this knowledgebase. Among those:
Indirect variable assignment in bash
Saving function output into a variable named in an argument
If that's what you actually intend to ask, then this question should be closed as a duplicate. I'm going to make a contrary assumption and focus on the literal question -- why your other approaches failed -- below.
What does the POSIX sh language specify as a valid assignment? Why does $var1=foo or 'var=foo' fail?
Background: On the POSIX sh specification
The POSIX shell command language specification is very specific about what constitutes an assignment, as quoted below:
4.21 Variable Assignment
In the shell command language, a word consisting of the following parts:
varname=value
When used in a context where assignment is defined to occur and at no other time, the value (representing a word or field) shall be assigned as the value of the variable denoted by varname.
The varname and value parts shall meet the requirements for a name and a word, respectively, except that they are delimited by the embedded unquoted equals-sign, in addition to other delimiters.
Also, from section 2.9.1, on Simple Commands, with emphasis added:
The words that are recognized as variable assignments or redirections according to Shell Grammar Rules are saved for processing in steps 3 and 4.
The words that are not variable assignments or redirections shall be expanded. If any fields remain following their expansion, the first field shall be considered the command name and remaining fields are the arguments for the command.
Redirections shall be performed as described in Redirection.
Each variable assignment shall be expanded for tilde expansion, parameter expansion, command substitution, arithmetic expansion, and quote removal prior to assigning the value.
Also, from the grammar:
If all the characters preceding '=' form a valid name (see the Base Definitions volume of IEEE Std 1003.1-2001, Section 3.230, Name), the token ASSIGNMENT_WORD shall be returned. (Quoted characters cannot participate in forming a valid name.)
Note from this:
The command must be recognized as an assignment at the very beginning of the parsing sequence, before any expansions (or quote removal!) have taken place.
The name must be a valid name. Literal quotes are not part of a valid variable name.
The equals sign must be unquoted. In your second example, the entire string was quoted.
Assignments are recognized before tilde expansion, parameter expansion, command substitution, etc.
Why $var1=foo fails to act as an assignment
As given in the grammar, all characters before the = in an assignment must be valid characters within a variable name for an assignment to be recognized. $ is not a valid character in a name. Because assignments are recognized in step 1 of simple command processing, before expansion takes place, the literal text $var1, not the value of that variable, is used for this matching.
Why 'var=foo' fails to act as an assignment
First, all characters before the = must be valid in variable names, and ' is not valid in a variable name.
Second, an assignment is only recognized if the = is not quoted.
1)
VAR1=VAR2
$VAR1=FOO
You want to use a variable name contained in a variable for the assignment. Bash syntax does not allow this. However, there is an easy workaround :
VAR1=VAR2
declare "$VAR1"=FOO
It works with local and export too.
2)
By using single quotes (double quotes would yield the same result), you are telling Bash that what is inside is a string and to treat it as a single entity. Since it is the first item on the line, Bash tries to find an alias, or shell builtin, or an executable file in its PATH, that would be named VAR=FOO. Not finding it, it tells you there is no such command.
An assignment is not a normal command. To perform an assignment contained in a quote, you would need to use eval, like so :
eval "$VAR1=FOO" # But please don't do that in real life
Most experienced bash programmers would probably tell you to avoid eval, as it has serious drawbacks, and I am giving it as an example just to recommend against its use : while in the example above it would not involve any security risk or error potential because the value of VAR1 is known and safe, there are many cases where an arbitrary (i.e. user-supplied) value could cause a crash or unexpected behavior. Quoting inside an eval statement is also more difficult and reduces readability.
You declare VAR2 earlier in the program, right?
If you are trying to assign the value of VAR2 to VAR1, then you need to make sure and use $ in front of VAR2, like so:
VAR1=$VAR2
That will set the value of VAR2 equal to VAR1, because when you utilize the $, you are saying that value that is stored in the variable. Otherwise it doesn't recognize it as a variable.
Basically, a variable that doesn't have a $ in front of it will be interpreted as a command. Any word will. That's why we have the $ to clarify "hey this is a variable".
I want to define variable in shell script as:
value1 = 40 (this can be number or character)
and want to use as in a text like:
$value1_position.xyz (I basically want 40_position.xyz)
How do I do this?
this should do:
${value1}_position.xyz
beware that the variable should be declared with this syntax
value1=40
notice the absence of spaces around the =
To define a variable, simply make sure there are no spaces between the variable name and value
value1=40
To use that variable in bash substitution, creating what you want, use the $ replacement symbol like so:
${value1}_position.xyz
To append that to your text file
echo "${value1}_position.xyz" >> file.txt
can someone explain me with this code
data=$(date +"%Y-%m-%dS%H:%M:%S")
name="/home/cft/"$data"_test.tar"
touch $name
works, creating a new .tar file but this code doesn't work
data=$(date +"%Y-%m-%dS%H:%M:%S")
name= "/home/cft/"$data"_test.tar"
touch $name
and gives me this error: no such file or directory?
why the space between = and inverted commas creates this error?
Shell allows you to provide per-command environment overrides by prefixing the command with one or more variable assignments.
name= "/home/cft/"$data"_test.tar"
asks the shell to run the program named /home/cft/2013-10-08S12:00:00_test.tar (for example) with the value of name set to the empty string in its environment.
(In your case, the error occurs because the named tar file either doesn't exist or, if it does, is not an executable file.)
A variable assignment is identified by having no whitespace after the equal sign.
(name = whatever, of course, is simply a command called name with two string arguments, = and whatever.)
You can't have whitespace between the equal sign and the definition.
http://www.tldp.org/LDP/abs/html/varassignment.html
There is no theory behind this. It's just a decision the language designers made, and which the parser enforces.
In BASH (and other Bourne type shells like zsh and Kornshell), the equal sign cannot have spaces around it when setting variables.
Good:
$ foo="bar"
Bad:
$ foo= "bar"
$ foo = "bar"
There's no real reason that would prevent spaces from being used. Other programming languages have no problems with this. It's just the syntax of the shell itself.
The reason might be related to the original Bourne shell parsing where the shell would break up a command line based upon whitespace. That would make foo=bar a single parameter instead of two or three (depending if you have white space on both sides or just one side of the equal sign). The shell could see the = sign, and know this parameter is an assignment.
The shell parameter parsing is very primitive in many ways. Whitespace is very important. The shell has to be small and fast in order to be responsive. That means stripping down unessential things like complex line parsing.
Inverted commas I believe you mean quotation marks. Double quotes are used to override the breaking out of parameters over white space:
Bad:
$ foo=this is a test
bash: is: command not found
Good:
$ foo="this is a test"
Double quotes allow interpolation. Single quotes don't:
$ foo="bar"
$ echo "The value of foo is $foo"
The value of foo is bar
$ echo 'The value of foo is $foo'
The value of foo is $foo.
If you start out with single quotes, you can put double quotes inside. If you have single quotes, you can put double quotes inside.
$ foo="bar"
$ echo "The value of foo is '$foo'"
The value of foo is 'bar'
$ echo 'The value of foo is "$foo"'
The value of foo is "$foo"
This means you didn't have to unquote $data. However, you would have to put curly braces around it because underscores are legal characters in variable names. Thus, you want to make sure that the shell understand that the variable is $data and not $data_backup:
name="/home/cft/${data}_test.tar"