We started using shellcheck to check our scripts for errors/warnings,
Now common warning what we see in all our scripts is unquoted variables.
is there any script to correct those simple warnings/errors ?
I have below command which I use to change $VAR to ${VAR}
sed -i -r 's:\$([_a-zA-Z?][_a-zA-Z0-9]*):${\1}:g' <scriptname>
I modified it as follows,
sed -i -r 's:\$([_a-zA-Z?][_a-zA-Z0-9]*):"${\1}":g' <scriptname>
above command works fine when variables are unquoted but when they are quoted e.g. "$VAR" it changes to ""${VAR}""
any suggestion to whether continue doing it with sed or better write script to do it ?
any particular suggestions?
Edit carefully.
When you write echo "This is example ${var} in the middle of the line" you do not want to put quotes around ${var}.
You should put all variables (except PATH, PWD and some other system vars) in lowercase.
You might want to add some mappings in .vimrc, that will execute your sed first or second commandline using F4 of F5 (something like . ! ~/bin/make_my_var) making the editing easier. In make_my_var you can add logic for lowercasing the vars when they are not one of a list of exceptions.
And (edited):
You might want some more standards, perhaps use a styleguide.
Related
I wrote a little bash script to make a "shell" that prints the date before and after a command is executed:
while true
do date
printf "Prompt: "
read x
date
$x #Solution: this should be eval "$x"
done
The problem is that this "shell" doesn't recognize variables in $x, so for instance echo $PWD outputs $PWD instead of the current directory. A related consequence is that cd does not work. How to fix/work around this? Thanks.
EDIT: Solved. Thanks everyone :-)
LATE EDIT: Actually the above isn't quite the solution; I needed to add the -r option to the read command, to prevent backslash sequences from being prematurely evaluated. Furthermore, the double quotes appear to be superfluous after all in this case (since the prepass would expand $x and "$x" to the same thing unless $x is empty, and eval does the same thing with no argument as it does with an empty string), but I suppose it never hurts to always include the quotes as a matter of security and good style.
If you're just trying to print out the date before and after, you can do that with PROMPT_COMMAND="$(date)" in your .bashrc. This will print out the date before your command prompt (meaning that it will also functionally be after your command since you'll get a prompt after the command finishes). It will be on a separate line from your PS1, but you could add this to your PS1 instead if you'd rather it be there.
If you're trying to type in a literal echo $PWD and have x give the value of that, you can do eval "$x" but most people will recommend against using eval because anyone can type anything, like sudo rm -rf / --no-preserve-root
If you're trying to set x to be the name of a variable, i.e., x=PWD, and you want to get the value of PWD, you can do ${!x}. This is called "indirection".
This is a simple question but i am unable to find it in tutorials. Could anybody please explain what this statement does when executed in a bash shell within a folder containing .sh scripts. I know -i does in place editing, i understand that it will run sed on all scripts within the current directory. And i know that it does some sort of substitution. But what does this \(.*\) mean?
sed -i 's/MY_BASE_DIR=\(.*\)/MY_BASE_DIR=${MY_BASE_DIR-\1}/' *.sh
Thanks in advance.
You have an expression like:
sed -i 's/XXX=\(YYY\)/XXX=ZZZ/' file
This looks for a string XXX= in a file and captures what goes after. Then, it replaces this captured content with ZZZ. Since there is a captured group, it is accessed with \1. Finally, using the -i flag in sed makes the edition to be in-place.
For the replacement, it uses the following syntax described in Shell parameter expansion:
${parameter:-word}
If parameter is unset or null, the expansion of word is substituted.
Otherwise, the value of parameter is substituted.
Example:
$ d=5
$ echo ${d-3}
5
$ echo ${a-3}
3
So with ${MY_BASE_DIR-SOMETHING-\1} you are saying: print $MY_BAS_DIR. And if this variable is unset or null, print what is stored in \1.
All together, this is resetting MY_BASE_DIR to the value in the variable $MY_BASE_DIR unless this is not set; in such case, the value remains the same.
Note though that the variable won't be expanded unless you use double quotes.
Test:
$ d=5
$ cat a
d=23
blabla
$ sed "s/d=\(.*\)/d=${d-\1}/" a # double quotes -> value is replaced
d=5
blabla
$ sed 's/d=\(.*\)/d=${d-\1}/' a # single quotes -> variable is not expanded
d=${d-23}
blabla
Andd see how the value remains the same if $d is not set:
$ unset d
$ sed "s/d=\(.*\)/d=${d-\1}/" a
d=23
The scripts contain lines like this:
MY_BASE_DIR=/usr/local
The sed expression changes them to:
MY_BASE_DIR=${MY_BASE_DIR-/usr/local}
The effect is that /usr/local is not used as a fixed value, but only as the default value. You can override it by setting the environment variable MY_BASE_DIR.
For future reference, I would take a look at the ExplainShell website:
http://explainshell
that will give you a breakdown of the command structure etc. In this instance, let step through the details...Let's start with a simple example, let's assume that we were going to make the simple change - commenting out all lines by adding a "#" before each line. We can do this for all *.sh files in a directory with the ".sh" extension in the current directory:
sed 's/^/\#/' *.sh
i.e. Substitute beginning of line ^, with a # ...
Caveat: You did not specify the OS you are using. You may get different results with different versions of sed and OS...
ok, now we can drill into the substitution in the script. An example is probably easier to explain:
File: t.sh
MY_BASE_DIR="/important data/data/bin"
the command 's/MY_BASE_DIR=\(.*\)/MY_BASE_DIR=${MY_BASE_DIR-\1}/' *.sh
will search for "MY_BASE_DIR" in each .sh file in the directory.
When it encounters the string "MY_BASE_DIR=.*", in the file, it expands it to be MY_BASE_DIR="/important data/data/bin", this is now replaced on the right side of the expression /MY_BASE_DIR=${MY_BASE_DIR-\1}/ which becomes
MY_BASE_DIR=${MY_BASE_DIR-"/important data/data/bin"}
essentially what happens is that the substitute operation takes
MY_BASE_DIR="/important data/data/bin"
and inserts
MY_BASE_DIR=${MY_BASE_DIR-"/important data/data/bin"}
now if we run the script with the variable MY_BASE_DIR set
export MY_BASE_DIR="/new/import/dir"
the scripts modified by the sed script referenced will now substitute /important data/data/bin with /new/import/dir...
I want to issue this command from the bash script
sed -e $beginning,$s/pattern/$variable/ file
but any possible combination of quotes gives me an error, only one that works:
sed -e "$beginning,$"'s/pattern/$variable/' file
also not good, because it do not dereferences the variable.
Does my approach can be implemented with sed?
Feel free to switch the quotes up. The shell can keep things straight.
sed -e "$beginning"',$s/pattern/'"$variable"'/' file
You can try this:
$ sed -e "$beginning,$ s/pattern/$variable/" file
Example
file.txt:
one
two
three
Try:
$ beginning=1
$ variable=ONE
$ sed -e "$beginning,$ s/one/$variable/" file.txt
Output:
ONE
two
three
There are two types of quotes:
Single quotes preserve their contents (> is the prompt):
> var=blah
> echo '$var'
$var
Double quotes allow for parameter expansion:
> var=blah
> echo "$var"
blah
And two types of $ sign:
One to tell the shell that what follows is the name of a parameter to be expanded
One that stands for "last line" in sed.
You have to combine these so
The shell doesn't think sed's $ has anything to do with a parameter
The shell parameters still get expanded (can't be within single quotes)
The whole sed command is quoted.
One possibility would be
sed "$beginning,\$s/pattern/$variable/" file
The whole command is in double quotes, i.e., parameters get expanded ($beginning and $variable). To make sure the shell doesn't try to expand $s, which doesn't exist, the "end of line" $ is escaped so the shell doesn't try anything funny.
Other options are
Double quoting everything but adding a space between $ and s (see Ren's answer)
Mixing quoting types as needed (see Ignacio's answer)
Methods that don't work
sed '$beginning,$s/pattern/$variable/' file
Everything in single quotes: the shell parameters are not expanded (doesn't follow rule 2 above). $beginning is not a valid address, and pattern would be literally replaced by $variable.
sed "$beginning,$s/pattern/$variable/" file
Everything in double qoutes: the parameters are expanded, including $s, which isn't supposed to (doesn't follow rule 1 above).
the following form worked for me from within script
sed $beg,$ -e s/pattern/$variable/ file
the same form will also work if executed from the shell
I have written a bash script which calls a sed command (amongst other things) on a file to complete a find/replace of 2 different strings.
The trouble is, after running the script, I check the files and nothing has been updated. However, if I run the commands that are being produced (I echo them as output anyway) then they work.
For example, inside the script I have:
echo "/usr/local/bin/sed -i -e 's/${String1}/${String1R}/g;s/\/${String2}\//\/${String2R}\//g' ${ROOT_DIR}/data/file.sql"
/usr/local/bin/sed -i -e 's/${String1}/${String1R}/g;s/\/${String2}\//\/${TString2R}\//g' ${ROOT_DIR}/data/file.sql
Running the script does not change file.sql; however, if I run the command that is printed to console e.g. /usr/local/bin/sed -i -e 's/file_name1/file_name2/g;s//path_substring1///path_substring2//g' /path/to/file/file.sql it works perfectly!
Use double quotes instead of single quotes. Single quotes would prevent variable expansion.
/usr/local/bin/sed -i -e "s/${String1}/${String1R}/g;s/\/${String2}\//\/${TString2R}\//g" ${ROOT_DIR}/data/file.sql
Moreover, it seems that your variables are path strings which might contain forward slashes, i.e. /. In that event use a different separator:
"s|${String1}|${String1R}|g"
Using a different separator would obviate the need of escaping / in the pattern and replacement.
I’m trying to build a command string based to pass in a “-e” flag and another variable into a another base script being call as a subroutine and have run into a strange problem; I’m losing the “-e” portion of the string when I pass it into the subroutine. I create a couple example which illustrate the issue, any help?
This works as you would expect:
$echo "-e $HOSTNAME"
-e ops-wfm
This does NOT; we lose the “-e” because it is interpreted as a special qualifier.
$myFlag="-e $HOSTNAME"; echo $myFlag
ops-wfm
Adding the “\” escape charactor doesn’t work either, I get the correct string with the "\" in front:
$myFlag="\-e $HOSTNAME"; echo $myFlag
\-e ops-wfm
How can I prevent -e being swallowed?
Use double-quotes:
$ myFlag="-e $HOSTNAME"; echo "${myFlag}"
-e myhost.local
I use ${var} rather than $var out of habit as it means that I can add characters after the variable without the shell interpreting them as part of the variable name.
echo may not be the best example here. Most Unix commands will accept -- to mark no more switches.
$ var='-e .bashrc' ; ls -l -- "${var}"
ls: -e .bashrc: No such file or directory
Well, you could put your variable in quotes:
echo "$myFlag"
...making it equivalent to your first example, which, as you say, works just fine.