I have written a bash script which calls a sed command (amongst other things) on a file to complete a find/replace of 2 different strings.
The trouble is, after running the script, I check the files and nothing has been updated. However, if I run the commands that are being produced (I echo them as output anyway) then they work.
For example, inside the script I have:
echo "/usr/local/bin/sed -i -e 's/${String1}/${String1R}/g;s/\/${String2}\//\/${String2R}\//g' ${ROOT_DIR}/data/file.sql"
/usr/local/bin/sed -i -e 's/${String1}/${String1R}/g;s/\/${String2}\//\/${TString2R}\//g' ${ROOT_DIR}/data/file.sql
Running the script does not change file.sql; however, if I run the command that is printed to console e.g. /usr/local/bin/sed -i -e 's/file_name1/file_name2/g;s//path_substring1///path_substring2//g' /path/to/file/file.sql it works perfectly!
Use double quotes instead of single quotes. Single quotes would prevent variable expansion.
/usr/local/bin/sed -i -e "s/${String1}/${String1R}/g;s/\/${String2}\//\/${TString2R}\//g" ${ROOT_DIR}/data/file.sql
Moreover, it seems that your variables are path strings which might contain forward slashes, i.e. /. In that event use a different separator:
"s|${String1}|${String1R}|g"
Using a different separator would obviate the need of escaping / in the pattern and replacement.
Related
We started using shellcheck to check our scripts for errors/warnings,
Now common warning what we see in all our scripts is unquoted variables.
is there any script to correct those simple warnings/errors ?
I have below command which I use to change $VAR to ${VAR}
sed -i -r 's:\$([_a-zA-Z?][_a-zA-Z0-9]*):${\1}:g' <scriptname>
I modified it as follows,
sed -i -r 's:\$([_a-zA-Z?][_a-zA-Z0-9]*):"${\1}":g' <scriptname>
above command works fine when variables are unquoted but when they are quoted e.g. "$VAR" it changes to ""${VAR}""
any suggestion to whether continue doing it with sed or better write script to do it ?
any particular suggestions?
Edit carefully.
When you write echo "This is example ${var} in the middle of the line" you do not want to put quotes around ${var}.
You should put all variables (except PATH, PWD and some other system vars) in lowercase.
You might want to add some mappings in .vimrc, that will execute your sed first or second commandline using F4 of F5 (something like . ! ~/bin/make_my_var) making the editing easier. In make_my_var you can add logic for lowercasing the vars when they are not one of a list of exceptions.
And (edited):
You might want some more standards, perhaps use a styleguide.
I want to issue this command from the bash script
sed -e $beginning,$s/pattern/$variable/ file
but any possible combination of quotes gives me an error, only one that works:
sed -e "$beginning,$"'s/pattern/$variable/' file
also not good, because it do not dereferences the variable.
Does my approach can be implemented with sed?
Feel free to switch the quotes up. The shell can keep things straight.
sed -e "$beginning"',$s/pattern/'"$variable"'/' file
You can try this:
$ sed -e "$beginning,$ s/pattern/$variable/" file
Example
file.txt:
one
two
three
Try:
$ beginning=1
$ variable=ONE
$ sed -e "$beginning,$ s/one/$variable/" file.txt
Output:
ONE
two
three
There are two types of quotes:
Single quotes preserve their contents (> is the prompt):
> var=blah
> echo '$var'
$var
Double quotes allow for parameter expansion:
> var=blah
> echo "$var"
blah
And two types of $ sign:
One to tell the shell that what follows is the name of a parameter to be expanded
One that stands for "last line" in sed.
You have to combine these so
The shell doesn't think sed's $ has anything to do with a parameter
The shell parameters still get expanded (can't be within single quotes)
The whole sed command is quoted.
One possibility would be
sed "$beginning,\$s/pattern/$variable/" file
The whole command is in double quotes, i.e., parameters get expanded ($beginning and $variable). To make sure the shell doesn't try to expand $s, which doesn't exist, the "end of line" $ is escaped so the shell doesn't try anything funny.
Other options are
Double quoting everything but adding a space between $ and s (see Ren's answer)
Mixing quoting types as needed (see Ignacio's answer)
Methods that don't work
sed '$beginning,$s/pattern/$variable/' file
Everything in single quotes: the shell parameters are not expanded (doesn't follow rule 2 above). $beginning is not a valid address, and pattern would be literally replaced by $variable.
sed "$beginning,$s/pattern/$variable/" file
Everything in double qoutes: the parameters are expanded, including $s, which isn't supposed to (doesn't follow rule 1 above).
the following form worked for me from within script
sed $beg,$ -e s/pattern/$variable/ file
the same form will also work if executed from the shell
I'm implementing a template renderer in shell script. The template variables are represented in a template by #VAR_NAME# and their values are defined in a separate shell script.
Sample code:
# template variables values
CONTACT_EMAIL="myemail"
CONTACT_NAME="myname"
VARS="CONTACT_EMAIL CONTACT_NAME"
TEMPLATE_FILEPATH="mytemplate.txt"
# template renderer
set -x
SEDARGS=
for VAR in $VARS; do
SEDARGS+=" -e \"s,#$VAR#,${!VAR},\""
done
sed -r $SEDARGS $TEMPLATE_FILEPATH
sed command executed by shell and printed by it because of "set -x":
+ sed -r -e '"s,#CONTACT_EMAIL#,myemail,"' -e '"s,#CONTACT_NAME#,myname,"' mytemplate.txt
sed output:
sed: -e expression #1, char 1: unknown command: `"'
I know the single quotes around each sed expression are causing this non-intuitive error message, but I do not know why they are added.
What is wrong?
You have embedded quotes inside your SEDARGS variable. These are NOT removed when the command is executed. To remove them, you need to call the interpreter again, which you can do using eval. For example:
eval sed -r $SEDARGS $TEMPLATE_FILEPATH
You may need to play around that some more (adding quotes, etc.).
The single quotes aren't part of the actual arguments. They get added by your shell for the output caused by set -x only.
Why would the shell do that? So that you can use that output to re-run exactly what was executed during the script execution. As you correctly noticed, they are needed to protect the " that came from SEDARGS content (i.e., the inner ones, in your script escaped as \").
I'm trying to use the send command with a sed which uses a variable.
Having trouble to escape correctly.
send "sed "1i//$VAR" /file > /tmp/out\r"
If I use the sed command separately (which adds the $VAR text as first line of file), it works:
sed "1i//$VAR" /file > /tmp/out
But I can't figure out how to escape within the send command.
Inside double quotes, single quotes lose their special meaning, so you probably need:
send "sed '1i//$VAR' /file > /tmp/out\r"
On the local machine, the $VAR is placed into the command. On the remote machine, the sed command is enclosed in single quotes, protecting it from further abuse.
This should work:
send "sed '1i//$VAR' /file > /tmp/out\r"
Only the type of the outermost quotes matters for determining whether variables are interpolated.
I want to use sed for replacing multiple files from bash script.
When I call it from bash I get below error
DEBUG FLOW:-
FILELIST='/tmp/components/ab.sql /tmp/b.sql'
+ SUBSTITUTE_STRING=abc
+ sed -i.bak -e s/abc/xyz/g '/tmp/components/ab.sql /tmp/b.sql': No such file or directory
however when I used this command directly on terminal it executes successfully
sed -i.bak -e s/abc/xyz/g /tmp/components/ab.sql /tmp/b.sql
The difference from terminal and script is of quotes around the file.
I have tried defining File list variable without quotes as well
kindly suggest
Instead of saying:
FILELIST='/tmp/components/ab.sql /tmp/b.sql'
make it an array by saying:
FILELIST=(/tmp/components/ab.sql /tmp/b.sql)
and while invoking say:
sed -i.bak -e "s/abc/xyz/g" "${FILELIST[#]}"
If you look at the debug flow, it'd be evident that the shell parses the filenames as a single token ('/tmp/components/ab.sql /tmp/b.sql') which causes the No such file or directory error.