I'm implementing a template renderer in shell script. The template variables are represented in a template by #VAR_NAME# and their values are defined in a separate shell script.
Sample code:
# template variables values
CONTACT_EMAIL="myemail"
CONTACT_NAME="myname"
VARS="CONTACT_EMAIL CONTACT_NAME"
TEMPLATE_FILEPATH="mytemplate.txt"
# template renderer
set -x
SEDARGS=
for VAR in $VARS; do
SEDARGS+=" -e \"s,#$VAR#,${!VAR},\""
done
sed -r $SEDARGS $TEMPLATE_FILEPATH
sed command executed by shell and printed by it because of "set -x":
+ sed -r -e '"s,#CONTACT_EMAIL#,myemail,"' -e '"s,#CONTACT_NAME#,myname,"' mytemplate.txt
sed output:
sed: -e expression #1, char 1: unknown command: `"'
I know the single quotes around each sed expression are causing this non-intuitive error message, but I do not know why they are added.
What is wrong?
You have embedded quotes inside your SEDARGS variable. These are NOT removed when the command is executed. To remove them, you need to call the interpreter again, which you can do using eval. For example:
eval sed -r $SEDARGS $TEMPLATE_FILEPATH
You may need to play around that some more (adding quotes, etc.).
The single quotes aren't part of the actual arguments. They get added by your shell for the output caused by set -x only.
Why would the shell do that? So that you can use that output to re-run exactly what was executed during the script execution. As you correctly noticed, they are needed to protect the " that came from SEDARGS content (i.e., the inner ones, in your script escaped as \").
Related
I am trying to change the values in a text file using sed in a Bash script with the line,
sed 's/draw($prev_number;n_)/draw($number;n_)/g' file.txt > tmp
This will be in a for loop. Why is it not working?
Variables inside ' don't get substituted in Bash. To get string substitution (or interpolation, if you're familiar with Perl) you would need to change it to use double quotes " instead of the single quotes:
# Enclose the entire expression in double quotes
$ sed "s/draw($prev_number;n_)/draw($number;n_)/g" file.txt > tmp
# Or, concatenate strings with only variables inside double quotes
# This would restrict expansion to the relevant portion
# and prevent accidental expansion for !, backticks, etc.
$ sed 's/draw('"$prev_number"';n_)/draw('"$number"';n_)/g' file.txt > tmp
# A variable cannot contain arbitrary characters
# See link in the further reading section for details
$ a='foo
bar'
$ echo 'baz' | sed 's/baz/'"$a"'/g'
sed: -e expression #1, char 9: unterminated `s' command
Further Reading:
Difference between single and double quotes in Bash
Is it possible to escape regex metacharacters reliably with sed
Using different delimiters for sed substitute command
Unless you need it in a different file you can use the -i flag to change the file in place
Variables within single quotes are not expanded, but within double quotes they are. Use double quotes in this case.
sed "s/draw($prev_number;n_)/draw($number;n_)/g" file.txt > tmp
You could also make it work with eval, but don’t do that!!
This may help:
sed "s/draw($prev_number;n_)/draw($number;n_)/g"
You can use variables like below. Like here, I wanted to replace hostname i.e., a system variable in the file. I am looking for string look.me and replacing that whole line with look.me=<system_name>
sed -i "s/.*look.me.*/look.me=`hostname`/"
You can also store your system value in another variable and can use that variable for substitution.
host_var=`hostname`
sed -i "s/.*look.me.*/look.me=$host_var/"
Input file:
look.me=demonic
Output of file (assuming my system name is prod-cfm-frontend-1-usa-central-1):
look.me=prod-cfm-frontend-1-usa-central-1
I needed to input github tags from my release within github actions. So that on release it will automatically package up and push code to artifactory.
Here is how I did it. :)
- name: Invoke build
run: |
# Gets the Tag number from the release
TAGNUMBER=$(echo $GITHUB_REF | cut -d / -f 3)
# Setups a string to be used by sed
FINDANDREPLACE='s/${GITHUBACTIONSTAG}/'$(echo $TAGNUMBER)/
# Updates the setup.cfg file within version number
sed -i $FINDANDREPLACE setup.cfg
# Installs prerequisites and pushes
pip install -r requirements-dev.txt
invoke build
Retrospectively I wish I did this in python with tests. However it was fun todo some bash.
Another variant, using printf:
SED_EXPR="$(printf -- 's/draw(%s;n_)/draw(%s;n_)/g' $prev_number $number)"
sed "${SED_EXPR}" file.txt
or in one line:
sed "$(printf -- 's/draw(%s;n_)/draw(%s;n_)/g' $prev_number $number)" file.txt
Using printf to build the replacement expression should be safe against all kinds of weird things, which is why I like this variant.
Searching pattern and displaying with SED which have double code (") and variable need to pass.
Actual log is:
<confirmation ID="123456-109" status>
Want to Print 109
This is with actual sed command which worked :
sed -n 's%.*confirmation="123456-\(.*\)" status.*%\%p' /tmp/log
output: 109
But when I tried use this in script where passing 123456 as variable it is not working.
req_data=`sed -n 's%.*confirmation="$variable-\(.*\)" status.*%\%p' /tmp/log`
When I run in Script is is not giving me any output.
I am expecting output: 109
Execute a command and assign its output to variable res as res="$(command)".
You need to used sed -E to make parenthesis work for grouping (or in normal mode, you have to escape each parenthesis). (note that -E option of sed is not part of the posix standard)
Also, as stated in the comments, variables are not expanded/interpolated inside single quotes. If you are in a single quoted block, terminate that before the variable and start a different block after the variable.
Following should addresses all the above and should solve your problem.
req_data="$(sed -En 's%.*confirmation ID=\\"'$variable'-(.+)\\" status.*%\1%p' /tmp/log)"
I am trying to change the values in a text file using sed in a Bash script with the line,
sed 's/draw($prev_number;n_)/draw($number;n_)/g' file.txt > tmp
This will be in a for loop. Why is it not working?
Variables inside ' don't get substituted in Bash. To get string substitution (or interpolation, if you're familiar with Perl) you would need to change it to use double quotes " instead of the single quotes:
# Enclose the entire expression in double quotes
$ sed "s/draw($prev_number;n_)/draw($number;n_)/g" file.txt > tmp
# Or, concatenate strings with only variables inside double quotes
# This would restrict expansion to the relevant portion
# and prevent accidental expansion for !, backticks, etc.
$ sed 's/draw('"$prev_number"';n_)/draw('"$number"';n_)/g' file.txt > tmp
# A variable cannot contain arbitrary characters
# See link in the further reading section for details
$ a='foo
bar'
$ echo 'baz' | sed 's/baz/'"$a"'/g'
sed: -e expression #1, char 9: unterminated `s' command
Further Reading:
Difference between single and double quotes in Bash
Is it possible to escape regex metacharacters reliably with sed
Using different delimiters for sed substitute command
Unless you need it in a different file you can use the -i flag to change the file in place
Variables within single quotes are not expanded, but within double quotes they are. Use double quotes in this case.
sed "s/draw($prev_number;n_)/draw($number;n_)/g" file.txt > tmp
You could also make it work with eval, but don’t do that!!
This may help:
sed "s/draw($prev_number;n_)/draw($number;n_)/g"
You can use variables like below. Like here, I wanted to replace hostname i.e., a system variable in the file. I am looking for string look.me and replacing that whole line with look.me=<system_name>
sed -i "s/.*look.me.*/look.me=`hostname`/"
You can also store your system value in another variable and can use that variable for substitution.
host_var=`hostname`
sed -i "s/.*look.me.*/look.me=$host_var/"
Input file:
look.me=demonic
Output of file (assuming my system name is prod-cfm-frontend-1-usa-central-1):
look.me=prod-cfm-frontend-1-usa-central-1
I needed to input github tags from my release within github actions. So that on release it will automatically package up and push code to artifactory.
Here is how I did it. :)
- name: Invoke build
run: |
# Gets the Tag number from the release
TAGNUMBER=$(echo $GITHUB_REF | cut -d / -f 3)
# Setups a string to be used by sed
FINDANDREPLACE='s/${GITHUBACTIONSTAG}/'$(echo $TAGNUMBER)/
# Updates the setup.cfg file within version number
sed -i $FINDANDREPLACE setup.cfg
# Installs prerequisites and pushes
pip install -r requirements-dev.txt
invoke build
Retrospectively I wish I did this in python with tests. However it was fun todo some bash.
Another variant, using printf:
SED_EXPR="$(printf -- 's/draw(%s;n_)/draw(%s;n_)/g' $prev_number $number)"
sed "${SED_EXPR}" file.txt
or in one line:
sed "$(printf -- 's/draw(%s;n_)/draw(%s;n_)/g' $prev_number $number)" file.txt
Using printf to build the replacement expression should be safe against all kinds of weird things, which is why I like this variant.
I want to issue this command from the bash script
sed -e $beginning,$s/pattern/$variable/ file
but any possible combination of quotes gives me an error, only one that works:
sed -e "$beginning,$"'s/pattern/$variable/' file
also not good, because it do not dereferences the variable.
Does my approach can be implemented with sed?
Feel free to switch the quotes up. The shell can keep things straight.
sed -e "$beginning"',$s/pattern/'"$variable"'/' file
You can try this:
$ sed -e "$beginning,$ s/pattern/$variable/" file
Example
file.txt:
one
two
three
Try:
$ beginning=1
$ variable=ONE
$ sed -e "$beginning,$ s/one/$variable/" file.txt
Output:
ONE
two
three
There are two types of quotes:
Single quotes preserve their contents (> is the prompt):
> var=blah
> echo '$var'
$var
Double quotes allow for parameter expansion:
> var=blah
> echo "$var"
blah
And two types of $ sign:
One to tell the shell that what follows is the name of a parameter to be expanded
One that stands for "last line" in sed.
You have to combine these so
The shell doesn't think sed's $ has anything to do with a parameter
The shell parameters still get expanded (can't be within single quotes)
The whole sed command is quoted.
One possibility would be
sed "$beginning,\$s/pattern/$variable/" file
The whole command is in double quotes, i.e., parameters get expanded ($beginning and $variable). To make sure the shell doesn't try to expand $s, which doesn't exist, the "end of line" $ is escaped so the shell doesn't try anything funny.
Other options are
Double quoting everything but adding a space between $ and s (see Ren's answer)
Mixing quoting types as needed (see Ignacio's answer)
Methods that don't work
sed '$beginning,$s/pattern/$variable/' file
Everything in single quotes: the shell parameters are not expanded (doesn't follow rule 2 above). $beginning is not a valid address, and pattern would be literally replaced by $variable.
sed "$beginning,$s/pattern/$variable/" file
Everything in double qoutes: the parameters are expanded, including $s, which isn't supposed to (doesn't follow rule 1 above).
the following form worked for me from within script
sed $beg,$ -e s/pattern/$variable/ file
the same form will also work if executed from the shell
I have written a bash script which calls a sed command (amongst other things) on a file to complete a find/replace of 2 different strings.
The trouble is, after running the script, I check the files and nothing has been updated. However, if I run the commands that are being produced (I echo them as output anyway) then they work.
For example, inside the script I have:
echo "/usr/local/bin/sed -i -e 's/${String1}/${String1R}/g;s/\/${String2}\//\/${String2R}\//g' ${ROOT_DIR}/data/file.sql"
/usr/local/bin/sed -i -e 's/${String1}/${String1R}/g;s/\/${String2}\//\/${TString2R}\//g' ${ROOT_DIR}/data/file.sql
Running the script does not change file.sql; however, if I run the command that is printed to console e.g. /usr/local/bin/sed -i -e 's/file_name1/file_name2/g;s//path_substring1///path_substring2//g' /path/to/file/file.sql it works perfectly!
Use double quotes instead of single quotes. Single quotes would prevent variable expansion.
/usr/local/bin/sed -i -e "s/${String1}/${String1R}/g;s/\/${String2}\//\/${TString2R}\//g" ${ROOT_DIR}/data/file.sql
Moreover, it seems that your variables are path strings which might contain forward slashes, i.e. /. In that event use a different separator:
"s|${String1}|${String1R}|g"
Using a different separator would obviate the need of escaping / in the pattern and replacement.