Preserving whitespaces in a string as a command line argument - bash

I'm facing a small problem here, I want to pass a string containing whitespaces , to another program such that the whole string is treated as a command line argument.
In short I want to execute a command of the following structure through a bash shell script:
command_name -a arg1 -b arg2 -c "arg with whitespaces here"
But no matter how I try, the whitespaces are not preserved in the string, and is tokenized by default. A solution please,
edit: This is the main part of my script:
#!/bin/bash
#-------- BLACKRAY CONFIG ---------------#
# Make sure the current user is in the sudoers list
# Running all instances with sudo
BLACKRAY_BIN_PATH='/opt/blackray/bin'
BLACKRAY_LOADER_DEF_PATH='/home/crozzfire'
BLACKRAY_LOADER_DEF_NAME='load.xml'
BLACKRAY_CSV_PATH='/home/crozzfire'
BLACKRAY_END_POINT='default -p 8890'
OUT_FILE='/tmp/out.log'
echo "The current binary path is $BLACKRAY_BIN_PATH"
# Starting the blackray 0.9.0 server
sudo "$BLACKRAY_BIN_PATH/blackray_start"
# Starting the blackray loader utility
BLACKRAY_INDEX_CMD="$BLACKRAY_BIN_PATH/blackray_loader -c $BLACKRAY_LOADER_DEF_PATH/$BLACKRAY_LOADER_DEF_NAME -d $BLACKRAY_CSV_PATH -e "\"$BLACKRAY_END_POINT\"""
sudo time $BLACKRAY_INDEX_CMD -a $OUT_FILE
#--------- END BLACKRAY CONFIG ---------#

You're running into this problem because you store the command in a variable, then expand it later; unless there's a good reason to do this, don't:
sudo time $BLACKRAY_BIN_PATH/blackray_loader -c $BLACKRAY_LOADER_DEF_PATH/$BLACKRAY_LOADER_DEF_NAME -d $BLACKRAY_CSV_PATH -e "$BLACKRAY_END_POINT" -a $OUT_FILE
If you really do need to store the command and use it later, there are several options; the bash-hackers.org wiki has a good page on the subject. It looks to me like the most useful one here is to put the command in an array rather than a simple variable:
BLACKRAY_INDEX_CMD=($BLACKRAY_BIN_PATH/blackray_loader -c $BLACKRAY_LOADER_DEF_PATH/$BLACKRAY_LOADER_DEF_NAME -d $BLACKRAY_CSV_PATH -e "$BLACKRAY_END_POINT")
sudo time "${BLACKRAY_INDEX_CMD[#]}" -a $OUT_FILE
This avoids the whole confusion between spaces-separating-words and spaces-within-words because words aren't separated by spaces -- they're in separate elements of the array. Expanding the array in double-quotes with the [#] suffix preserves that structure.
(BTW, another option would be to use escaped quotes rather like you're doing, then run the command with eval. Don't do this; it's a good way to introduce weird parsing bugs.)

Edit:
Try:
BLACKRAY_END_POINT="'default -p 8890'"
or
BLACKRAY_END_POINT='"default -p 8890"'
or
BLACKRAY_END_POINT="default\ -p\ 8890"
or
BLACKRAY_END_POINT='default\ -p\ 8890'
and
BLACKRAY_INDEX_CMD="$BLACKRAY_BIN_PATH/blackray_loader -c $BLACKRAY_LOADER_DEF_PATH/$BLACKRAY_LOADER_DEF_NAME -d $BLACKRAY_CSV_PATH -e $BLACKRAY_END_POINT"
Original answer:
Is blackray_loader a shell script?
Here is a demonstration that you have to deal with this issue both when specifying the parameter and when handling it:
A text file called "test.txt" (include the line numbers):
1 two words
2 two words
3 two
4 words
A script called "spacetest":
#!/bin/bash
echo "No quotes in script"
echo $1
grep $1 test.txt
echo
echo "With quotes in script"
echo "$1"
grep "$1" test.txt
echo
Running it with ./spacetest "two--------words" (replace the hyphens with spaces):
No quotes in script
two words
grep: words: No such file or directory
test.txt:1 two words
test.txt:2 two words
test.txt:3 two
With quotes in script
two words
2 two words
You can see that in the "No quotes" section it tried to do grep two words test.txt which interpreted "words" as a filename in addition to "test.txt". Also, the echo dropped the extra spaces.
When the parameter is quoted, as in the second section, grep saw it as one argument (including the extra spaces) and handled it correctly. And echo preserved the extra spaces.
I used the extra spaces, by the way, merely to aid in the demonstration.

I have a suggestion:
# iterate through the passed arguments, save them to new properly quoted ARGS string
while [ -n "$1" ]; do
ARGS="$ARGS '$1'"
shift
done
# invoke the command with properly quoted arguments
my_command $ARGS

probably you need to surround the argument by double quotes (e.g. "${6}").
Following OP comment it should be "$BLACKRAY_END_POINT"

Below is my example of restarting a script via exec su USER or exec su - USER. It accommodates:
being called from a relative path or current working directory
spaces in script name and arguments
single and double-quotes in arguments, without crazy escapes like: \\"
#
# This script should always be run-as a specific user
#
user=jimbob
if [ $(whoami) != "$user" ]; then
exec su -c "'$(readlink -f "$0")' $(printf " %q" "$#")" - $user
exit $?
fi

A post on other blog saved me for this whitespaces problem: http://logbuffer.wordpress.com/2010/09/23/bash-scripting-preserve-whitespaces-in-variables/
By default, whitespaces are trimed:
bash> VAR1="abc def gh ijk"
bash> echo $VAR1
abc def gh ijk
bash>
"The cause of this behaviour is the internal shell variable $IFS (Internal Field Separator), that defaults to whitespace, tab and newline.
To preserve all contiguous whitespaces you have to set the IFS to something different"
With IFS bypass:
bash> IFS='%'
bash> echo $VAR1
abc def gh ijk
bash>unset IFS
bash>
It works wonderfully for my command case:
su - user1 -c 'test -r "'${filepath}'"; ....'
Hope this helps.

Related

Bash: expand variables, but not special characters

I have a bash script that creates and executes an expect script by stitching together dozens of different files containing pieces of expect code. Those files contain environment variables that need to be expanded. Example:
expect.piece:
send "command\r"
sleep $timeout
send "command argument\r"
script.sh:
#let's try it like this
eval echo $(cat expect.piece)
#or maybe like this
eval "echo \"$(cat expect.piece)\""
output:
send command\r sleep 1 send command argument\r
send commandr
sleep 1
send command argumentr
Desired otput:
send "command\r"
sleep 1
send "command argument\r"
I need a solution without sed string substitution (there is a lot of environment variables) and without modifying original expect script files. I guess it could be done line by line, but is there a more elegant solution?
Default field separator in bash is a space so set input file separator as a new line like IFS=$(echo -e '\n') before executing eval echo $(cat expect.piece) .
Final script would be :
#Storing original field separator in variable OFS
OFS=$IFS
#Setting IFS as new line using echo -e. Unfortunately IFS="\n" does not work in bash
IFS=$(echo -e '\n')
eval echo $(cat expect.piece)
#Resetting the field separator as space
IFS=$OFS
There is one another way which you can put your expect code with -c flag in the shell script as shown below.
script.sh
#Calling the expect.piece file code here
expect -c expect.piece
You can make use of the optional command line values as ,
expect -c "set count 1" myscript.exp
where the variable count will be used in the expect script file myscript.exp.
You can directly give the whole code as
expect -c "
send \"command\r\"
sleep $timeout
send \"command argument\r\"
"
Notice the use of backslash to escape the double quotes wherever needed. Single quotes can also be used. But, if you use double quotes, then only shell substitution can happen.
Please refer here to know more about the -c flag in expect.
It's not clear from your question in what context you want this output. If it's okay to embed the Expect script as a here document, what you want is trivial.
#!/bin/sh
timeout=1
cat <<____HERE
send "command\r"
sleep $timeout
send "command argument\r"
____HERE
(Maybe you can even replace the cat with expect but I'm not familiar enough with Expect to make any recommendations.)
If you need to take the input from a file, and only have a limited set of variables you want expanded, you could do that with sed.
sed "s/\$timeout/$timeout/g" file
If you need a more general solution, you might want to swich to Perl:
perl -pe 's/\$(\w+)/$ENV{$1} || "\$$1" /ge' file
but this requires you to export (or otherwise expose to Perl) the shell environment variables you want exported. (This will just not substitute any undefined variables; change the value after || if you want to change that aspect of the behavior.)
I invented the solution for this problem, it is a kludge, but it works.
expect.piece:
sleep $timeout
send "foo bar\r"
send "$(date)\r"
script.sh:
timeout=1
eval echo $(sed " \
s/\\\/\\\\\\\/g; \
s/\"/\\\\\"/g; \
s/\"/\\\\\`/\"/\\\\\`/g; \
" "expect.piece" | tr '\n' '+') | tr '+' '\n'
output:
sleep 1
send "foo bar\r"
send "Wed Feb 18 03:19:24 2015\r"
First, we need to escape all the backslashes, backticks and quotes in the file, because they will be removed during the evaluation. Then, we need to replace all the newline characters with pluses, in order to make it in a single line. After that, we evaluate that line (the evaluation will "expand" all the environment variables and command substitutions) and replace pluses back to newlines.

Bash tokenizing quoted string with spaces as individual words

I'm trying to execute the following commands:
mkdir 'my dir'
CMD="ls 'my dir'"
RESULT=$($CMD)
This results in:
ls: 'my: No such file or directory
ls: dir': No such file or directory
Using "set -x" before the second command reveals that the command that's actually being issued is:
++ ls ''\''my' 'dir'\'''
This is obviously abstracted from what I was actually trying to do; this code on its own doesn't serve any purpose. But my question is why does bash tokenize the quoted string like this and how can I make it stop?
(Almost) all languages differentiate between code and data:
args="1, 2"
myfunc(args) != myfunc(1, 2)
The same is true in bash. Putting single quotes in a literal string will not make bash interpret them.
The correct way of storing a program name and arguments (aka a simple command) is using an array:
cmd=(ls 'my dir')
result=$("${cmd[#]}")
Bash will automatically split words on spaces unless you double quote, which is why your example sometimes appears to work. This is surprising and error prone, and the reason why you should always double quote your variables unless you have a good reason not to.
It's also possible to get bash to interpret a string as if it was bash code entirely, using eval. This is frequently recommended, but almost always wrong.
To provide another approach -- one that works on POSIX systems -- xargs can do this parsing for you, as long as you can guarantee that the argument list is short enough that it doesn't get split into multiple separate commands:
CMD="ls 'my dir'"
printf '%s\n' "$CMD" | xargs sh -c '"$#"' sh
Mind, to do this securely (against an attacker who intentionally generates a string that goes over the maximum argv length to cause xargs to split it into multiple commands) you'd want to break out the first word of CMD to be something known/trusted, and parameterize only the following arguments. For example:
args="'my dir' 'other dir'"
printf '%s\n' "$args" | xargs sh -c 'ls "$#"' sh
...or simpler, at that point...
printf '%s\n' "$args" | xargs ls
Solution 1: Use sh -c
mkdir 'my dir'
CMD="ls 'my dir'" # careful: your example was missing a '
RESULT=$(sh -c "$CMD")
Solution 2: Declare CMD as an array
mkdir 'my dir'
CMD=(ls 'my dir') # array with 2 elements
RESULT=$("${CMD[#]}")

Capturing verbatim command line (including quotes!) to call inside script

I'm trying to write a "phone home" script, which will log the exact command line (including any single or double quotes used) into a MySQL database. As a backend, I have a cgi script which wraps the database. The scripts themselves call curl on the cgi script and include as parameters various arguments, including the verbatim command line.
Obviously I have quite a variety of quote escaping to do here and I'm already stuck at the bash stage. At the moment, I can't even get bash to print verbatim the arguments provided:
Desired output:
$ ./caller.sh -f -hello -q "blah"
-f hello -q "blah"
Using echo:
caller.sh:
echo "$#"
gives:
$ ./caller.sh -f -hello -q "blah"
-f hello -q blah
(I also tried echo $# and echo $*)
Using printf %q:
caller.sh:
printf %q $#
printf "\n"
gives:
$ ./caller.sh -f hello -q "blah"
-fhello-qblah
(I also tried print %q "$#")
I would welcome not only help to fix my bash problem, but any more general advice on implementing this "phone home" in a tidier way!
There is no possible way you can write caller.sh to distinguish between these two commands invoked on the shell:
./caller.sh -f -hello -q "blah"
./caller.sh -f -hello -q blah
There are exactly equivalent.
If you want to make sure the command receives special characters, surround the argument with single quotes:
./caller.sh -f -hello -q '"blah"'
Or if you want to pass just one argument to caller.sh:
./caller.sh '-f -hello -q "blah"'
You can get this info from the shell history:
function myhack {
line=$(history 1)
line=${line#* }
echo "You wrote: $line"
}
alias myhack='myhack #'
Which works as you describe:
$ myhack --args="stuff" * {1..10} $PATH
You wrote: myhack --args="stuff" * {1..10} $PATH
However, quoting is just the user's way of telling the shell how to construct the program's argument array. Asking to log how the user quotes their arguments is like asking to log how hard the user punched the keys and what they were wearing at the time.
To log a shell command line which unambiguously captures all of the arguments provided, you don't need any interactive shell hacks:
#!/bin/bash
line=$(printf "%q " "$#")
echo "What you wrote would have been indistinguishable from: $line"
I understand you want to capture the arguments given by the caller.
Firstly, quotes used by the caller are used to protect during the interpretation of the call. But they do not exist as argument.
An example: If someone call your script with one argument "Hello World!" with two spaces between Hello and World. Then you have to protect ALWAYS $1 in your script to not loose this information.
If you want to log all arguments correctly escaped (in the case where they contains, for example, consecutive spaces...) you HAVE to use "$#" with double quotes. "$#" is equivalent to "$1" "$2" "$3" "$4" etc.
So, to log arguments, I suggest the following at the start of the caller:
i=0
for arg in "$#"; do
echo "arg$i=$arg"
let ++i
done
## Example of calls to the previous script
#caller.sh '1' "2" 3 "4 4" "5 5"
#arg1=1
#arg2=2
#arg3=3
#arg4=4 4
#arg5=5 5
#Flimm is correct, there is no way to distinguish between arguments "foo" and foo, simply because the quotes are removed by the shell before the program receives them. What you need is "$#" (with the quotes).

All arguments into files with correct quoting using "$#"

I need my bashscript to cat all of its parameters into a file. I tried to use cat for this because I need to add a lot of lines:
#!/bin/sh
cat > /tmp/output << EOF
I was called with the following parameters:
"$#"
or
$#
EOF
cat /tmp/output
Which leads to the following output
$./test.sh "dsggdssgd" "dsggdssgd dgdsdsg"
I was called with the following parameters:
"dsggdssgd dsggdssgd dgdsdsg"
or
dsggdssgd dsggdssgd dgdsdsg
I want neither of these two things: I need the exact quoting which was used on the command line. How can I achieve this? I always thought $# does everything right in regards to quoting.
Well, you are right that "$#" has the args including the whitespace in each arg. However, since the shell performs quote removal before executing a command, you can never know how exactly the args were quoted (e.g. whether with single or double quotes, or backslashes or any combination thereof--but you shouldn't need to know, since all you should care for are the argument values).
Placing "$#" in a here-document is pointless because you lose the information about where each arg starts and ends (they're joined with a space inbetween). Here's a way to see just this:
$ cat test.sh
#!/bin/sh
printf 'I was called with the following parameters:\n'
printf '"%s"\n' "$#"
$ ./test.sh "dsggdssgd" "dsggdssgd dgdsdsg"
I was called with the following parameters:
"dsggdssgd"
"dsggdssgd dgdsdsg"
Try:
#!/bin/bash
for x in "$#"; do echo -ne "\"$x\" "; done; echo
To see what's interpreted by Bash, use:
bash -x ./script.sh
or add this to the beginning of your script:
set -x
You might want add this on the parent script.

Bash Templating: How to build configuration files from templates with Bash?

I'm writing a script to automate creating configuration files for Apache and PHP for my own webserver. I don't want to use any GUIs like CPanel or ISPConfig.
I have some templates of Apache and PHP configuration files. Bash script needs to read templates, make variable substitution and output parsed templates into some folder. What is the best way to do that? I can think of several ways. Which one is the best or may be there are some better ways to do that? I want to do that in pure Bash (it's easy in PHP for example)
How to replace ${} placeholders in a text file?
template.txt:
The number is ${i}
The word is ${word}
script.sh:
#!/bin/sh
#set variables
i=1
word="dog"
#read in template one line at the time, and replace variables
#(more natural (and efficient) way, thanks to Jonathan Leffler)
while read line
do
eval echo "$line"
done < "./template.txt"
BTW, how do I redirect output to external file here? Do I need to escape something if variables contain, say, quotes?
Using cat & sed for replacing each variable with its value:
Given template.txt (see above)
Command:
cat template.txt | sed -e "s/\${i}/1/" | sed -e "s/\${word}/dog/"
Seems bad to me because of the need to escape many different symbols and with many variables the line will be tooooo long.
Can you think of some other elegant and safe solution?
Try envsubst
$ cat envsubst-template.txt
Variable FOO is (${FOO}).
Variable BAR is (${BAR}).
$ FOO=myfoo
$ BAR=mybar
$ export FOO BAR
$ cat envsubst-template.txt | envsubst
Variable FOO is (myfoo).
Variable BAR is (mybar).
A heredoc is a builtin way to template a conf file.
STATUS_URI="/hows-it-goin"; MONITOR_IP="10.10.2.15";
cat >/etc/apache2/conf.d/mod_status.conf <<EOF
<Location ${STATUS_URI}>
SetHandler server-status
Order deny,allow
Deny from all
Allow from ${MONITOR_IP}
</Location>
EOF
Regarding yottsa's answer: envsubst was new to me. Fantastic.
You can use this:
perl -p -i -e 's/\$\{([^}]+)\}/defined $ENV{$1} ? $ENV{$1} : $&/eg' < template.txt
to replace all ${...} strings with corresponding enviroment variables (do not forget to export them before running this script).
For pure bash this should work (assuming that variables do not contain ${...} strings):
#!/bin/bash
while read -r line ; do
while [[ "$line" =~ (\$\{[a-zA-Z_][a-zA-Z_0-9]*\}) ]] ; do
LHS=${BASH_REMATCH[1]}
RHS="$(eval echo "\"$LHS\"")"
line=${line//$LHS/$RHS}
done
echo "$line"
done
. Solution that does not hang if RHS references some variable that references itself:
#!/bin/bash
line="$(cat; echo -n a)"
end_offset=${#line}
while [[ "${line:0:$end_offset}" =~ (.*)(\$\{([a-zA-Z_][a-zA-Z_0-9]*)\})(.*) ]] ; do
PRE="${BASH_REMATCH[1]}"
POST="${BASH_REMATCH[4]}${line:$end_offset:${#line}}"
VARNAME="${BASH_REMATCH[3]}"
eval 'VARVAL="$'$VARNAME'"'
line="$PRE$VARVAL$POST"
end_offset=${#PRE}
done
echo -n "${line:0:-1}"
WARNING: I do not know a way to correctly handle input with NULs in bash or preserve the amount of trailing newlines. Last variant is presented as it is because shells “love” binary input:
read will interpret backslashes.
read -r will not interpret backslashes, but still will drop the last line if it does not end with a newline.
"$(…)" will strip as many trailing newlines as there are present, so I end … with ; echo -n a and use echo -n "${line:0:-1}": this drops the last character (which is a) and preserves as many trailing newlines as there was in the input (including no).
I agree with using sed: it is the best tool for search/replace. Here is my approach:
$ cat template.txt
the number is ${i}
the dog's name is ${name}
$ cat replace.sed
s/${i}/5/
s/${name}/Fido/
$ sed -f replace.sed template.txt > out.txt
$ cat out.txt
the number is 5
the dog's name is Fido
I have a bash solution like mogsie but with heredoc instead of herestring to allow you to avoid escaping double quotes
eval "cat <<EOF
$(<template.txt)
EOF
" 2> /dev/null
Try eval
I think eval works really well. It handles templates with linebreaks, whitespace, and all sorts of bash stuff. If you have full control over the templates themselves of course:
$ cat template.txt
variable1 = ${variable1}
variable2 = $variable2
my-ip = \"$(curl -s ifconfig.me)\"
$ echo $variable1
AAA
$ echo $variable2
BBB
$ eval "echo \"$(<template.txt)\"" 2> /dev/null
variable1 = AAA
variable2 = BBB
my-ip = "11.22.33.44"
This method should be used with care, of course, since eval can execute arbitrary code. Running this as root is pretty much out of the question. Quotes in the template need to be escaped, otherwise they will be eaten by eval.
You can also use here documents if you prefer cat to echo
$ eval "cat <<< \"$(<template.txt)\"" 2> /dev/null
#plockc provoded a solution that avoids the bash quote escaping issue:
$ eval "cat <<EOF
$(<template.txt)
EOF
" 2> /dev/null
Edit: Removed part about running this as root using sudo...
Edit: Added comment about how quotes need to be escaped, added plockc's solution to the mix!
Edit Jan 6, 2017
I needed to keep double quotes in my configuration file so double escaping double quotes with sed helps:
render_template() {
eval "echo \"$(sed 's/\"/\\\\"/g' $1)\""
}
I can't think of keeping trailing new lines, but empty lines in between are kept.
Although it is an old topic, IMO I found out more elegant solution here: http://pempek.net/articles/2013/07/08/bash-sh-as-template-engine/
#!/bin/sh
# render a template configuration file
# expand variables + preserve formatting
render_template() {
eval "echo \"$(cat $1)\""
}
user="Gregory"
render_template /path/to/template.txt > path/to/configuration_file
All credits to Grégory Pakosz.
Instead of reinventing the wheel go with envsubst
Can be used in almost any scenario, for instance building configuration files from environment variables in docker containers.
If on mac make sure you have homebrew then link it from gettext:
brew install gettext
brew link --force gettext
./template.cfg
# We put env variables into placeholders here
this_variable_1 = ${SOME_VARIABLE_1}
this_variable_2 = ${SOME_VARIABLE_2}
./.env:
SOME_VARIABLE_1=value_1
SOME_VARIABLE_2=value_2
./configure.sh
#!/bin/bash
cat template.cfg | envsubst > whatever.cfg
Now just use it:
# make script executable
chmod +x ./configure.sh
# source your variables
. .env
# export your variables
# In practice you may not have to manually export variables
# if your solution depends on tools that utilise .env file
# automatically like pipenv etc.
export SOME_VARIABLE_1 SOME_VARIABLE_2
# Create your config file
./configure.sh
I'd have done it this way, probably less efficient, but easier to read/maintain.
TEMPLATE='/path/to/template.file'
OUTPUT='/path/to/output.file'
while read LINE; do
echo $LINE |
sed 's/VARONE/NEWVALA/g' |
sed 's/VARTWO/NEWVALB/g' |
sed 's/VARTHR/NEWVALC/g' >> $OUTPUT
done < $TEMPLATE
If you want to use Jinja2 templates, see this project: j2cli.
It supports:
Templates from JSON, INI, YAML files and input streams
Templating from environment variables
A longer but more robust version of the accepted answer:
perl -pe 's;(\\*)(\$([a-zA-Z_][a-zA-Z_0-9]*)|\$\{([a-zA-Z_][a-zA-Z_0-9]*)\})?;substr($1,0,int(length($1)/2)).($2&&length($1)%2?$2:$ENV{$3||$4});eg' template.txt
This expands all instances of $VAR or ${VAR} to their environment values (or, if they're undefined, the empty string).
It properly escapes backslashes, and accepts a backslash-escaped $ to inhibit substitution (unlike envsubst, which, it turns out, doesn't do this).
So, if your environment is:
FOO=bar
BAZ=kenny
TARGET=backslashes
NOPE=engi
and your template is:
Two ${TARGET} walk into a \\$FOO. \\\\
\\\$FOO says, "Delete C:\\Windows\\System32, it's a virus."
$BAZ replies, "\${NOPE}s."
the result would be:
Two backslashes walk into a \bar. \\
\$FOO says, "Delete C:\Windows\System32, it's a virus."
kenny replies, "${NOPE}s."
If you only want to escape backslashes before $ (you could write "C:\Windows\System32" in a template unchanged), use this slightly-modified version:
perl -pe 's;(\\*)(\$([a-zA-Z_][a-zA-Z_0-9]*)|\$\{([a-zA-Z_][a-zA-Z_0-9]*)\});substr($1,0,int(length($1)/2)).(length($1)%2?$2:$ENV{$3||$4});eg' template.txt
Here's another pure bash solution:
it's using heredoc, so:
complexity doesn't increase because of additionaly required syntax
template can include bash code
that also allows you to indent stuff properly. See below.
it doesn't use eval, so:
no problems with the rendering of trailing empty lines
no problems with quotes in the template
$ cat code
#!/bin/bash
LISTING=$( ls )
cat_template() {
echo "cat << EOT"
cat "$1"
echo EOT
}
cat_template template | LISTING="$LISTING" bash
Input:
$ cat template (with trailing newlines and double quotes)
<html>
<head>
</head>
<body>
<p>"directory listing"
<pre>
$( echo "$LISTING" | sed 's/^/ /' )
<pre>
</p>
</body>
</html>
Output:
<html>
<head>
</head>
<body>
<p>"directory listing"
<pre>
code
template
<pre>
</p>
</body>
</html>
Here is another solution: generate a bash script with all the variables and the contents of the template file, that script would look like this:
word=dog
i=1
cat << EOF
the number is ${i}
the word is ${word}
EOF
If we feed this script into bash it would produce the desired output:
the number is 1
the word is dog
Here is how to generate that script and feed that script into bash:
(
# Variables
echo word=dog
echo i=1
# add the template
echo "cat << EOF"
cat template.txt
echo EOF
) | bash
Discussion
The parentheses opens a sub shell, its purpose is to group together all the output generated
Within the sub shell, we generate all the variable declarations
Also in the sub shell, we generate the cat command with HEREDOC
Finally, we feed the sub shell output to bash and produce the desired output
If you want to redirect this output into a file, replace the last line with:
) | bash > output.txt
Taking the answer from ZyX using pure bash but with new style regex matching and indirect parameter substitution it becomes:
#!/bin/bash
regex='\$\{([a-zA-Z_][a-zA-Z_0-9]*)\}'
while read line; do
while [[ "$line" =~ $regex ]]; do
param="${BASH_REMATCH[1]}"
line=${line//${BASH_REMATCH[0]}/${!param}}
done
echo $line
done
If using Perl is an option and you're content with basing expansions on environment variables only (as opposed to all shell variables), consider Stuart P. Bentley's robust answer.
This answer aims to provide a bash-only solution that - despite use of eval - should be safe to use.
The goals are:
Support expansion of both ${name} and $name variable references.
Prevent all other expansions:
command substitutions ($(...) and legacy syntax `...`)
arithmetic substitutions ($((...)) and legacy syntax $[...]).
Allow selective suppression of variable expansion by prefixing with \ (\${name}).
Preserve special chars. in the input, notably " and \ instances.
Allow input either via arguments or via stdin.
Function expandVars():
expandVars() {
local txtToEval=$* txtToEvalEscaped
# If no arguments were passed, process stdin input.
(( $# == 0 )) && IFS= read -r -d '' txtToEval
# Disable command substitutions and arithmetic expansions to prevent execution
# of arbitrary commands.
# Note that selectively allowing $((...)) or $[...] to enable arithmetic
# expressions is NOT safe, because command substitutions could be embedded in them.
# If you fully trust or control the input, you can remove the `tr` calls below
IFS= read -r -d '' txtToEvalEscaped < <(printf %s "$txtToEval" | tr '`([' '\1\2\3')
# Pass the string to `eval`, escaping embedded double quotes first.
# `printf %s` ensures that the string is printed without interpretation
# (after processing by by bash).
# The `tr` command reconverts the previously escaped chars. back to their
# literal original.
eval printf %s "\"${txtToEvalEscaped//\"/\\\"}\"" | tr '\1\2\3' '`(['
}
Examples:
$ expandVars '\$HOME="$HOME"; `date` and $(ls)'
$HOME="/home/jdoe"; `date` and $(ls) # only $HOME was expanded
$ printf '\$SHELL=${SHELL}, but "$(( 1 \ 2 ))" will not expand' | expandVars
$SHELL=/bin/bash, but "$(( 1 \ 2 ))" will not expand # only ${SHELL} was expanded
For performance reasons, the function reads stdin input all at once into memory, but it's easy to adapt the function to a line-by-line approach.
Also supports non-basic variable expansions such as ${HOME:0:10}, as long as they contain no embedded command or arithmetic substitutions, such as ${HOME:0:$(echo 10)}
Such embedded substitutions actually BREAK the function (because all $( and ` instances are blindly escaped).
Similarly, malformed variable references such as ${HOME (missing closing }) BREAK the function.
Due to bash's handling of double-quoted strings, backslashes are handled as follows:
\$name prevents expansion.
A single \ not followed by $ is preserved as is.
If you want to represent multiple adjacent \ instances, you must double them; e.g.:
\\ -> \ - the same as just \
\\\\ -> \\
The input mustn't contain the following (rarely used) characters, which are used for internal purposes: 0x1, 0x2, 0x3.
There's a largely hypothetical concern that if bash should introduce new expansion syntax, this function might not prevent such expansions - see below for a solution that doesn't use eval.
If you're looking for a more restrictive solution that only supports ${name} expansions - i.e., with mandatory curly braces, ignoring $name references - see this answer of mine.
Here is an improved version of the bash-only, eval-free solution from the accepted answer:
The improvements are:
Support for expansion of both ${name} and $name variable references.
Support for \-escaping variable references that shouldn't be expanded.
Unlike the eval-based solution above,
non-basic expansions are ignored
malformed variable references are ignored (they don't break the script)
IFS= read -d '' -r lines # read all input from stdin at once
end_offset=${#lines}
while [[ "${lines:0:end_offset}" =~ (.*)\$(\{([a-zA-Z_][a-zA-Z_0-9]*)\}|([a-zA-Z_][a-zA-Z_0-9]*))(.*) ]] ; do
pre=${BASH_REMATCH[1]} # everything before the var. reference
post=${BASH_REMATCH[5]}${lines:end_offset} # everything after
# extract the var. name; it's in the 3rd capture group, if the name is enclosed in {...}, and the 4th otherwise
[[ -n ${BASH_REMATCH[3]} ]] && varName=${BASH_REMATCH[3]} || varName=${BASH_REMATCH[4]}
# Is the var ref. escaped, i.e., prefixed with an odd number of backslashes?
if [[ $pre =~ \\+$ ]] && (( ${#BASH_REMATCH} % 2 )); then
: # no change to $lines, leave escaped var. ref. untouched
else # replace the variable reference with the variable's value using indirect expansion
lines=${pre}${!varName}${post}
fi
end_offset=${#pre}
done
printf %s "$lines"
To follow up on plockc's answer on this page, here is a dash-suitable version, for those of you looking to avoid bashisms.
eval "cat <<EOF >outputfile
$( cat template.in )
EOF
" 2> /dev/null
Try shtpl
Perfect case for shtpl. (project of mine, so it is not widely in use and lacks in documentation. But here is the solution it offers anyhow. May you want to test it.)
Just execute:
$ i=1 word=dog sh -c "$( shtpl template.txt )"
Result is:
the number is 1
the word is dog
Have fun.
This page describes an answer with awk
awk '{while(match($0,"[$]{[^}]*}")) {var=substr($0,RSTART+2,RLENGTH -3);gsub("[$]{"var"}",ENVIRON[var])}}1' < input.txt > output.txt
# Usage: template your_file.conf.template > your_file.conf
template() {
local IFS line
while IFS=$'\n\r' read -r line ; do
line=${line//\\/\\\\} # escape backslashes
line=${line//\"/\\\"} # escape "
line=${line//\`/\\\`} # escape `
line=${line//\$/\\\$} # escape $
line=${line//\\\${/\${} # de-escape ${ - allows variable substitution: ${var} ${var:-default_value} etc
# to allow arithmetic expansion or command substitution uncomment one of following lines:
# line=${line//\\\$\(/\$\(} # de-escape $( and $(( - allows $(( 1 + 2 )) or $( command ) - UNSECURE
# line=${line//\\\$\(\(/\$\(\(} # de-escape $(( - allows $(( 1 + 2 ))
eval "echo \"${line}\"";
done < "$1"
}
This is the pure bash function adjustable to your liking, used in production and should not break on any input.
If it breaks - let me know.
You can also use bashible (which internally uses the evaluating approach described above/below).
There is an example, how to generate a HTML from multiple parts:
https://github.com/mig1984/bashible/tree/master/examples/templates
Look at simple variables substitution python script here: https://github.com/jeckep/vsubst
It is very simple to use:
python subst.py --props secure.properties --src_path ./templates --dst_path ./dist
Here's a bash function that preserves whitespace:
# Render a file in bash, i.e. expand environment variables. Preserves whitespace.
function render_file () {
while IFS='' read line; do
eval echo \""${line}"\"
done < "${1}"
}
Here's a modified perl script based on a few of the other answers:
perl -pe 's/([^\\]|^)\$\{([a-zA-Z_][a-zA-Z_0-9]*)\}/$1.$ENV{$2}/eg' -i template
Features (based on my needs, but should be easy to modify):
Skips escaped parameter expansions (e.g. \${VAR}).
Supports parameter expansions of the form ${VAR}, but not $VAR.
Replaces ${VAR} with a blank string if there is no VAR envar.
Only supports a-z, A-Z, 0-9 and underscore characters in the name (excluding digits in the first position).
You can also use printf to fill a template.
#!/bin/bash
IFS='' read -rd '' TEMPL <<-'EOB'
The number is %d
The word is "%s"
Birds of Massachusetts:
%s
EOB
N=12
WORD="Bird"
MULTILINE="Eastern Bluebirds
Common Grackles"
echo "START"
printf "${TEMPL}" ${N} ${WORD} "${MULTILINE}"
echo "END"
Here's the output, with quotes and whitespace intact:
START
The number is 12
The word is "Bird"
Birds of Massachusetts:
Eastern Bluebirds
Common Grackles
END

Resources