In my program I need to know the maximum number of process I can run. So I write a script. It works when I run it in shell but but when in program using system("./limit.sh"). I work in bash.
Here is my code:
#/bin/bash
LIMIT=\`ulimit -u\`
ACTIVE=\`ps -u | wc -l \`
echo $LIMIT > limit.txt
echo $ACTIVE >> limit.txt
Anyone can help?
Why The Original Fails
Command substitution syntax doesn't work if escaped. When you run:
LIMIT=\`ulimit -u\`
...what you're doing is running a command named
-u`
...with the environment variable named LIMIT containing the value
`ulimit
...and unless you actually have a command that starts with -u and contains a backtick in its name, this can be expected to fail.
This is because using backticks makes characters which would otherwise be syntax into literals, and running a command with one or more var=value pairs preceding it treats those pairs as variables to export in the environment for the duration of that single command.
Doing It Better
#!/bin/bash
limit=$(ulimit -u)
active=$(ps -u | wc -l)
printf '%s\n' "$limit" "$active" >limit.txt
Leave off the backticks.
Use modern $() command substitution syntax.
Avoid multiple redirections.
Avoid all-caps names for your own variables (these names are used for variables with meaning to the OS or system; lowercase names are reserved for application use).
Doing It Right
#!/bin/bash
exec >limit.txt # open limit.txt as output for the rest of the script
ulimit -u # run ulimit -u, inheriting that FD for output
ps -u | wc -l # run your pipeline, likewise with output to the existing FD
You have a typo on the very first line: #/bin/bash should be #!/bin/bash - this is often known as a "shebang" line, for "hash" (#) + "bang" (!)
Without that syntax written correctly, the script is run through the system's default shell, which will see that line as just a comment.
As pointed out in comments, that also means only the standardised options available to the builtin ulimit command, which doesn't include -u.
I am trying to write an fgrep statement removing records with a full record match from a file. I can do this on the command line, but not inside a ksh script. The code I am using boils down to these 4 lines of code:
Header='abc def|ghi jkl' #I use the head command to populate this variable
workfile=abc.txt
command="fgrep -Fxv \'$Header\' $workfile" >$outfile
$command
When I echo $command to STDIN the command is exactly what I would type on the command line (with the single quotes) and that works on the command line. When I execute it within the ksh script (file) the single quotes seem not to be recognized because the errors show it is parsing on spaces.
I have tried back ticks, exec, eval, double quotes instead of single quotes, and not using the $command variable. The problem remains.
I can do this on the command line, but not inside a ksh script
Here's a simple, portable, reliable solution using a heredoc.
#!/usr/bin/env ksh
workfile=abc.txt
outfile=out.txt
IFS= read -r Header <<'EOF'
abc def|ghi jul
EOF
IFS= read -r command <<'EOF'
grep -Fxv "$Header" "$workfile" > "$outfile"
EOF
eval "$command"
Explanation :
(Comments can't be added to the script above because they would affect the lines in the heredoc)
IFS= read -r Header <<'EOF' # Line separated literal strings
abc def|ghi jul # Set into the $Header variable
EOF # As if it were a text file
IFS= read -r command <<'EOF' # Command to execute
grep -Fxv "$Header" "$workfile" > "$outfile" # As if it were typed into
EOF # the shell command line
eval "$command" # Execute the command
The above example is the same as having a text file called header.txt, which contains the contents: abc def|ghi jul and typing the following command:
grep -Fxvf header.txt abc.txt
The heredoc addresses the problem of the script operating differently than the command line as a result of quoting/expansions/escaping issues.
A Word of caution regarding eval:
The use of eval in this example is specific. Please see Eval command and security issues for information on how eval can be misused and cause potentially very damaging results.
More Detail / Alternate Example:
For the sake of completeness, clarity, and ability to apply this concept to other situations, some notes about the heredoc and an alternative demonstration:
This implementation of the heredoc in this example is specifically designed with the following criteria:
Literal string assignment of contents, to the variables (using 'EOF')
Use of the eval command to evaluate and execute the referenced variables within the heredoc itself.
File or heredoc ?
One strength of using a heredoc combined with grep -F (fgrep), is the ability to treat a section of the script as if it were a file.
Case for file:
You want to frequently paste "pattern" lines into the file, and remove them as necessary, without having to modify the script file.
Case for heredoc:
You apply the script in an environment where specific files already exist, and you want to match specific exact literal patterns against it.
Example:
Scenario: I have 5 VPS Servers, and I want a script to produce a new fstab file but to ensure it doesn't contain the exact line:
/dev/xvda1 / ext3 errors=remount-ro,noatime,barrier=0 0 1
This scenario fits the type of situation addressed in this question. I could use the boilerplate from the above code in this answer and modify it as following:
#!/usr/bin/env ksh
workfile=/etc/fstab
IFS= read -r Header <<'EOF'
/dev/xvda1 / ext3 errors=remount-ro,noatime,barrier=0 0 1
EOF
IFS= read -r command <<'EOF'
grep -Fxv "$Header" "$workfile"
EOF
eval "$command"
This would give me a new fstab file, without the line contained in the heredoc.
Bash FAQ #50: I'm trying to put a command in a variable, but the complex cases always fail! provides comprehensive guidance - while it is written for Bash, most of it applies to Ksh as well.[1]
If you want to stick with storing your command in a variable (defining a function is the better choice), use an array, which bypasses the quoting issues:
#!/usr/bin/env ksh
Header='abc def|ghi jkl'
workfile=abc.txt
# Store command and arguments as elements of an array
command=( 'fgrep' '-Fxv' "$Header" "$workfile" )
# Invoke the array as a command.
"${command[#]}" > "$outfile"
Note: only a simple command can be stored in an array, and redirections can't be part of it.
[1] The function examples use local to create local variables, which ksh doesn't support. Omit local to make do with shell-global variables instead, or use function <name> {...} syntax with typeset instead of local to declare local variables in ksh.
To get started, here's the script I'm running to get the offending string:
# sed finds all sourced file paths from inputted file.
#
# while reads each match output from sed to $SOURCEFILE variable.
# Each should be a file path, or a variable that represents a file path.
# Any variables found should be expanded to the full path.
#
# echo and calls are used for demonstractive purposes only
# I intend to do something else with the path once it's expanded.
PATH_SOME_SCRIPT="/path/to/bash/script"
while read -r SOURCEFILE; do
echo "$SOURCEFILE"
"$SOURCEFILE"
$SOURCEFILE
done < <(cat $PATH_SOME_SCRIPT | sed -n -e "s/^\(source\|\.\|\$include\) //p")
You may also wish to use the following to test this out as mock data:
[ /path/to/bash/script ]
#!/bin/bash
source "$HOME/bash_file"
source "$GLOBAL_VAR_SCRIPT_PATH"
echo "No cow powers here"
For the tl;dr crew, basically the while loop spits out the following on the mock data:
"$HOME/bash_file"
bash: "$HOME/bash_file": no such file or directory
bash: "$HOME/bash_file": no such file or directory
"$GLOBAL_VAR_SCRIPT_PATH"
"$GLOBAL_VAR_SCRIPT_PATH": command not found
"$GLOBAL_VAR_SCRIPT_PATH": command not found
My question is, can you get the variable to expand correctly, e.g., print "/home//bash_file" and "/expanded/variable/path"? I should also state that although eval works I do not intend to use it because of its potential insecurities.
Protip that any variable value used in cat | sed would be available globally, including to the calling script, so it's not because the script cannot call the variable value.
FIRST SOLUTION ATTEMPT
Using anubhava's envsubst solution:
SOMEVARIABLE="/home/nick/.some_path"
while read -r SOURCEFILE; do
echo "$SOURCEFILE"
envsubst <<< "$SOURCEFILE";
done < <(echo -e "\"\$SOMEVARIABLE\"\n\"$HOME/.another_file\"")
This outputs the following:
"$SOMEVARIABLE"
""
"/home/nick/.another_file"
"/home/nick/.another_file"
Unfortunately, it does not expand the variable! Oh dear :(
SECOND SOLUTION ATTEMPT
Based upon the first attempt:
export SOMEVARIABLE="/home/nick/.some_path"
while read -r SOURCEFILE; do
echo "$SOURCEFILE"
envsubst <<< "$SOURCEFILE";
done < <(echo -e "\"\$SOMEVARIABLE\"\n\"$HOME/.another_file\"")
unset SOMEVARIABLE
which produces the results we wanted without eval and without messing with global variables (for too long anyway), hoorah!
Good runner-ups were further suggested using eval (although potentially unsafe) which can be found in this answer and here (link courtesy of anubhava's extended comments).
My question is, can you get the variable to expand correctly, e.g., print "/home//bash_file" and "/expanded/variable/path"?
Yes you can use envsubst program, that substitutes the values of environment variables:
while read -r sourceFile; do
envsubst <<< "$sourceFile"
done < <(sed -n "s/^\(source\|\.\|\$include\) //p" "$PATH_SOME_SCRIPT")
I think you are asking how to recursively expand variables in bash. Try
expanded=$(eval echo $SOURCEFILE)
inside your loop. eval runs the expanded command you give it. Since $SOURCEFILE isn't in quotes, it will be expanded to, e.g., $HOME/whatever. Then the eval will expand the $HOME before passing it to echo. echo will print the result, and expanded=$(...) will put the printed result in $expanded.
I can redirect the output and then cat the file and grep/awk the variable, but I would like to use this file for multiple variables.
So If it was one variable say STATUS then i could do some thing like
echo "STATUS $STATUS" >> variable.file
#later perhaps in a remote shell where varible.file was copied
NEW_VAR=`cat variable.file | awk print '{$2}'`
I guess some inline editing with sed would help. The smaller the code the better.
One common way of storing variables in a file is to just store NAME=value lines in the file, and then just source that in to the shell you want to pick up the variables.
echo 'STATUS="'"$STATUS"'"' >> variable.file
# later
. variable.file
In Bash, you can also use source instead of ., though this may not be portable to other shells. Note carefully the exact sequence of quotes necessary to get the correct double quotes printed out in the file.
If you want to put multiple variables at once into the file, you could do the following. Apologies for the quoting contortions that this takes to do properly and portably; if you restrict yourself to Bash, you can use $"" to make the quoting a little simpler:
for var in STATUS FOO BAR
do
echo "$var="'"'"$(eval echo '$'"$var")"'"'
done >> variable.file
The declare builtin is useful here
for var in STATUS FOO BAR; do
declare -p $var | cut -d ' ' -f 3- >> filename
done
As Brian says, later you can source filename
declare is great because it handles quoting for you:
$ FOO='"I'"'"'m here," she said.'
$ declare -p FOO
declare -- FOO="\"I'm here,\" she said."
$ declare -p FOO | cut -d " " -f 3-
FOO="\"I'm here,\" she said."
I'm writing a script to automate creating configuration files for Apache and PHP for my own webserver. I don't want to use any GUIs like CPanel or ISPConfig.
I have some templates of Apache and PHP configuration files. Bash script needs to read templates, make variable substitution and output parsed templates into some folder. What is the best way to do that? I can think of several ways. Which one is the best or may be there are some better ways to do that? I want to do that in pure Bash (it's easy in PHP for example)
How to replace ${} placeholders in a text file?
template.txt:
The number is ${i}
The word is ${word}
script.sh:
#!/bin/sh
#set variables
i=1
word="dog"
#read in template one line at the time, and replace variables
#(more natural (and efficient) way, thanks to Jonathan Leffler)
while read line
do
eval echo "$line"
done < "./template.txt"
BTW, how do I redirect output to external file here? Do I need to escape something if variables contain, say, quotes?
Using cat & sed for replacing each variable with its value:
Given template.txt (see above)
Command:
cat template.txt | sed -e "s/\${i}/1/" | sed -e "s/\${word}/dog/"
Seems bad to me because of the need to escape many different symbols and with many variables the line will be tooooo long.
Can you think of some other elegant and safe solution?
Try envsubst
$ cat envsubst-template.txt
Variable FOO is (${FOO}).
Variable BAR is (${BAR}).
$ FOO=myfoo
$ BAR=mybar
$ export FOO BAR
$ cat envsubst-template.txt | envsubst
Variable FOO is (myfoo).
Variable BAR is (mybar).
A heredoc is a builtin way to template a conf file.
STATUS_URI="/hows-it-goin"; MONITOR_IP="10.10.2.15";
cat >/etc/apache2/conf.d/mod_status.conf <<EOF
<Location ${STATUS_URI}>
SetHandler server-status
Order deny,allow
Deny from all
Allow from ${MONITOR_IP}
</Location>
EOF
Regarding yottsa's answer: envsubst was new to me. Fantastic.
You can use this:
perl -p -i -e 's/\$\{([^}]+)\}/defined $ENV{$1} ? $ENV{$1} : $&/eg' < template.txt
to replace all ${...} strings with corresponding enviroment variables (do not forget to export them before running this script).
For pure bash this should work (assuming that variables do not contain ${...} strings):
#!/bin/bash
while read -r line ; do
while [[ "$line" =~ (\$\{[a-zA-Z_][a-zA-Z_0-9]*\}) ]] ; do
LHS=${BASH_REMATCH[1]}
RHS="$(eval echo "\"$LHS\"")"
line=${line//$LHS/$RHS}
done
echo "$line"
done
. Solution that does not hang if RHS references some variable that references itself:
#!/bin/bash
line="$(cat; echo -n a)"
end_offset=${#line}
while [[ "${line:0:$end_offset}" =~ (.*)(\$\{([a-zA-Z_][a-zA-Z_0-9]*)\})(.*) ]] ; do
PRE="${BASH_REMATCH[1]}"
POST="${BASH_REMATCH[4]}${line:$end_offset:${#line}}"
VARNAME="${BASH_REMATCH[3]}"
eval 'VARVAL="$'$VARNAME'"'
line="$PRE$VARVAL$POST"
end_offset=${#PRE}
done
echo -n "${line:0:-1}"
WARNING: I do not know a way to correctly handle input with NULs in bash or preserve the amount of trailing newlines. Last variant is presented as it is because shells “love” binary input:
read will interpret backslashes.
read -r will not interpret backslashes, but still will drop the last line if it does not end with a newline.
"$(…)" will strip as many trailing newlines as there are present, so I end … with ; echo -n a and use echo -n "${line:0:-1}": this drops the last character (which is a) and preserves as many trailing newlines as there was in the input (including no).
I agree with using sed: it is the best tool for search/replace. Here is my approach:
$ cat template.txt
the number is ${i}
the dog's name is ${name}
$ cat replace.sed
s/${i}/5/
s/${name}/Fido/
$ sed -f replace.sed template.txt > out.txt
$ cat out.txt
the number is 5
the dog's name is Fido
I have a bash solution like mogsie but with heredoc instead of herestring to allow you to avoid escaping double quotes
eval "cat <<EOF
$(<template.txt)
EOF
" 2> /dev/null
Try eval
I think eval works really well. It handles templates with linebreaks, whitespace, and all sorts of bash stuff. If you have full control over the templates themselves of course:
$ cat template.txt
variable1 = ${variable1}
variable2 = $variable2
my-ip = \"$(curl -s ifconfig.me)\"
$ echo $variable1
AAA
$ echo $variable2
BBB
$ eval "echo \"$(<template.txt)\"" 2> /dev/null
variable1 = AAA
variable2 = BBB
my-ip = "11.22.33.44"
This method should be used with care, of course, since eval can execute arbitrary code. Running this as root is pretty much out of the question. Quotes in the template need to be escaped, otherwise they will be eaten by eval.
You can also use here documents if you prefer cat to echo
$ eval "cat <<< \"$(<template.txt)\"" 2> /dev/null
#plockc provoded a solution that avoids the bash quote escaping issue:
$ eval "cat <<EOF
$(<template.txt)
EOF
" 2> /dev/null
Edit: Removed part about running this as root using sudo...
Edit: Added comment about how quotes need to be escaped, added plockc's solution to the mix!
Edit Jan 6, 2017
I needed to keep double quotes in my configuration file so double escaping double quotes with sed helps:
render_template() {
eval "echo \"$(sed 's/\"/\\\\"/g' $1)\""
}
I can't think of keeping trailing new lines, but empty lines in between are kept.
Although it is an old topic, IMO I found out more elegant solution here: http://pempek.net/articles/2013/07/08/bash-sh-as-template-engine/
#!/bin/sh
# render a template configuration file
# expand variables + preserve formatting
render_template() {
eval "echo \"$(cat $1)\""
}
user="Gregory"
render_template /path/to/template.txt > path/to/configuration_file
All credits to Grégory Pakosz.
Instead of reinventing the wheel go with envsubst
Can be used in almost any scenario, for instance building configuration files from environment variables in docker containers.
If on mac make sure you have homebrew then link it from gettext:
brew install gettext
brew link --force gettext
./template.cfg
# We put env variables into placeholders here
this_variable_1 = ${SOME_VARIABLE_1}
this_variable_2 = ${SOME_VARIABLE_2}
./.env:
SOME_VARIABLE_1=value_1
SOME_VARIABLE_2=value_2
./configure.sh
#!/bin/bash
cat template.cfg | envsubst > whatever.cfg
Now just use it:
# make script executable
chmod +x ./configure.sh
# source your variables
. .env
# export your variables
# In practice you may not have to manually export variables
# if your solution depends on tools that utilise .env file
# automatically like pipenv etc.
export SOME_VARIABLE_1 SOME_VARIABLE_2
# Create your config file
./configure.sh
I'd have done it this way, probably less efficient, but easier to read/maintain.
TEMPLATE='/path/to/template.file'
OUTPUT='/path/to/output.file'
while read LINE; do
echo $LINE |
sed 's/VARONE/NEWVALA/g' |
sed 's/VARTWO/NEWVALB/g' |
sed 's/VARTHR/NEWVALC/g' >> $OUTPUT
done < $TEMPLATE
If you want to use Jinja2 templates, see this project: j2cli.
It supports:
Templates from JSON, INI, YAML files and input streams
Templating from environment variables
A longer but more robust version of the accepted answer:
perl -pe 's;(\\*)(\$([a-zA-Z_][a-zA-Z_0-9]*)|\$\{([a-zA-Z_][a-zA-Z_0-9]*)\})?;substr($1,0,int(length($1)/2)).($2&&length($1)%2?$2:$ENV{$3||$4});eg' template.txt
This expands all instances of $VAR or ${VAR} to their environment values (or, if they're undefined, the empty string).
It properly escapes backslashes, and accepts a backslash-escaped $ to inhibit substitution (unlike envsubst, which, it turns out, doesn't do this).
So, if your environment is:
FOO=bar
BAZ=kenny
TARGET=backslashes
NOPE=engi
and your template is:
Two ${TARGET} walk into a \\$FOO. \\\\
\\\$FOO says, "Delete C:\\Windows\\System32, it's a virus."
$BAZ replies, "\${NOPE}s."
the result would be:
Two backslashes walk into a \bar. \\
\$FOO says, "Delete C:\Windows\System32, it's a virus."
kenny replies, "${NOPE}s."
If you only want to escape backslashes before $ (you could write "C:\Windows\System32" in a template unchanged), use this slightly-modified version:
perl -pe 's;(\\*)(\$([a-zA-Z_][a-zA-Z_0-9]*)|\$\{([a-zA-Z_][a-zA-Z_0-9]*)\});substr($1,0,int(length($1)/2)).(length($1)%2?$2:$ENV{$3||$4});eg' template.txt
Here's another pure bash solution:
it's using heredoc, so:
complexity doesn't increase because of additionaly required syntax
template can include bash code
that also allows you to indent stuff properly. See below.
it doesn't use eval, so:
no problems with the rendering of trailing empty lines
no problems with quotes in the template
$ cat code
#!/bin/bash
LISTING=$( ls )
cat_template() {
echo "cat << EOT"
cat "$1"
echo EOT
}
cat_template template | LISTING="$LISTING" bash
Input:
$ cat template (with trailing newlines and double quotes)
<html>
<head>
</head>
<body>
<p>"directory listing"
<pre>
$( echo "$LISTING" | sed 's/^/ /' )
<pre>
</p>
</body>
</html>
Output:
<html>
<head>
</head>
<body>
<p>"directory listing"
<pre>
code
template
<pre>
</p>
</body>
</html>
Here is another solution: generate a bash script with all the variables and the contents of the template file, that script would look like this:
word=dog
i=1
cat << EOF
the number is ${i}
the word is ${word}
EOF
If we feed this script into bash it would produce the desired output:
the number is 1
the word is dog
Here is how to generate that script and feed that script into bash:
(
# Variables
echo word=dog
echo i=1
# add the template
echo "cat << EOF"
cat template.txt
echo EOF
) | bash
Discussion
The parentheses opens a sub shell, its purpose is to group together all the output generated
Within the sub shell, we generate all the variable declarations
Also in the sub shell, we generate the cat command with HEREDOC
Finally, we feed the sub shell output to bash and produce the desired output
If you want to redirect this output into a file, replace the last line with:
) | bash > output.txt
Taking the answer from ZyX using pure bash but with new style regex matching and indirect parameter substitution it becomes:
#!/bin/bash
regex='\$\{([a-zA-Z_][a-zA-Z_0-9]*)\}'
while read line; do
while [[ "$line" =~ $regex ]]; do
param="${BASH_REMATCH[1]}"
line=${line//${BASH_REMATCH[0]}/${!param}}
done
echo $line
done
If using Perl is an option and you're content with basing expansions on environment variables only (as opposed to all shell variables), consider Stuart P. Bentley's robust answer.
This answer aims to provide a bash-only solution that - despite use of eval - should be safe to use.
The goals are:
Support expansion of both ${name} and $name variable references.
Prevent all other expansions:
command substitutions ($(...) and legacy syntax `...`)
arithmetic substitutions ($((...)) and legacy syntax $[...]).
Allow selective suppression of variable expansion by prefixing with \ (\${name}).
Preserve special chars. in the input, notably " and \ instances.
Allow input either via arguments or via stdin.
Function expandVars():
expandVars() {
local txtToEval=$* txtToEvalEscaped
# If no arguments were passed, process stdin input.
(( $# == 0 )) && IFS= read -r -d '' txtToEval
# Disable command substitutions and arithmetic expansions to prevent execution
# of arbitrary commands.
# Note that selectively allowing $((...)) or $[...] to enable arithmetic
# expressions is NOT safe, because command substitutions could be embedded in them.
# If you fully trust or control the input, you can remove the `tr` calls below
IFS= read -r -d '' txtToEvalEscaped < <(printf %s "$txtToEval" | tr '`([' '\1\2\3')
# Pass the string to `eval`, escaping embedded double quotes first.
# `printf %s` ensures that the string is printed without interpretation
# (after processing by by bash).
# The `tr` command reconverts the previously escaped chars. back to their
# literal original.
eval printf %s "\"${txtToEvalEscaped//\"/\\\"}\"" | tr '\1\2\3' '`(['
}
Examples:
$ expandVars '\$HOME="$HOME"; `date` and $(ls)'
$HOME="/home/jdoe"; `date` and $(ls) # only $HOME was expanded
$ printf '\$SHELL=${SHELL}, but "$(( 1 \ 2 ))" will not expand' | expandVars
$SHELL=/bin/bash, but "$(( 1 \ 2 ))" will not expand # only ${SHELL} was expanded
For performance reasons, the function reads stdin input all at once into memory, but it's easy to adapt the function to a line-by-line approach.
Also supports non-basic variable expansions such as ${HOME:0:10}, as long as they contain no embedded command or arithmetic substitutions, such as ${HOME:0:$(echo 10)}
Such embedded substitutions actually BREAK the function (because all $( and ` instances are blindly escaped).
Similarly, malformed variable references such as ${HOME (missing closing }) BREAK the function.
Due to bash's handling of double-quoted strings, backslashes are handled as follows:
\$name prevents expansion.
A single \ not followed by $ is preserved as is.
If you want to represent multiple adjacent \ instances, you must double them; e.g.:
\\ -> \ - the same as just \
\\\\ -> \\
The input mustn't contain the following (rarely used) characters, which are used for internal purposes: 0x1, 0x2, 0x3.
There's a largely hypothetical concern that if bash should introduce new expansion syntax, this function might not prevent such expansions - see below for a solution that doesn't use eval.
If you're looking for a more restrictive solution that only supports ${name} expansions - i.e., with mandatory curly braces, ignoring $name references - see this answer of mine.
Here is an improved version of the bash-only, eval-free solution from the accepted answer:
The improvements are:
Support for expansion of both ${name} and $name variable references.
Support for \-escaping variable references that shouldn't be expanded.
Unlike the eval-based solution above,
non-basic expansions are ignored
malformed variable references are ignored (they don't break the script)
IFS= read -d '' -r lines # read all input from stdin at once
end_offset=${#lines}
while [[ "${lines:0:end_offset}" =~ (.*)\$(\{([a-zA-Z_][a-zA-Z_0-9]*)\}|([a-zA-Z_][a-zA-Z_0-9]*))(.*) ]] ; do
pre=${BASH_REMATCH[1]} # everything before the var. reference
post=${BASH_REMATCH[5]}${lines:end_offset} # everything after
# extract the var. name; it's in the 3rd capture group, if the name is enclosed in {...}, and the 4th otherwise
[[ -n ${BASH_REMATCH[3]} ]] && varName=${BASH_REMATCH[3]} || varName=${BASH_REMATCH[4]}
# Is the var ref. escaped, i.e., prefixed with an odd number of backslashes?
if [[ $pre =~ \\+$ ]] && (( ${#BASH_REMATCH} % 2 )); then
: # no change to $lines, leave escaped var. ref. untouched
else # replace the variable reference with the variable's value using indirect expansion
lines=${pre}${!varName}${post}
fi
end_offset=${#pre}
done
printf %s "$lines"
To follow up on plockc's answer on this page, here is a dash-suitable version, for those of you looking to avoid bashisms.
eval "cat <<EOF >outputfile
$( cat template.in )
EOF
" 2> /dev/null
Try shtpl
Perfect case for shtpl. (project of mine, so it is not widely in use and lacks in documentation. But here is the solution it offers anyhow. May you want to test it.)
Just execute:
$ i=1 word=dog sh -c "$( shtpl template.txt )"
Result is:
the number is 1
the word is dog
Have fun.
This page describes an answer with awk
awk '{while(match($0,"[$]{[^}]*}")) {var=substr($0,RSTART+2,RLENGTH -3);gsub("[$]{"var"}",ENVIRON[var])}}1' < input.txt > output.txt
# Usage: template your_file.conf.template > your_file.conf
template() {
local IFS line
while IFS=$'\n\r' read -r line ; do
line=${line//\\/\\\\} # escape backslashes
line=${line//\"/\\\"} # escape "
line=${line//\`/\\\`} # escape `
line=${line//\$/\\\$} # escape $
line=${line//\\\${/\${} # de-escape ${ - allows variable substitution: ${var} ${var:-default_value} etc
# to allow arithmetic expansion or command substitution uncomment one of following lines:
# line=${line//\\\$\(/\$\(} # de-escape $( and $(( - allows $(( 1 + 2 )) or $( command ) - UNSECURE
# line=${line//\\\$\(\(/\$\(\(} # de-escape $(( - allows $(( 1 + 2 ))
eval "echo \"${line}\"";
done < "$1"
}
This is the pure bash function adjustable to your liking, used in production and should not break on any input.
If it breaks - let me know.
You can also use bashible (which internally uses the evaluating approach described above/below).
There is an example, how to generate a HTML from multiple parts:
https://github.com/mig1984/bashible/tree/master/examples/templates
Look at simple variables substitution python script here: https://github.com/jeckep/vsubst
It is very simple to use:
python subst.py --props secure.properties --src_path ./templates --dst_path ./dist
Here's a bash function that preserves whitespace:
# Render a file in bash, i.e. expand environment variables. Preserves whitespace.
function render_file () {
while IFS='' read line; do
eval echo \""${line}"\"
done < "${1}"
}
Here's a modified perl script based on a few of the other answers:
perl -pe 's/([^\\]|^)\$\{([a-zA-Z_][a-zA-Z_0-9]*)\}/$1.$ENV{$2}/eg' -i template
Features (based on my needs, but should be easy to modify):
Skips escaped parameter expansions (e.g. \${VAR}).
Supports parameter expansions of the form ${VAR}, but not $VAR.
Replaces ${VAR} with a blank string if there is no VAR envar.
Only supports a-z, A-Z, 0-9 and underscore characters in the name (excluding digits in the first position).
You can also use printf to fill a template.
#!/bin/bash
IFS='' read -rd '' TEMPL <<-'EOB'
The number is %d
The word is "%s"
Birds of Massachusetts:
%s
EOB
N=12
WORD="Bird"
MULTILINE="Eastern Bluebirds
Common Grackles"
echo "START"
printf "${TEMPL}" ${N} ${WORD} "${MULTILINE}"
echo "END"
Here's the output, with quotes and whitespace intact:
START
The number is 12
The word is "Bird"
Birds of Massachusetts:
Eastern Bluebirds
Common Grackles
END