Related
I am planning a script to manage some pieces of my Linux systems and am at the point of deciding if I want to use bash or python.
I would prefer to do this as a Bash script simply because the commands are easier, but the real deciding factor is configuration. I need to be able to store a multi-dimensional array in the configuration file to tell the script what to do with itself. Storing simple key=value pairs in config files is easy enough with bash, but the only way I can think of to do a multi-dimensional array is a two layer parsing engine, something like
array=&d1|v1;v2;v3&d2|v1;v2;v3
but the marshall/unmarshall code could get to be a bear and its far from user friendly for the next poor sap that has to administer this. If i can't do this easily in bash i will simply write the configs to an xml file and write the script in python.
Is there an easy way to do this in bash?
thanks everyone.
Bash does not support multidimensional arrays, nor hashes, and it seems that you want a hash that values are arrays. This solution is not very beautiful, a solution with an xml file should be better :
array=('d1=(v1 v2 v3)' 'd2=(v1 v2 v3)')
for elt in "${array[#]}";do eval $elt;done
echo "d1 ${#d1[#]} ${d1[#]}"
echo "d2 ${#d2[#]} ${d2[#]}"
EDIT: this answer is quite old, since since bash 4 supports hash tables, see also this answer for a solution without eval.
Bash doesn't have multi-dimensional array. But you can simulate a somewhat similar effect with associative arrays. The following is an example of associative array pretending to be used as multi-dimensional array:
declare -A arr
arr[0,0]=0
arr[0,1]=1
arr[1,0]=2
arr[1,1]=3
echo "${arr[0,0]} ${arr[0,1]}" # will print 0 1
If you don't declare the array as associative (with -A), the above won't work. For example, if you omit the declare -A arr line, the echo will print 2 3 instead of 0 1, because 0,0, 1,0 and such will be taken as arithmetic expression and evaluated to 0 (the value to the right of the comma operator).
This works thanks to 1. "indirect expansion" with ! which adds one layer of indirection, and 2. "substring expansion" which behaves differently with arrays and can be used to "slice" them as described https://stackoverflow.com/a/1336245/317623
# Define each array and then add it to the main one
SUB_0=("name0" "value 0")
SUB_1=("name1" "value;1")
MAIN_ARRAY=(
SUB_0[#]
SUB_1[#]
)
# Loop and print it. Using offset and length to extract values
COUNT=${#MAIN_ARRAY[#]}
for ((i=0; i<$COUNT; i++))
do
NAME=${!MAIN_ARRAY[i]:0:1}
VALUE=${!MAIN_ARRAY[i]:1:1}
echo "NAME ${NAME}"
echo "VALUE ${VALUE}"
done
It's based off of this answer here
If you want to use a bash script and keep it easy to read recommend putting the data in structured JSON, and then use lightweight tool jq in your bash command to iterate through the array. For example with the following dataset:
[
{"specialId":"123",
"specialName":"First"},
{"specialId":"456",
"specialName":"Second"},
{"specialId":"789",
"specialName":"Third"}
]
You can iterate through this data with a bash script and jq like this:
function loopOverArray(){
jq -c '.[]' testing.json | while read i; do
# Do stuff here
echo "$i"
done
}
loopOverArray
Outputs:
{"specialId":"123","specialName":"First"}
{"specialId":"456","specialName":"Second"}
{"specialId":"789","specialName":"Third"}
Independent of the shell being used (sh, ksh, bash, ...) the following approach works pretty well for n-dimensional arrays (the sample covers a 2-dimensional array).
In the sample the line-separator (1st dimension) is the space character. For introducing a field separator (2nd dimension) the standard unix tool tr is used. Additional separators for additional dimensions can be used in the same way.
Of course the performance of this approach is not very well, but if performance is not a criteria this approach is quite generic and can solve many problems:
array2d="1.1:1.2:1.3 2.1:2.2 3.1:3.2:3.3:3.4"
function process2ndDimension {
for dimension2 in $*
do
echo -n $dimension2 " "
done
echo
}
function process1stDimension {
for dimension1 in $array2d
do
process2ndDimension `echo $dimension1 | tr : " "`
done
}
process1stDimension
The output of that sample looks like this:
1.1 1.2 1.3
2.1 2.2
3.1 3.2 3.3 3.4
After a lot of trial and error i actually find the best, clearest and easiest multidimensional array on bash is to use a regular var. Yep.
Advantages: You don't have to loop through a big array, you can just echo "$var" and use grep/awk/sed. It's easy and clear and you can have as many columns as you like.
Example:
$ var=$(echo -e 'kris hansen oslo\nthomas jonson peru\nbibi abu johnsonville\njohnny lipp peru')
$ echo "$var"
kris hansen oslo
thomas johnson peru
bibi abu johnsonville
johnny lipp peru
If you want to find everyone in peru
$ echo "$var" | grep peru
thomas johnson peru
johnny lipp peru
Only grep(sed) in the third field
$ echo "$var" | sed -n -E '/(.+) (.+) peru/p'
thomas johnson peru
johnny lipp peru
If you only want x field
$ echo "$var" | awk '{print $2}'
hansen
johnson
abu
johnny
Everyone in peru that's called thomas and just return his lastname
$ echo "$var" |grep peru|grep thomas|awk '{print $2}'
johnson
Any query you can think of... supereasy.
To change an item:
$ var=$(echo "$var"|sed "s/thomas/pete/")
To delete a row that contains "x"
$ var=$(echo "$var"|sed "/thomas/d")
To change another field in the same row based on a value from another item
$ var=$(echo "$var"|sed -E "s/(thomas) (.+) (.+)/\1 test \3/")
$ echo "$var"
kris hansen oslo
thomas test peru
bibi abu johnsonville
johnny lipp peru
Of course looping works too if you want to do that
$ for i in "$var"; do echo "$i"; done
kris hansen oslo
thomas jonson peru
bibi abu johnsonville
johnny lipp peru
The only gotcha iv'e found with this is that you must always quote the
var(in the example; both var and i) or things will look like this
$ for i in "$var"; do echo $i; done
kris hansen oslo thomas jonson peru bibi abu johnsonville johnny lipp peru
and someone will undoubtedly say it won't work if you have spaces in your input, however that can be fixed by using another delimeter in your input, eg(using an utf8 char now to emphasize that you can choose something your input won't contain, but you can choose whatever ofc):
$ var=$(echo -e 'field one☥field two hello☥field three yes moin\nfield 1☥field 2☥field 3 dsdds aq')
$ for i in "$var"; do echo "$i"; done
field one☥field two hello☥field three yes moin
field 1☥field 2☥field 3 dsdds aq
$ echo "$var" | awk -F '☥' '{print $3}'
field three yes moin
field 3 dsdds aq
$ var=$(echo "$var"|sed -E "s/(field one)☥(.+)☥(.+)/\1☥test☥\3/")
$ echo "$var"
field one☥test☥field three yes moin
field 1☥field 2☥field 3 dsdds aq
If you want to store newlines in your input, you could convert the newline to something else before input and convert it back again on output(or don't use bash...). Enjoy!
I am posting the following because it is a very simple and clear way to mimic (at least to some extent) the behavior of a two-dimensional array in Bash. It uses a here-file (see the Bash manual) and read (a Bash builtin command):
## Store the "two-dimensional data" in a file ($$ is just the process ID of the shell, to make sure the filename is unique)
cat > physicists.$$ <<EOF
Wolfgang Pauli 1900
Werner Heisenberg 1901
Albert Einstein 1879
Niels Bohr 1885
EOF
nbPhysicists=$(wc -l physicists.$$ | cut -sf 1 -d ' ') # Number of lines of the here-file specifying the physicists.
## Extract the needed data
declare -a person # Create an indexed array (necessary for the read command).
while read -ra person; do
firstName=${person[0]}
familyName=${person[1]}
birthYear=${person[2]}
echo "Physicist ${firstName} ${familyName} was born in ${birthYear}"
# Do whatever you need with data
done < physicists.$$
## Remove the temporary file
rm physicists.$$
Output:
Physicist Wolfgang Pauli was born in 1900 Physicist Werner Heisenberg was born in 1901 Physicist Albert Einstein was born in 1879 Physicist Niels Bohr was born in 1885
The way it works:
The lines in the temporary file created play the role of one-dimensional vectors, where the blank spaces (or whatever separation character you choose; see the description of the read command in the Bash manual) separate the elements of these vectors.
Then, using the read command with its -a option, we loop over each line of the file (until we reach end of file). For each line, we can assign the desired fields (= words) to an array, which we declared just before the loop. The -r option to the read command prevents backslashes from acting as escape characters, in case we typed backslashes in the here-document physicists.$$.
In conclusion a file is created as a 2D-array, and its elements are extracted using a loop over each line, and using the ability of the read command to assign words to the elements of an (indexed) array.
Slight improvement:
In the above code, the file physicists.$$ is given as input to the while loop, so that it is in fact passed to the read command. However, I found that this causes problems when I have another command asking for input inside the while loop. For example, the select command waits for standard input, and if placed inside the while loop, it will take input from physicists.$$, instead of prompting in the command-line for user input.
To correct this, I use the -u option of read, which allows to read from a file descriptor. We only have to create a file descriptor (with the exec command) corresponding to physicists.$$ and to give it to the -u option of read, as in the following code:
## Store the "two-dimensional data" in a file ($$ is just the process ID of the shell, to make sure the filename is unique)
cat > physicists.$$ <<EOF
Wolfgang Pauli 1900
Werner Heisenberg 1901
Albert Einstein 1879
Niels Bohr 1885
EOF
nbPhysicists=$(wc -l physicists.$$ | cut -sf 1 -d ' ') # Number of lines of the here-file specifying the physicists.
exec {id_file}<./physicists.$$ # Create a file descriptor stored in 'id_file'.
## Extract the needed data
declare -a person # Create an indexed array (necessary for the read command).
while read -ra person -u "${id_file}"; do
firstName=${person[0]}
familyName=${person[1]}
birthYear=${person[2]}
echo "Physicist ${firstName} ${familyName} was born in ${birthYear}"
# Do whatever you need with data
done
## Close the file descriptor
exec {id_file}<&-
## Remove the temporary file
rm physicists.$$
Notice that the file descriptor is closed at the end.
Bash does not supports multidimensional array, but we can implement using Associate array. Here the indexes are the key to retrieve the value. Associate array is available in bash version 4.
#!/bin/bash
declare -A arr2d
rows=3
columns=2
for ((i=0;i<rows;i++)) do
for ((j=0;j<columns;j++)) do
arr2d[$i,$j]=$i
done
done
for ((i=0;i<rows;i++)) do
for ((j=0;j<columns;j++)) do
echo ${arr2d[$i,$j]}
done
done
Expanding on Paul's answer - here's my version of working with associative sub-arrays in bash:
declare -A SUB_1=(["name1key"]="name1val" ["name2key"]="name2val")
declare -A SUB_2=(["name3key"]="name3val" ["name4key"]="name4val")
STRING_1="string1val"
STRING_2="string2val"
MAIN_ARRAY=(
"${SUB_1[*]}"
"${SUB_2[*]}"
"${STRING_1}"
"${STRING_2}"
)
echo "COUNT: " ${#MAIN_ARRAY[#]}
for key in ${!MAIN_ARRAY[#]}; do
IFS=' ' read -a val <<< ${MAIN_ARRAY[$key]}
echo "VALUE: " ${val[#]}
if [[ ${#val[#]} -gt 1 ]]; then
for subkey in ${!val[#]}; do
subval=${val[$subkey]}
echo "SUBVALUE: " ${subval}
done
fi
done
It works with mixed values in the main array - strings/arrays/assoc. arrays
The key here is to wrap the subarrays in single quotes and use * instead of # when storing a subarray inside the main array so it would get stored as a single, space separated string: "${SUB_1[*]}"
Then it makes it easy to parse an array out of that when looping through values with IFS=' ' read -a val <<< ${MAIN_ARRAY[$key]}
The code above outputs:
COUNT: 4
VALUE: name1val name2val
SUBVALUE: name1val
SUBVALUE: name2val
VALUE: name4val name3val
SUBVALUE: name4val
SUBVALUE: name3val
VALUE: string1val
VALUE: string2val
Lots of answers found here for creating multidimensional arrays in bash.
And without exception, all are obtuse and difficult to use.
If MD arrays are a required criteria, it is time to make a decision:
Use a language that supports MD arrays
My preference is Perl. Most would probably choose Python.
Either works.
Store the data elsewhere
JSON and jq have already been suggested. XML has also been suggested, though for your use JSON and jq would likely be simpler.
It would seem though that Bash may not be the best choice for what you need to do.
Sometimes the correct question is not "How do I do X in tool Y?", but rather "Which tool would be best to do X?"
I do this using associative arrays since bash 4 and setting IFS to a value that can be defined manually.
The purpose of this approach is to have arrays as values of associative array keys.
In order to set IFS back to default just unset it.
unset IFS
This is an example:
#!/bin/bash
set -euo pipefail
# used as value in asscciative array
test=(
"x3:x4:x5"
)
# associative array
declare -A wow=(
["1"]=$test
["2"]=$test
)
echo "default IFS"
for w in ${wow[#]}; do
echo " $w"
done
IFS=:
echo "IFS=:"
for w in ${wow[#]}; do
for t in $w; do
echo " $t"
done
done
echo -e "\n or\n"
for w in ${!wow[#]}
do
echo " $w"
for t in ${wow[$w]}
do
echo " $t"
done
done
unset IFS
unset w
unset t
unset wow
unset test
The output of the script below is:
default IFS
x3:x4:x5
x3:x4:x5
IFS=:
x3
x4
x5
x3
x4
x5
or
1
x3
x4
x5
2
x3
x4
x5
I've got a pretty simple yet smart workaround:
Just define the array with variables in its name. For example:
for (( i=0 ; i<$(($maxvalue + 1)) ; i++ ))
do
for (( j=0 ; j<$(($maxargument + 1)) ; j++ ))
do
declare -a array$i[$j]=((Your rule))
done
done
Don't know whether this helps since it's not exactly what you asked for, but it works for me. (The same could be achieved just with variables without the array)
echo "Enter no of terms"
read count
for i in $(seq 1 $count)
do
t=` expr $i - 1 `
for j in $(seq $t -1 0)
do
echo -n " "
done
j=` expr $count + 1 `
x=` expr $j - $i `
for k in $(seq 1 $x)
do
echo -n "* "
done
echo ""
done
I run a script with the param -A AA/BB . To get an array with AA and BB, i can do this.
INPUT_PARAM=(${AIRLINE_OPTION//-A / }) #get rid of the '-A ' in the begining
LIST=(${AIRLINES_PARAM//\// }) # split by '/'
Can we achieve this in a single line?
Thanks in advance.
One way
IFS=/ read -r -a LIST <<< "${AIRLINE_OPTION//-A /}"
This places the output from the parameter substitution ${AIRLINE_OPTION//-A /} into a "here-string" and uses the bash read built-in to parse this into an array. Splitting by / is achieved by setting the value of IFS to / for the read command.
LIST=( $(IFS=/; for x in ${AIRLINE_OPTION#-A }; do printf "$x "; done) )
This is a portable solution, but if your read supports -a and you don't mind portability then you should go for #1_CR's solution.
With awk, for example, you can create an array and store it in LIST variable:
$ LIST=($(awk -F"[\/ ]" '{print $2,$3}' <<< "-A AA/BB"))
Result:
$ echo ${LIST[0]}
AA
$ echo ${LIST[1]}
BB
Explanation
-F"[\/ ]" defines two possible field separators: a space or a slash /.
'{print $2$3}' prints the 2nd and 3rd fields based on those separators.
I'm trying to write a bash script that increments the version number which is given in
{major}.{minor}.{revision}
For example.
1.2.13
Is there a good way to easily extract those 3 numbers using something like sed or awk such that I could increment the {revision} number and output the full version number string.
$ v=1.2.13
$ echo "${v%.*}.$((${v##*.}+1))"
1.2.14
$ v=11.1.2.3.0
$ echo "${v%.*}.$((${v##*.}+1))"
11.1.2.3.1
Here is how it works:
The string is split in two parts.
the first one contains everything but the last dot and next characters: ${v%.*}
the second one contains everything but all characters up to the last dot: ${v##*.}
The first part is printed as is, followed by a plain dot and the last part incremented using shell arithmetic expansion: $((x+1))
Pure Bash using an array:
version='1.2.33'
a=( ${version//./ } ) # replace points, split into array
((a[2]++)) # increment revision (or other part)
version="${a[0]}.${a[1]}.${a[2]}" # compose new version
I prefer "cut" command for this kind of things
major=`echo $version | cut -d. -f1`
minor=`echo $version | cut -d. -f2`
revision=`echo $version | cut -d. -f3`
revision=`expr $revision + 1`
echo "$major.$minor.$revision"
I know this is not the shortest way, but for me it's simplest to understand and to read...
Yet another shell way (showing there's always more than one way to bugger around with this stuff...):
$ echo 1.2.3 | ( IFS=".$IFS" ; read a b c && echo $a.$b.$((c + 1)) )
1.2.4
So, we can do:
$ x=1.2.3
$ y=`echo $x | ( IFS=".$IFS" ; read a b c && echo $a.$b.$((c + 1)) )`
$ echo $y
1.2.4
Awk makes it quite simple:
echo "1.2.14" | awk -F \. {'print $1,$2, $3'} will print out 1 2 14.
flag -F specifies separator.
If you wish to save one of the values:
firstVariable=$(echo "1.2.14" | awk -F \. {'print $1'})
I use the shell's own word splitting; something like
oIFS="$IFS"
IFS=.
set -- $version
IFS="$oIFS"
although you need to be careful with version numbers in general due to alphabetic or date suffixes and other annoyingly inconsistent bits. After this, the positional parameters will be set to the components of $version:
$1 = 1
$2 = 2
$3 = 13
($IFS is a set of single characters, not a string, so this won't work with a multicharacter field separator, although you can use IFS=.- to split on either . or -.)
Inspired by the answer of jlliagre I made my own version which supports version numbers just having a major version given. jlliagre's version will make 1 -> 1.2 instead of 2.
This one is appropriate to both styles of version numbers:
function increment_version()
local VERSION="$1"
local INCREMENTED_VERSION=
if [[ "$VERSION" =~ .*\..* ]]; then
INCREMENTED_VERSION="${VERSION%.*}.$((${VERSION##*.}+1))"
else
INCREMENTED_VERSION="$((${VERSION##*.}+1))"
fi
echo "$INCREMENTED_VERSION"
}
This will produce the following outputs:
increment_version 1 -> 2
increment_version 1.2 -> 1.3
increment_version 1.2.9 -> 1.2.10
increment_version 1.2.9.101 -> 1.2.9.102
Small variation on fgm's solution using the builtin read command to split the string into an array. Note that the scope of the IFS variable is limited to the read command (so no need to store & restore the current IFS variable).
version='1.2.33'
IFS='.' read -r -a a <<<"$version"
((a[2]++))
printf '%s\n' "${a[#]}" | nl
version="${a[0]}.${a[1]}.${a[2]}"
echo "$version"
See: How do I split a string on a delimiter in Bash?
I'm surprised no one suggested grep yet.
Here's how to get the full version (not limited to the length of x.y.z...) from a file name:
filename="openshift-install-linux-4.12.0-ec.3.tar.gz"
find -name "$filename" | grep -Eo '([0-9]+)(\.?[0-9]+)*' | head -1
# 4.12.0
I have a file that has two columns of floating point values. I also have a C program that takes a floating point value as input and returns another floating point value as output.
What I'd like to do is the following: for each row in the original, execute the C program with the value in the first column as input, and then print out the first column (unchanged) followed by the second column minus the result of the C program.
As an example, suppose c_program returns the square of the input and behaves like this:
$ c_program 4
16
$
and suppose data_file looks like this:
1 10
2 11
3 12
4 13
What I'd like to return as output, in this case, is
1 9
2 7
3 3
4 -3
To write this in really sketchy pseudocode, I want to do something like this:
awk '{print $1, $2 - `c_program $1`}' data_file
But of course, I can't just pass $1, the awk variable, into a call to c_program. What's the right way to do this, and preferably, how could I do it while still maintaining the "awk one-liner"? (I don't want to pull out a sledgehammer and write a full-fledged C program to do this.)
you just do everything in awk
awk '{cmd="c_program "$1; cmd|getline l;print $1,$2-l}' file
This shows how to execute a command in awk:
ls | awk '/^a/ {system("ls -ld " $1)}'
You could use a bash script instead:
while read line
do
FIRST=`echo $line | cut -d' ' -f1`
SECOND=`echo $line | cut -d' ' -f2`
OUT=`expr $SECOND \* 4`
echo $FIRST $OUT `expr $OUT - $SECOND`
done
The shell is a better tool for this using a little used feature. There is a shell variable IFS which is the Input Field Separator that sh uses to split command lines when parsing; it defaults to <Space><Tab><Newline> which is why ls foo is interpreted as two words.
When set is given arguments not beginning with - it sets the positional parameters of the shell to the contents of the arguments as split via IFS, thus:
#!/bin/sh
while read line ; do
set $line
subtrahend=`c_program $1`
echo $1 `expr $2 - $subtrahend`
done < data_file
Pure Bash, without using any external executables other than your program:
#!/bin/bash
while read num1 num2
do
(( result = $(c_program num2) - num1 ))
echo "$num1 $result"
done
As others have pointed out: awk is not not well equipped for this job. Here is a suggestion in bash:
#!/bin/sh
data_file=$1
while read column_1 column_2 the_rest
do
((result=$(c_program $column_1)-$column_2))
echo $column_1 $result "$the_rest"
done < $data_file
Save this to a file, say myscript.sh, then invoke it as:
sh myscript.sh data_file
The read command reads each line from the data file (which was redirected to the standard input) and assign the first 2 columns to $column_1 and $column_2 variables. The rest of the line, if there is any, is stored in $the_rest.
Next, I calculate the result based on your requirements and prints out the line based on your requirements. Note that I surround $the_rest with quotes to reserve spacing. Failure to do so will result in multiple spaces in the input file to be squeezed into one.
I want to pipe the output of a "template" file into MySQL, the file having variables like ${dbName} interspersed. What is the command line utility to replace these instances and dump the output to standard output?
The input file is considered to be safe, but faulty substitution definitions could exist. Performing the replacement should avoid performing unintended code execution.
Update
Here is a solution from yottatsa on a similar question that only does replacement for variables like $VAR or ${VAR}, and is a brief one-liner
i=32 word=foo envsubst < template.txt
Of course if i and word are in your environment, then it is just
envsubst < template.txt
On my Mac it looks like it was installed as part of gettext and from MacGPG2
Old Answer
Here is an improvement to the solution from mogsie on a similar question, my solution does not require you to escale double quotes, mogsie's does, but his is a one liner!
eval "cat <<EOF
$(<template.txt)
EOF
" 2> /dev/null
The power on these two solutions is that you only get a few types of shell expansions that don't occur normally $((...)), `...`, and $(...), though backslash is an escape character here, but you don't have to worry that the parsing has a bug, and it does multiple lines just fine.
Sed!
Given template.txt:
The number is ${i}
The word is ${word}
we just have to say:
sed -e "s/\${i}/1/" -e "s/\${word}/dog/" template.txt
Thanks to Jonathan Leffler for the tip to pass multiple -e arguments to the same sed invocation.
Use /bin/sh. Create a small shell script that sets the variables, and then parse the template using the shell itself. Like so (edit to handle newlines correctly):
File template.txt:
the number is ${i}
the word is ${word}
File script.sh:
#!/bin/sh
#Set variables
i=1
word="dog"
#Read in template one line at the time, and replace variables (more
#natural (and efficient) way, thanks to Jonathan Leffler).
while read line
do
eval echo "$line"
done < "./template.txt"
Output:
#sh script.sh
the number is 1
the word is dog
I was thinking about this again, given the recent interest, and I think that the tool that I was originally thinking of was m4, the macro processor for autotools. So instead of the variable I originally specified, you'd use:
$echo 'I am a DBNAME' | m4 -DDBNAME="database name"
Create rendertemplate.sh:
#!/usr/bin/env bash
eval "echo \"$(cat $1)\""
And template.tmpl:
Hello, ${WORLD}
Goodbye, ${CHEESE}
Render the template:
$ export WORLD=Foo
$ CHEESE=Bar ./rendertemplate.sh template.tmpl
Hello, Foo
Goodbye, Bar
template.txt
Variable 1 value: ${var1}
Variable 2 value: ${var2}
data.sh
#!/usr/bin/env bash
declare var1="value 1"
declare var2="value 2"
parser.sh
#!/usr/bin/env bash
# args
declare file_data=$1
declare file_input=$2
declare file_output=$3
source $file_data
eval "echo \"$(< $file_input)\"" > $file_output
./parser.sh data.sh template.txt parsed_file.txt
parsed_file.txt
Variable 1 value: value 1
Variable 2 value: value 2
Here's a robust Bash function that - despite using eval - should be safe to use.
All ${varName} variable references in the input text are expanded based on the calling shell's variables.
Nothing else is expanded: neither variable references whose names are not enclosed in {...} (such as $varName), nor command substitutions ($(...) and legacy syntax `...`), nor arithmetic substitutions ($((...)) and legacy syntax $[...]).
To treat a $ as a literal, \-escape it; e.g.:\${HOME}
Note that input is only accepted via stdin.
Example:
$ expandVarsStrict <<<'$HOME is "${HOME}"; `date` and \$(ls)' # only ${HOME} is expanded
$HOME is "/Users/jdoe"; `date` and $(ls)
Function source code:
expandVarsStrict(){
local line lineEscaped
while IFS= read -r line || [[ -n $line ]]; do # the `||` clause ensures that the last line is read even if it doesn't end with \n
# Escape ALL chars. that could trigger an expansion..
IFS= read -r -d '' lineEscaped < <(printf %s "$line" | tr '`([$' '\1\2\3\4')
# ... then selectively reenable ${ references
lineEscaped=${lineEscaped//$'\4'{/\${}
# Finally, escape embedded double quotes to preserve them.
lineEscaped=${lineEscaped//\"/\\\"}
eval "printf '%s\n' \"$lineEscaped\"" | tr '\1\2\3\4' '`([$'
done
}
The function assumes that no 0x1, 0x2, 0x3, and 0x4 control characters are present in the input, because those chars. are used internally - since the function processes text, that should be a safe assumption.
here's my solution with perl based on former answer, replaces environment variables:
perl -p -e 's/\$\{(\w+)\}/(exists $ENV{$1}?$ENV{$1}:"missing variable $1")/eg' < infile > outfile
I would suggest using something like Sigil:
https://github.com/gliderlabs/sigil
It is compiled to a single binary, so it's extremely easy to install on systems.
Then you can do a simple one-liner like the following:
cat my-file.conf.template | sigil -p $(env) > my-file.conf
This is much safer than eval and easier then using regex or sed
Here is a way to get the shell to do the substitution for you, as if the contents of the file were instead typed between double quotes.
Using the example of template.txt with contents:
The number is ${i}
The word is ${word}
The following line will cause the shell to interpolate the contents of template.txt and write the result to standard out.
i='1' word='dog' sh -c 'echo "'"$(cat template.txt)"'"'
Explanation:
i and word are passed as environment variables scopped to the execution of sh.
sh executes the contents of the string it is passed.
Strings written next to one another become one string, that string is:
'echo "' + "$(cat template.txt)" + '"'
Since the substitution is between ", "$(cat template.txt)" becomes the output of cat template.txt.
So the command executed by sh -c becomes:
echo "The number is ${i}\nThe word is ${word}",
where i and word are the specified environment variables.
If you are open to using Perl, that would be my suggestion. Although there are probably some sed and/or AWK experts that probably know how to do this much easier. If you have a more complex mapping with more than just dbName for your replacements you could extend this pretty easily, but you might just as well put it into a standard Perl script at that point.
perl -p -e 's/\$\{dbName\}/testdb/s' yourfile | mysql
A short Perl script to do something slightly more complicated (handle multiple keys):
#!/usr/bin/env perl
my %replace = ( 'dbName' => 'testdb', 'somethingElse' => 'fooBar' );
undef $/;
my $buf = <STDIN>;
$buf =~ s/\$\{$_\}/$replace{$_}/g for keys %replace;
print $buf;
If you name the above script as replace-script, it could then be used as follows:
replace-script < yourfile | mysql
file.tpl:
The following bash function should only replace ${var1} syntax and ignore
other shell special chars such as `backticks` or $var2 or "double quotes".
If I have missed anything - let me know.
script.sh:
template(){
# usage: template file.tpl
while read -r line ; do
line=${line//\"/\\\"}
line=${line//\`/\\\`}
line=${line//\$/\\\$}
line=${line//\\\${/\${}
eval "echo \"$line\"";
done < ${1}
}
var1="*replaced*"
var2="*not replaced*"
template file.tpl > result.txt
I found this thread while wondering the same thing. It inspired me to this (careful with the backticks)
$ echo $MYTEST
pass!
$ cat FILE
hello $MYTEST world
$ eval echo `cat FILE`
hello pass! world
Lots of choices here, but figured I'd toss mine on the heap. It is perl based, only targets variables of the form ${...}, takes the file to process as an argument and outputs the converted file on stdout:
use Env;
Env::import();
while(<>) { $_ =~ s/(\${\w+})/$1/eeg; $text .= $_; }
print "$text";
Of course I'm not really a perl person, so there could easily be a fatal flaw (works for me though).
It can be done in bash itself if you have control of the configuration file format. You just need to source (".") the configuration file rather than subshell it. That ensures the variables are created in the context of the current shell (and continue to exist) rather than the subshell (where the variable disappear when the subshell exits).
$ cat config.data
export parm_jdbc=jdbc:db2://box7.co.uk:5000/INSTA
export parm_user=pax
export parm_pwd=never_you_mind
$ cat go.bash
. config.data
echo "JDBC string is " $parm_jdbc
echo "Username is " $parm_user
echo "Password is " $parm_pwd
$ bash go.bash
JDBC string is jdbc:db2://box7.co.uk:5000/INSTA
Username is pax
Password is never_you_mind
If your config file cannot be a shell script, you can just 'compile' it before executing thus (the compilation depends on your input format).
$ cat config.data
parm_jdbc=jdbc:db2://box7.co.uk:5000/INSTA # JDBC URL
parm_user=pax # user name
parm_pwd=never_you_mind # password
$ cat go.bash
cat config.data
| sed 's/#.*$//'
| sed 's/[ \t]*$//'
| sed 's/^[ \t]*//'
| grep -v '^$'
| sed 's/^/export '
>config.data-compiled
. config.data-compiled
echo "JDBC string is " $parm_jdbc
echo "Username is " $parm_user
echo "Password is " $parm_pwd
$ bash go.bash
JDBC string is jdbc:db2://box7.co.uk:5000/INSTA
Username is pax
Password is never_you_mind
In your specific case, you could use something like:
$ cat config.data
export p_p1=val1
export p_p2=val2
$ cat go.bash
. ./config.data
echo "select * from dbtable where p1 = '$p_p1' and p2 like '$p_p2%' order by p1"
$ bash go.bash
select * from dbtable where p1 = 'val1' and p2 like 'val2%' order by p1
Then pipe the output of go.bash into MySQL and voila, hopefully you won't destroy your database :-).
In place perl editing of potentially multiple files, with backups.
perl -e 's/\$\{([^}]+)\}/defined $ENV{$1} ? $ENV{$1} : ""/eg' \
-i.orig \
-p config/test/*
I created a shell templating script named shtpl. My shtpl uses a jinja-like syntax which, now that I use ansible a lot, I'm pretty familiar with:
$ cat /tmp/test
{{ aux=4 }}
{{ myarray=( a b c d ) }}
{{ A_RANDOM=$RANDOM }}
$A_RANDOM
{% if $(( $A_RANDOM%2 )) == 0 %}
$A_RANDOM is even
{% else %}
$A_RANDOM is odd
{% endif %}
{% if $(( $A_RANDOM%2 )) == 0 %}
{% for n in 1 2 3 $aux %}
\$myarray[$((n-1))]: ${myarray[$((n-1))]}
/etc/passwd field #$n: $(grep $USER /etc/passwd | cut -d: -f$n)
{% endfor %}
{% else %}
{% for n in {1..4} %}
\$myarray[$((n-1))]: ${myarray[$((n-1))]}
/etc/group field #$n: $(grep ^$USER /etc/group | cut -d: -f$n)
{% endfor %}
{% endif %}
$ ./shtpl < /tmp/test
6535
6535 is odd
$myarray[0]: a
/etc/group field #1: myusername
$myarray[1]: b
/etc/group field #2: x
$myarray[2]: c
/etc/group field #3: 1001
$myarray[3]: d
/etc/group field #4:
More info on my github
To me this is the easiest and most powerful solution, you can even include other templates using the same command eval echo "$(<template.txt):
Example with nested template
create the template files, the variables are in regular bash syntax ${VARIABLE_NAME} or $VARIABLE_NAME
you have to escape special characters with \ in your templates otherwhise they will be interpreted by eval.
template.txt
Hello ${name}!
eval echo $(<nested-template.txt)
nested-template.txt
Nice to have you here ${name} :\)
create source file
template.source
declare name=royman
parse the template
source template.source && eval echo "$(<template.txt)"
the output
Hello royman!
Nice to have you here royman :)
envsubst
please don't use anything else (ie. don't eval)