I want to make a few configuration files (for homeassistant) that are very similar to each other. I am aiming to use a template file as the base and put in a few substitution strings at the top of the file and use a bash script to read the substitutions and run sed with the applicable strings.
i.e.
# substitutions
# room = living_room
# switch = hallway_motion
# delay = 3
automations:
foo......
.........
entity_id: $switch
When I run the script it will look for any line beginning with a # that has a word (key) and then an = and another word (maybe string) (value) and replace anywhere that key with a $ in front is in the rest of the file.
Like what is done by esphome. https://esphome.io/guides/configuration-types.html#substitutions
I am getting stuck at finding the "keys" in the file. How can I script this so it can find all the "keys" recursively?
Or is there something that does this, or something similar, out there already?
You can do this with sed in two stages. The first stage will generate a second stage sed script to fill in your template. I'd make a small adjustment to your syntax and recommend that you require curly braces around your variable name. In other words, write your variable expansions like this:
# foo = bar
myentry: ${foo}
This makes it easier to avoid pitfalls when you have one variable name that's a prefix of another (e.g., foo and foobar).
#!/bin/bash
in="$1"
stage2=$(mktemp)
trap 'rm -f "$stage2"' EXIT
sed -n -e 's,^# \([[:alnum:]_]\+\) = \([^#]\+\),s#\${\1}#\2#g,p' "$in" > "$stage2"
sed -f "$stage2" "$in"
Provide a filename as the first argument, and it will print the filled out template on stdout.
This example code is pretty strict about white space on variable definition lines, but that can obviously be adjusted to your liking.
Related
I am a newbie in bash script.
Here is my environment:
Mac OS X Catalina
/bin/bash
I found here a mix of several commands to remove the duplicate string in a string.
I needed for my program which updates the .zhrc profile file.
Here is my code:
#!/bin/bash
a='export PATH="/Library/Frameworks/Python.framework/Versions/3.8/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/local/bin:"'
myvariable=$(echo "$a" | tr ':' '\n' | sort | uniq | xargs)
echo "myvariable : $myvariable"
Here is the output:
xargs: unterminated quote
myvariable :
After some test, I know that the source of the issue is due to some quotes "" inside my variable '$a'.
Why am I so sure?
Because when I execute this code for example:
#!/bin/bash
a="/Library/Java/JavaVirtualMachines/jdk1.8.0_271.jdk/Contents/Home:/Library/Java/JavaVirtualMachines/jdk1.8.0_271.jdk/Contents/Home"
myvariable=$(echo "$a" | tr ':' '\n' | sort | uniq | xargs)
echo "myvariable : $myvariable"
where $a doesn't contain any quotes, I get the correct output:
myvariable : /Library/Java/JavaVirtualMachines/jdk1.8.0_271.jdk/Contents/Home
I tried to search for a solution for "xargs: unterminated quote" but each answer found on the web is for a particular case which doesn't correspond to my problem.
As I am a newbie and this line command is using several complex commands, I was wondering if anyone know the magic trick to make it work.
Basically, you want to remove duplicates from a colon-separated list.
I don't know if this is considered cheating, but I would do this in another language and invoke it from bash. First I would write a script for this purpose in zsh: It accepts as parameter a string with colon separtors and outputs a colon-separated list with duplicates removed:
#!/bin/zsh
original=${1?Parameter missing} # Original string
# Auxiliary array, which is set up to act like a Set, i.e. without
# duplicates
typeset -aU nodups_array
# Split the original strings on the colons and store the pieces
# into the array, thereby removing duplicates. The core idea for
# this is stolen from:
# https://stackoverflow.com/questions/2930238/split-string-with-zsh-as-in-python
nodups_array=("${(#s/:/)original}")
# Join the array back with colons and write the resulting string
# to stdout.
echo ${(j':')nodups_array}
If we call this script nodups_string, you can invoke it in your bash-setting as:
#!/bin/bash
a_path="/Library/Frameworks/Python.framework/Versions/3.8/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/local/bin:"
nodups_a_path=$(nodups_string "$a_path")
my_variable="export PATH=$nodups_a_path"
echo "myvariable : $myvariable"
The overall effect would be literally what you asked for. However, there is still an open problem I should point out: If one of the PATH components happens to contain a space, the resulting export statement can not validly be executed. This problem is also inherent into your original problem; you just didn't mention it. You could do something like
my_variable=export\ PATH='"'$nodups_a_path"'"'
to avoid this. Of course, I wonder why you take such an effort to generat a syntactically valid export command, instead of simply building the PATH by directly where it is needed.
Side note: If you would use zsh as your shell instead of bash, and only want to keep your PATH free of duplicates, a simple
typeset -iU path
would suffice, and zsh takes care of the rest.
With awk:
awk -v RS=[:\"] 'NR > 1 { pth[$0]="" } END { for (i in pth) { if (i !~ /[[:space:]]+/ && i != "" ) { printf "%s:",i } } }' <<< "$a"
Set the record separator to : and double quotes. Then when the number record is greater than one, set up an array called pth with the path as the index. At the end, loop through the array, re printing the paths separated with :
I am using tcsh (contract required, cannot change to bash etc), but am having a problem building up a command based on various conditions for different pieces.
Some names changed to protect the innocent...
If new or old program name, is really chosen earlier on by a preprocessor, and is hardcoded by the time this shell script gets run:
set myCMDline = newProgName
set myCMDlineTmpFile = "/tmp/myCMDlineTmpScriptFile.csh"
set bsubQname = "typical"
set bsubResources = "span[hosts=1]"
set myCMDline = "bsub -q $bsubQname -n 8 -R \"$bsubResources\" $myCMDline"
($myCMDline)
Now, I have tried several variations of the above, all not working for some reason or another. The closest I think I get is a complaint about mismatched double-quotes, even when backspacing them.
When I do an echo of $myCMDline, then that looks OK, but the execution of same must somehow be different...
set bsubResources = '"span[hosts=1]"' #double-quotes inside, single-quotes outside
set myCMDline = "bsub -q $bsubQname -n 8 -R $bsubResources $myCMDline"
.
set bsubResources = "span[hosts=1]" #double-quotes inside, single-quotes outside
set myCMDline = 'bsub -q $bsubQname -n 8 -R "$bsubResources" $myCMDline'
.
set bsubResources = "span[hosts=1]" #double-quotes inside, single-quotes outside
set myCMDline = "bsub -q $bsubQname -n 8 -R '$bsubResources' $myCMDline"
etc.
I have also tried dumping to a separate temp script file to source, but that contains the $variable names, not resolved equivalents as I would prefer, as I am doing set, not setenv, and prefer not to put these into shell vars.
First I could not echo the "#!/bin/csh -f" line, it seems to try and execute that rather than echo redirected into the temp script file, and dies.
rm -f $myCMDlineTmpFile
echo "#!/bin/csh -f > $myCMDlineTmpFile
echo "$myCMDline" >> $myCMDlineTmpFile
($myCMDlineTmpFile)
Then I tried multi-line echo, which is where I am seeing the local variable names go into the file rather than their contents:
/bin/cat > $myCMDlineTmpFile <<EOF
#!/bin/csh -f
$myCMDline
EOF
source $myCMDlineTmpFile
And then I am trying to instead use eval:
eval `echo "$myCMDline &" `
with and without the backticks etc, but complains about unknown variables for the queue name, resources etc.
Adding this echo always looks like what I want to be the commandline, between the >>> and <<<
echo "DEBUG - myCMDline= >>>$myCMDline<<<"
Please help me solve this puzzle...
set myCMDline = "bsub -q $bsubQname -n 8 -R \"$bsubResources\" $myCMDline"
($myCMDline)
This won't work because csh considers this as a single string, so it treats the whole string as one big program name. You have to define an array instead:
set myCMDline = (bsub -q $bsubQname -n 8 -R "$bsubResources" $myCMDline:gaq)
($myCMDline:gaq)
Explanation: The :gaq is a substitution quotes all strings in the list and keeps each list element intact. This is quite similar to "$#" in bash.
This is documented in History Substitution
g Apply the following modifier once to each word.
a (+) Apply the following modifier as many times as possible to a single word. `a' and `g' can be used together to apply a modifier globally. In the current implementation, using the `a' and `s' modifiers together can lead to an infinite loop. For example, `:as/f/ff/' will never terminate. This behavior might change in the future.
q Quote the substituted words, preventing further substitutions.
This is relevant due to the text in variable substitution:
The `:' modifiers described under History substitution, except for `:p', can be applied to the substitutions above. More than one may be used. (+) Braces may be needed to insulate a variable substitution from a literal colon just as with History substitution (q.v.); any modifiers must appear within the braces.
I need to take specific variables from one script and use them in a different script.
Example:
Original script:
VARA=4 # Some description of VARA
VARB=6 # Some description of VARB
SOMEOTHERVAR="Foo"
/call/to/some/program
I want to write a second script that needs VARA and VARB, but not SOMEOTHERVAR or the call to the program.
I can already do:
eval $(grep 'VARA=' origscript.sh)
eval $(grep 'VARB=' origscript.sh)
This seems to work, but when I want to do both, like this, it only sets the first:
eval $(grep 'VAR[AB]=' origscript.sh)
because it seems to concatenate the two lines that grep returns. (Which probably means that the comments save the first assignments.)
Put quotes around it, so that the newlines in the output of grep will not be turned into spaces.
eval "$(grep 'VAR[AB]=' origscript.sh)"
I have this code snippet:
userjobs=$(grep -rw "$USER" /my/job/dir/|awk '{print $1}'|sort|uniq|rev|cut -c 2-|rev)
for job in "${userjobs[#]}"; do
cat "$job"
done
exit 0
When I run it as is, I get the following output:
cat: /my/job/dir/45
/my/job/dir/46: No such file or directory
However, if I unquote $job, I no longer receive this behavior, and it cats each of the files as expected.
I've done some reading up on globbingand splitting to see if this is occurring, but it seems like double-quoting should prevent that from happening. Can anyone explain why the behavior is different between "$job" and $job?
This happens because your variable looks like:
userjobs='/my/job/dir/45
/my/job/dir/46'
If you expand it as an array, with "${userjobs[#]}", that it acts as an array with exactly one element -- that string. Thus, behavior is identical to:
userjobs=( [0]='/my/job/dir/45
/my/job/dir/46' )
...still exactly one string with a literal newline in it.
Thus, cat "$job" looks for a file with a literal newline in its name.
To load your result into a real array you can iterate over with "${userjobs[#]}" expanding to a distinct element per line, use:
readarray -t userjobs < <(grep ...)
userjobs needs to be an array. Put parentheses around the value when assigning it:
userjobs=($(grep -rw "$USER" /my/job/dir/|awk '{print $1}'|sort|uniq|rev|cut -c 2-|rev))
I am trying to modify the variables of one shell script, using another script. This is what I have so far:
script1.sh
#!/bin/bash
var=123.45.67.890
script2.sh
#!/bin/bash
currVar=000.00.00.000
. /./script1.sh
var=$currVar
I understand that I am not modifying Script 1 here, but simply temporarily modifying var. How can I modify this var in script 1, via script 2?
Solution
. /./script1.sh
echo $var | sed "s/$var/$currVar/g" /./script1.sh > "temp.txt" && mv temp.txt /./script1.sh
Just use sed in 2nd script (script2.sh) as
currVar="000.00.00.000"
sed -r -i.bak "s/var=([[:graph:]]+)/var=$currVar/" script1.sh
var=000.00.00.000
where [[:graph:]] is a character class for [[:alnum:]] & [[:punct:]] to match values for var with printable characters/meta-characters.
Since you mentioned it is a proper IP address, use a proper regEx as
sed -r "s/(\b[0-9]{1,3}\.){3}[0-9]{1,3}\b/$currVar/" script1.sh
var=000.00.00.000
(\b[0-9]{1,3}\.){3}[0-9]{1,3} implies match 3 groups consisting of digits from 0-9, which each group could have from 1-3 digits each, preceded by a dot . and the 4th group also the same as the last. Remember the each group I am mentioning represents an IP octet
Normally, variables in scripts have local scope. If you export the variable, you can extend this scope to include any child processes. However, it looks like you might want to use the modified value when script1.sh runs. If that is the case, you can use the new var value as an input to script1.sh when you run it.
if [[ -z "$1" ]];
then
var=$1
else
var=123.45.67.890
This will check if you gave any parameters when you ran script1.sh, and if you did, then it should set your var equal to this value instead of the default IP.