Bash script with multiline variable - bash

Here is my code
vmname="$1"
EXCEPTLIST="desktop-01|desktop-02|desktop-03|desktop-04"
if [[ $vmname != #(${EXCEPTLIST}) ]]; then
echo "${vmname}"
else
echo "Its in the exceptlist"
fi
The above code works perfectly but my question is , the EXCEPTLIST can be a long line, say 100 server names. In that case its hard to put all that names in one line. In that situation is there any way to make the variable EXCEPTLIST to be a multiline variable ? something like as follows:
EXCEPTLIST="desktop-01|desktop-02|desktop-03| \n
desktop-04|desktop-05|desktop-06| \n
desktop-07|desktop-08"
I am not sure but was thinking of possibilities.
Apparently I would like to know the terminology of using #(${})- Is this called variable expansion or what ? Does anyone know the documentation/explain to me about how this works in bash. ?

One can declare an array if the data/string is long/large. Use IFS and printf for the format string, something like:
#!/usr/bin/env bash
exceptlist=(
desktop-01
desktop-02
desktop-03
desktop-04
desktop-05
desktop-06
)
pattern=$(IFS='|'; printf '#(%s)' "${exceptlist[*]}")
[[ "$vmname" != $pattern ]] && echo good
In that situation is there any way to make the variable EXCEPTLIST to be a multiline variable ?
With your given input/data an array is also a best option, something like:
exceptlist=(
'desktop-01|desktop-02|desktop-03'
'desktop-04|desktop-05|desktop-06'
'desktop-07|desktop-08'
)
Check what is the value of $pattern variable one way is:
declare -p pattern
Output:
declare -- pattern="#(desktop-01|desktop-02|desktop-03|desktop-04|desktop-05|desktop-06)"
Need to test/check if $vmname is an empty string too, since it will always be true.
On a side note, don't use all upper case variables for purely internal purposes.
The $(...) is called Command Substitution.
See LESS=+'/\ *Command Substitution' man bash
In addition to what was mentioned in the comments about pattern matching
See LESS=+/'(pattern-list)' man bash
See LESS=+/' *\[\[ expression' man bash

s there any way to make the variable EXCEPTLIST to be a multiline variable ?
I see no reason to use matching. Use a bash array and just compare.
exceptlist=(
desktop-01
desktop-02
desktop-03
desktop-04
desktop-05
desktop-06
)
is_in_list() {
local i
for i in "${#:2}"; do
if [[ "$1" = "$i" ]]; then
return 0
fi
done
return 1
}
if is_in_list "$vmname" "${EXCEPTLIST[#]}"; then
echo "is in exception list ${vmname}"
fi
#(${})- Is this called variable expansion or what ? Does anyone know the documentation/explain to me about how this works in bash. ?
${var} is a variable expansion.
#(...) are just characters # ( ).
From man bash in Compund commands:
[[ expression ]]
When the == and != operators are used, the string to the right of the operator is considered a pattern and matched according to the rules
described below under Pattern Matching, as if the extglob shell option were enabled. ...
From Pattern Matching in man bash:
#(pattern-list)
Matches one of the given patterns
[[ command receives the #(a|b|c) string and then matches the arguments.

There is absolutely no need to use Bash specific regex or arrays and loop for a match, if using grep for raw string on word boundary.
The exception list can be multi-line, it will work as well:
#!/usr/bin/sh
exceptlist='
desktop-01|desktop-02|desktop-03|
deskop-04|desktop-05|desktop-06|
desktop-07|deskop-08'
if printf %s "$exceptlist" | grep -qwF "$1"; then
printf '%s is in the exceptlist\n' "$1"
fi

I wouldn't bother with multiple lines of text. This is would be just fine:
EXCEPTLIST='desktop-01|desktop-02|desktop-03|'
EXCEPTLIST+='desktop-04|desktop-05|desktop-06|'
EXCEPTLIST+='desktop-07|desktop-08'
The #(...) construct is called extended globbing pattern and what it does is an extension of what you probably already know -- wildcards:
VAR='foobar'
if [[ "$VAR" == fo?b* ]]; then
echo "Yes!"
else
echo "No!"
fi
A quick walkthrough on extended globbing examples: https://www.linuxjournal.com/content/bash-extended-globbing

#!/bin/bash
set +o posix
shopt -s extglob
vmname=$1
EXCEPTLIST=(
desktop-01 desktop-02 desktop-03
...
)
if IFS='|' eval '[[ ${vmname} == #(${EXCEPTLIST[*]}) ]]'; then
...

Here's one way to load a multiline string into a variable:
fn() {
cat <<EOF
desktop-01|desktop-02|desktop-03|
desktop-04|desktop-05|desktop-06|
desktop-07|desktop-08
EOF
}
exceptlist="$(fn)"
echo $exceptlist
As to solving your specific problem, I can think of a variety of approaches.
Solution 1, since all the desktop has the same desktop-0 prefix and only differ in the last letter, we can make use of {,} or {..} expansion as follows:
vmname="$1"
found=0
for d in desktop-{01..08}
do
if [[ "$vmname" == $d ]]; then
echo "It's in the exceptlist"
found=1
break
fi
done
if (( !found )); then
echo "Not found"
fi
Solution 2, sometimes, it is good to provide a list in a maintainable clear text list. We can use a while loop and iterate through the list
vmname="$1"
found=0
while IFS= read -r d
do
if [[ "$vmname" == $d ]]; then
echo "It's in the exceptlist"
found=1
break
fi
done <<EOF
desktop-01
desktop-02
desktop-03
desktop-04
desktop-05
desktop-06
desktop-07
desktop-08
EOF
if (( !found )); then
echo "Not found"
fi
Solution 3, we can desktop the servers using regular expressions:
vmname="$1"
if [[ "$vmname" =~ ^desktop-0[1-8]$ ]]; then
echo "It's in the exceptlist"
else
echo "Not found"
fi
Solution 4, we populate an array, then iterate through an array:
vmname="$1"
exceptlist=()
exceptlist+=(desktop-01 desktop-02 desktop-03 deskop-04)
exceptlist+=(desktop-05 desktop-06 desktop-07 deskop-08)
found=0
for d in ${exceptlist[#]}
do
if [[ "$vmname" == "$d" ]]; then
echo "It's in the exceptlist"
found=1
break;
fi
done
if (( !found )); then
echo "Not found"
fi

Related

Looping over shell script arguments and passing quoted arguments to function

I have a script below that sources a directory of bash scripts and then parses the flags of the command to run a specific function from the sourced files.
Given this function within the scripts dir:
function reggiEcho () {
echo $1
}
Here are some examples of current output
$ reggi --echo hello
hello
$ reggi --echo hello world
hello
$ reggi --echo "hello world"
hello
$ reggi --echo "hello" --echo "world"
hello
world
As you can see quoted parameters are not honored as they should be `"hello world" should echo properly.
This is the script, the issue is within the while loop.
How do I parse these flags, and maintain passing in quoted parameters into the function?
DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
STR="$(find $DIR/scripts -type f -name '*.sh' -print)"
ARR=( $STR )
TUSAGE="\n"
for f in "${ARR[#]}"; do
if [ -f $f ]
then
. $f --source-only
if [ -z "$USAGE" ]
then
:
else
TUSAGE="$TUSAGE \t$USAGE\n"
fi
USAGE=""
else
echo "$f not found"
fi
done
TUSAGE="$TUSAGE \t--help (shows this help output)\n"
function usage() {
echo "Usage: --function <args> [--function <args>]"
echo $TUSAGE
exit 1
}
HELP=false
cmd=()
while [ $# -gt 0 ]; do # loop until no args left
if [[ $1 = '--help' ]] || [[ $1 = '-h' ]] || [[ $1 = '--h' ]] || [[ $1 = '-help' ]]; then
HELP=true
fi
if [[ $1 = --* ]] || [[ $1 = -* ]]; then # arg starts with --
if [[ ${#cmd[#]} -gt 0 ]]; then
"${cmd[#]}"
fi
top=`echo $1 | tr -d -` # remove all flags
top=`echo ${top:0:1} | tr '[a-z]' '[A-Z]'`${top:1} # make sure first letter is uppercase
top=reggi$top # prepend reggi
cmd=( "$top" ) # start new array
else
echo $1
cmd+=( "$1" )
fi
shift
done
if [[ "$HELP" = true ]]; then
usage
elif [[ ${#cmd[#]} -gt 0 ]]; then
${cmd[#]}
else
usage
fi
There are many places in this script where you have variable references without double-quotes around them. This means the variables' values will be subject to word spitting and wildcard expansion, which can have various weird effects.
The specific problem you're seeing is due to an unquoted variable reference on the fourth-from-last line, ${cmd[#]}. With cmd=( echo "hello world" ), word splitting makes this equivalent to echo hello world rather than echo "hello world".
Fixing that one line will fix your current problem, but there are a number of other unquoted variable references that may cause other problems later. I recommend fixing all of them. Cyrus' recommendation of shellcheck.net is good at pointing them out, and will also note some other issues I won't cover here. One thing it won't mention is that you should avoid all-caps variable names (DIR, TUSAGE, etc) -- there are a bunch of all-caps variables with special meanings, and it's easy to accidentally reuse one of them and wind up with weird effects. Lowercase and mixed-case variables are safer.
I also recommend against using \t and \n in strings, and counting on echo to translate them into tabs and newlines, respectively. Some versions of echo do this automatically, some require the -e option to tell them to do it, some will print "-e" as part of their output... it's a mess. In bash, you can use $'...' to translate those escape sequences directly, e.g:
tusage="$tusage"$' \t--help (shows this help output)\n' # Note mixed quoting modes
echo "$tusage" # Note that double-quoting is *required* for this to work right
You should also fix the file listing so it doesn't depend on being unquoted (see chepner's comment). If you don't need to scan subdirectories of $DIR/scripts, you can do this with a simple wildcard (note lowercase vars and that the var is double-quoted, but the wildcard isn't):
arr=( "$dir/scripts"/*.sh )
If you need to look in subdirectories, it's more complicated. If you have bash v4 you can use a globstar wildcard, like this:
shopt -s globstar
arr=( "$dir/scripts"/**/*.sh )
If your script might have to run under bash v3, see BashFAQ #20: "How can I find and safely handle file names containing newlines, spaces or both?", or just use this:
while IFS= read -r -d '' f <&3; do
if [ -f $f ]
# ... etc
done 3< <(find "$dir/scripts" -type f -name '*.sh' -print0)
(That's my favorite it-just-works idiom for iterating over find's matches. Although it does require bash, not some generic POSIX shell.)

Bash problems with string comparison

I have a problem with writing bash script. The problem is in comparison of strings. When I launch it, there's no errors. However in result, it is always changing the variable client.
So if for an example we have two lines in file
apple A
orange D
and if I give the who=A I expect to see in result apple, or if at D - orange
But no matter of what I choose A or D it is always giving me the result - orange
No matter of the strings, it always change the variable client, like ignoring the comparison. Please help.
while read line
do
IFS=" "
set -- $line
echo $2" "$who":"$1
if [[ "$2"="$who" ]]
then
echo "change"
client=$1
fi
done < $file
echo $client
So now I changed the code as in one of the comment below, but now the caparison always false therefore the variable client is always empty
while read -r line
do
#IFS=" "
#set -- $line
#echo $2" "$who":"$1
#if [[ "$2" = "$who" ]]
a="${line% *}"
l="${line#* }"
if [[ "$l" == "$who" ]]
then
echo "hi"
client="$a"
fi
done < $file
If you have data in a file with each line like apple D and you want to read the file and separate then items, the parameter expansion/substring extraction is the correct way to process the line. For example (note $who is taken from your problem statement):
while read -r line
do
fruit="${line% *}" # remove from end to space
letter="${line#* }" # remove from start to space
if [[ "$letter" == "$who" ]]
then
echo "change"
client="$fruit"
fi
done < $file
Short Example
Here is a quick example of splitting the words with parameter expansion/substring extraction:
#!/bin/bash
while read -r line
do
fruit="${line% *}"
letter="${line#* }"
echo "fruit: $fruit letter: $letter"
done
exit 0
input
$ cat dat/apple.txt
Apple A
Orange D
output
$ bash apple.sh <dat/apple.txt
fruit: Apple letter: A
fruit: Orange letter: D
Change if [[ "$2"="$who" ]] to
if [[ "$2" = "$who" ]]
spaces around =
Example (for clarification):
who=A
while read line
do
IFS=" "
set -- $line
echo $2" "$who":"$1
if [[ "$2" = "$who" ]]
then
echo "change"
client=$1
fi
done < file #this is the file I used for testing
echo $client
Output:
A A:apple
change
D A:orange
apple
For who=D:
A D:apple
D D:orange
change
orange
You do need spaces around that = operator.
However, I think you're facing yet another issue as you're trying to change the value of the client variable from inside the while loop (which executes in a subshell). I don't think that will work; see this quesion for details.

How to get first character of variable

I'm trying to get the first character of a variable, but I'm getting a Bad substitution error. Can anyone help me fix it?
code is:
while IFS=$'\n' read line
do
if [ ! ${line:0:1} == "#"] # Error on this line
then
eval echo "$line"
eval createSymlink $line
fi
done < /some/file.txt
Am I doing something wrong or is there a better way of doing this?
-- EDIT --
As requested - here's some sample input which is stored in /some/file.txt
$MOZ_HOME/mobile/android/chrome/content/browser.js
$MOZ_HOME/mobile/android/locales/en-US/chrome/browser.properties
$MOZ_HOME/mobile/android/components/ContentPermissionPrompt.js
To get the first character of a variable you need to say:
v="hello"
$ echo "${v:0:1}"
h
However, your code has a syntax error:
[ ! ${line:0:1} == "#"]
# ^-- missing space
So this can do the trick:
$ a="123456"
$ [ ! "${a:0:1}" == "#" ] && echo "doesnt start with #"
doesnt start with #
$ a="#123456"
$ [ ! "${a:0:1}" == "#" ] && echo "doesnt start with #"
$
Also it can be done like this:
$ a="#123456"
$ [ "$(expr substr $a 1 1)" != "#" ] && echo "does not start with #"
$
$ a="123456"
$ [ "$(expr substr $a 1 1)" != "#" ] && echo "does not start with #"
does not start with #
Update
Based on your update, this works to me:
while IFS=$'\n' read line
do
echo $line
if [ ! "${line:0:1}" == "#" ] # Error on this line
then
eval echo "$line"
eval createSymlink $line
fi
done < file
Adding the missing space (as suggested in fedorqui's answer ;) ) works for me.
An alternative method/syntax
Here's what I would do in Bash if I want to check the first character of a string
if [[ $line != "#"* ]]
On the right hand side of ==, the quoted part is treated literally whereas * is a wildcard for any sequence of character.
For more information, see the last part of Conditional Constructs of Bash reference manual:
When the ‘==’ and ‘!=’ operators are used, the string to the right of the operator is considered a pattern and matched according to the rules described below in Pattern Matching
Checking that you're using the right shell
If you are getting errors such as "Bad substitution error" and "[[: not found" (see comment) even though your syntax is fine (and works fine for others), it might indicate that you are using the wrong shell (i.e. not Bash).
So to make sure you are using Bash to run the script, either
make the script executable and use an appropriate shebang e.g. #!/bin/bash
or execute it via bash my_script
Also note that sh is not necessarily bash, sometimes it can be dash (e.g. in Ubuntu) or just plain ol' Bourne shell.
Try this:
while IFS=$'\n' read line
do
if ! [ "${line:0:1}" = "#" ]; then
eval echo "$line"
eval createSymlink $line
fi
done < /some/file.txt
or you can use the following for your if syntax:
if [[ ! ${line:0:1} == "#" ]]; then
TIMTOWTDI ^^
while IFS='' read -r line
do
case "${line}" in
"#"*) echo "${line}"
;;
*) createSymlink ${line}
;;
esac
done < /some/file.txt
Note: I dropped the eval, which could be needed in some (rare!) cases (and are dangerous usually).
Note2: I added a "safer" IFS & read (-r, raw) but you can revert to your own if it is better suited. Note that it still reads line by line.
Note3: I took the habit of using always ${var} instead of $var ... works for me (easy to find out vars in complex text, and easy to see where they begin and end at all times) but not necessary here.
Note4: you can also change the test to : *"#"*) if some of the (comments?) lines can have spaces or tabs before the '#' (and none of the symlink lines does contain a '#')

parse and expand interval

In my script I need to expand an interval, e.g.:
input: 1,5-7
to get something like the following:
output: 1,5,6,7
I've found other solutions here, but they involve python and I can't use it in my script.
Solution with Just Bash 4 Builtins
You can use Bash range expansions. For example, assuming you've already parsed your input you can perform a series of successive operations to transform your range into a comma-separated series. For example:
value1=1
value2='5-7'
value2=${value2/-/..}
value2=`eval echo {$value2}`
echo "input: $value1,${value2// /,}"
All the usual caveats about the dangers of eval apply, and you'd definitely be better off solving this problem in Perl, Ruby, Python, or AWK. If you can't or won't, then you should at least consider including some pipeline tools like tr or sed in your conversions to avoid the need for eval.
Try something like this:
#!/bin/bash
for f in ${1//,/ }; do
if [[ $f =~ - ]]; then
a+=( $(seq ${f%-*} 1 ${f#*-}) )
else
a+=( $f )
fi
done
a=${a[*]}
a=${a// /,}
echo $a
Edit: As #Maxim_united mentioned in the comments, appending might be preferable to re-creating the array over and over again.
This should work with multiple ranges too.
#! /bin/bash
input="1,5-7,13-18,22"
result_str=""
for num in $(tr ',' ' ' <<< "$input"); do
if [[ "$num" == *-* ]]; then
res=$(seq -s ',' $(sed -n 's#\([0-9]\+\)-\([0-9]\+\).*#\1 \2#p' <<< "$num"))
else
res="$num"
fi
result_str="$result_str,$res"
done
echo ${result_str:1}
Will produce the following output:
1,5,6,7,13,14,15,16,17,18,22
expand_commas()
{
local arg
local st en i
set -- ${1//,/ }
for arg
do
case $arg in
[0-9]*-[0-9]*)
st=${arg%-*}
en=${arg#*-}
for ((i = st; i <= en; i++))
do
echo $i
done
;;
*)
echo $arg
;;
esac
done
}
Usage:
result=$(expand_commas arg)
eg:
result=$(expand_commas 1,5-7,9-12,3)
echo $result
You'll have to turn the separated words back into commas, of course.
It's a bit fragile with bad inputs but it's entirely in bash.
Here's my stab at it:
input=1,5-7,10,17-20
IFS=, read -a chunks <<< "$input"
output=()
for chunk in "${chunks[#]}"
do
IFS=- read -a args <<< "$chunk"
if (( ${#args[#]} == 1 )) # single number
then
output+=(${args[*]})
else # range
output+=($(seq "${args[#]}"))
fi
done
joined=$(sed -e 's/ /,/g' <<< "${output[*]}")
echo $joined
Basically split on commas, then interpret each piece. Then join back together with commas at the end.
A generic bash solution using the sequence expression `{x..y}'
#!/bin/bash
function doIt() {
local inp="${#/,/ }"
declare -a args=( $(echo ${inp/-/..}) )
local item
local sep
for item in "${args[#]}"
do
case ${item} in
*..*) eval "for i in {${item}} ; do echo -n \${sep}\${i}; sep=, ; done";;
*) echo -n ${sep}${item};;
esac
sep=,
done
}
doIt "1,5-7"
Should work with any input following the sample in the question. Also with multiple occurrences of x-y
Use only bash builtins
Using ideas from both #Ansgar Wiechers and #CodeGnome:
input="1,5-7,13-18,22"
for s in ${input//,/ }
do
if [[ $f =~ - ]]
then
a+=( $(eval echo {${s//-/..}}) )
else
a+=( $s )
fi
done
oldIFS=$IFS; IFS=$','; echo "${a[*]}"; IFS=$oldIFS
Works in Bash 3
Considering all the other answers, I came up with this solution, which does not use any sub-shells (but one call to eval for brace expansion) or separate processes:
# range list is assumed to be in $1 (e.g. 1-3,5,9-13)
# convert $1 to an array of ranges ("1-3" "5" "9-13")
IFS=,
local range=($1)
unset IFS
list=() # initialize result list
local r
for r in "${range[#]}"; do
if [[ $r == *-* ]]; then
# if the range is of the form "x-y",
# * convert to a brace expression "{x..y}",
# * using eval, this gets expanded to "x" "x+1" … "y" and
# * append this to the list array
eval list+=( {${r/-/..}} )
else
# otherwise, it is a simple number and can be appended to the array
list+=($r)
fi
done
# test output
echo ${list[#]}

String manipulation, optional character

With grep, you can use a question mark ? to signify an optional character, that is a character that is to be matches 0 or 1 times.
$ foo=qwerasdf
$ grep -Eo fx? <<< $foo
f
The question is does Bash String Manipulation have a similar feature? Something like
$ echo ${foo%fx?}
You're probably talking about parameter expansion. It uses shell patterns, not regular expression, so the answer is no.
Upon further reading, I noticed that if you
shopt -s extglob
you can use extended pattern matching which can achieve something similar to regex, albeit with slightly different syntax.
Check this out:
word="mre"
# this returns true
if [[ $word == m?(o)re ]]; then echo true; else echo false; fi
word="more"
# this also returns true
if [[ $word == m?(o)re ]]; then echo true; else echo false; fi
word="mooooooooooore"
# again, true
if [[ $word == m+(o)re ]]; then echo true; else echo false; fi
Works with parameter expansion too,
word="noooooooooooo"
# outputs 'nay'
echo ${word/+(o)/ay}
# outputs 'nayooooooooooo'
echo ${word/o/ay}

Resources