What is the meaning of "${psql[#]}" in this script? - bash

I came across a script that is supposed to set up postgis in a docker container, but it references this "${psql[#]}" command in several places:
#!/bin/sh
# Perform all actions as $POSTGRES_USER
export PGUSER="$POSTGRES_USER"
# Create the 'template_postgis' template db
"${psql[#]}" <<- 'EOSQL'
CREATE DATABASE template_postgis;
UPDATE pg_database SET datistemplate = TRUE WHERE datname = 'template_postgis';
EOSQL
I'm guessing it's supposed to use the psql command, but the command is always empty so it gives an error. Replacing it with psql makes the script run as expected. Is my guess correct?
Edit: In case it's important, the command is being run in a container based on postgres:11-alpine.

$psql is supposed to be an array containing the psql command and its arguments.
The script is apparently expected to be run from here, which does
psql=( psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --no-password )
and later sources the script in this loop:
for f in /docker-entrypoint-initdb.d/*; do
case "$f" in
*.sh)
# https://github.com/docker-library/postgres/issues/450#issuecomment-393167936
# https://github.com/docker-library/postgres/pull/452
if [ -x "$f" ]; then
echo "$0: running $f"
"$f"
else
echo "$0: sourcing $f"
. "$f"
fi
;;
*.sql) echo "$0: running $f"; "${psql[#]}" -f "$f"; echo ;;
*.sql.gz) echo "$0: running $f"; gunzip -c "$f" | "${psql[#]}"; echo ;;
*) echo "$0: ignoring $f" ;;
esac
echo
done
See Setting an argument with bash for the reason to use an array rather than a string.

The #!/bin/sh and the [#] are incongruous. This is a bash-ism, where the psql variable is an array. This literal quote dollarsign psql bracket at bracket quote is expanded into "psql" "array" "values" "each" "listed" "and" "quoted" "separately." It's the safer way, e.g., to accumulate arguments to a command where any of them might have spaces in them.
psql=(/foo/psql arg arg arg) is the best way to define the array you need there.

It might look obscure, but it would work like so...
Let's say we have a bash array wc, which contains a command wc, and an argument -w, and we feed that a here document with some words:
wc=(wc -w)
"${wc[#]}" <<- words
one
two three
four
words
Since there are four words in the here document, the output is:
4
In the quoted code, there needs to be some prior point, (perhaps a calling script), that does something like:
psql=(psql -option1 -option2 arg1 arg2 ... )
As to why the programmer chose to invoke a command with an array, rather than just invoke the command, I can only guess... Maybe it's a crude sort of operator overloading to compensate for different *nix distros, (i.e. BSD vs. Linux), where the local variants of some necessary command might have different names from the same option, or even use different commands. So one might check for BSD or Linux or a given version, and reset psql accordingly.

The answer from #Barmar is correct.
The script was intended to be "sourced" and not "executed".
I faced the same problem and came to the same answer after I read that it had been reported here and fixed by "chmod".
https://github.com/postgis/docker-postgis/issues/119
Therefore, the fix is to change the permissions.
This can be done either in your git repository:
chmod -x initdb-postgis.sh
or add a line to your docker file.
RUN chmod -x /docker-entrypoint-initdb.d/10_postgis.sh
I like to do both so that it is clear to others.
Note: if you are using git on windows then permission can be lost. Therefore, "chmod" in the docker file is needed.

Related

bash "$#" not working with arguments starting with '-'

I am working on an option driven bash script that will use getopts. The script has cases where it can accept multiple options and specific cases where only one option is accepted. While testing a few cases out I ran into this issue which I have reduced down to pseudo-code for this question.
for arg in "$#"; do
echo ${arg}
done
echo "end"
Running below returns:
$ ./test.sh -a -b
-a
end
I am running bash 4.1.2, why isn't the -b returned on the empty line? I assume this has to do with the '-'.
I cannot reproduce your exact error, but this is the risk of using echo: if $arg is a valid option, it will be treated as such, not a string to print. Use printf instead:
printf '%s\n' "$arg"
Also check if you have applied any "shift" commands that might remove the arguments before you test then (typical in a argument collection block that might include a case statement)

How to iterate over double-quoted strings in POSIX shell?

I am trying to check if all the non POSIX commands that my script depends on are present before my script proceeds with its main job. This will help me to ensure that my script does not generate errors later due to missing commands.
I want to keep the list of all such non POSIX commands in a variable called DEPS so that as the script evolves and depends on more commands, I can edit this variable.
I want the script to support commands with spaces in them, e.g. my program.
This is my script.
#!/bin/sh
DEPS='ssh scp "my program" sftp'
for i in $DEPS
do
echo "Checking $i ..."
if ! command -v "$i"
then
echo "Error: $i not found"
else
echo "Success: $i found"
fi
echo
done
However, this doesn't work, because "my program" is split into two words while the for loop iterates: "my and program" as you can see in the output below.
# sh foo.sh
Checking ssh ...
/usr/bin/ssh
Success: ssh found
Checking scp ...
/usr/bin/scp
Success: scp found
Checking "my ...
Error: "my not found
Checking program" ...
Error: program" not found
Checking sftp ...
/usr/bin/sftp
Success: sftp found
The output I expected is:
# sh foo.sh
Checking ssh ...
/usr/bin/ssh
Success: ssh found
Checking scp ...
/usr/bin/scp
Success: scp found
Checking my program ...
Error: my program not found
Checking sftp ...
/usr/bin/sftp
Success: sftp found
How can I solve this problem while keeping the script POSIX compliant?
I'll repeat the answer I gave to your previous question: use a while loop with a here document rather than a for loop. You can embed newlines in a string, which is all you need to separate command names in a string if those command names might contain whitespace. (If your command names contain newlines, strongly consider renaming them.)
For maximum POSIX compatibility, use printf, since the POSIX specification of echo is remarkably lax due to differences in how echo was implemented in various shells prior to the definition of the standard.
deps="ssh
scp
my program
sftp
"
while read -r cmd; do
printf "Checking $cmd ...\n"
if ! command -v "$cmd"; then
printf "Error: $i not found\n"
else
printf "Success: $cmd found\n"
fi
printf "\n"
done <<EOF
$deps
EOF
This happens because the steps after parameter expansion are string-splitting and glob-expansion -- not syntax-level parsing (such as handling quoting). To go all the way back to the beginning of the parsing process, you need to use eval.
Frankly, the best approaches are to either:
Target a shell that supports arrays (ksh, bash, zsh, etc) rather than trying to support POSIX
Don't try to retrieve the value from a variable.
...there's a reason proper array support is ubiquitous in modern shells; writing unambiguously correct code, particularly when handling untrusted data, is much harder without it.
That said, you have the option of using $# to store your contents, which can be set, albeit dangerously, using eval:
deps='goodbye "cruel world"'
eval "set -- $deps"
for program; do
echo "processing $program"
done
If you do this inside of a function, you'll override only the function's argument list, leaving the global list unmodified.
Alternately, eval "yourfunction $deps" will have the same effect, setting the argument list within the function to the results of running all the usual parsing and expansion phases on the contents of $deps.
Because the script is in your controll, you can use the eval with reasonable safety, so #Charles Duffy's answer is an simple and good solution. Use it. :)
Also, consider to use the autoconf for generating the usual configure script what is doing good job for what you need - e.g. checking commands and much more... At least, check some configure scripts for ideas how to solvle common problems...
If you want play with your own implementation:
divide the dependecies into two groups
core_deps - unix tools, what are commonly needed for the script itself, like sed, cat cp and such. Those programs doesn't contains spaces in their names, nor in the $PATH.
runtime_deps - programs, what are needed for your application, but not for the script itself.
do the checks in two steps (or more, for example if you need check e.g. libraries)
never use the for loop for space delimited elements unless you getting them as the function arguments - so you can use the "$#"
As starting script could be something like the following:
_check_core_deps() {
for _cmd
do
_cpath=$(command -v "$_cmd")
case "$_cpath" in
/*) continue;;
*) echo "Missing install dependency [$_cmd] - can't continue" ; exit 1 ;;
esac
done
return 0
}
core_deps="grep sed hooloovoo cp" #list of "core" commands - they doesn't contains spaces
_check_core_deps $core_deps || exit 1
The above will blow up on non-existent "hooloovoo" command. :)
Now you can safely continue, all core commands needed for the install script are available. In the next step, you can check other strange dependencies.
Some ideas:
# function what returns your dependecies as lines from HEREDOC
# (e.g. could contain any character except "\n")
# you can decorate the dependecies with comments...
# because we have sed (checked in the 1st step, can use it)
# if want, you can add "fields" too, for some extended functinality with an specified delimiter
list_deps() {
_sptab=$(printf " \t") # the $' \t' is approved by POSIX for the next version only
#the "sed" removes comments and empty lines
#the UUOC (useless use of cat) is intentional here
#for example if you want add "tr" before the "sed"
#of course, you can remove it...
cat - <<DEPS |sed "s/[$_sptab]*#.*//;/^[$_sptab]*$/d"
########## DEPENDECIES ############
#some comment
ssh
scp
sftp
#comment
#bla bla
my program #some comment
/Applications/Some Long And Spaced OSX Apllication.app
DEPS
########## END of DEPENDECIES #####
}
_check_deps() {
#in the "while" loop you can use IFS=: or such and adding anouter variable to read
#for getting more fields for some extended functionality
list_deps | while read -r line
do
#do any checks with the line
#implement additional functionalities as functions
#etc...
#remember - your in an subshell here
printf "command:%s\n" "$line"
done
}
_check_deps
One more thing :), (or two)
if you doubt about the content of some variables, don't use the echo. The POSIX isn't defines how it should act when contains escaped characters (e.g. echo "some\nwed"). Use:
printf '%s' "$variable"
never use uppercase only variables like "DEPS"... they're only for environment variables...

Specify command line arguments like name=value pairs for shell script

Is it possible to pass command line arguments to shell script as name value pairs, something like
myscript action=build module=core
and then in my script, get the variable like
$action and process it?
I know that $1....and so on can be used to get variables, but then won't be name value like pairs. Even if they are, then the developer using the script will have to take care of declaring variables in the same order. I do not want that.
This worked for me:
for ARGUMENT in "$#"
do
KEY=$(echo $ARGUMENT | cut -f1 -d=)
KEY_LENGTH=${#KEY}
VALUE="${ARGUMENT:$KEY_LENGTH+1}"
export "$KEY"="$VALUE"
done
# from this line, you could use your variables as you need
cd $FOLDER
mkdir $REPOSITORY_NAME
Usage
bash my_scripts.sh FOLDER="/tmp/foo" REPOSITORY_NAME="stackexchange"
STEPS and REPOSITORY_NAME are ready to use in the script.
It does not matter what order the arguments are in.
Changelog
v1.0.0
In the Bourne shell, there is a seldom-used option '-k' which automatically places any values specified as name=value on the command line into the environment. Of course, the Bourne/Korn/POSIX shell family (including bash) also do that for name=value items before the command name:
name1=value1 name2=value2 command name3=value3 -x name4=value4 abc
Under normal POSIX-shell behaviour, the command is invoked with name1 and name2 in the environment, and with four arguments. Under the Bourne (and Korn and bash, but not POSIX) shell -k option, it is invoked with name1, name2, name3, and name4 in the environment and just two arguments. The bash manual page (as in man bash) doesn't mention the equivalent of -k but it works like the Bourne and Korn shells do.
I don't think I've ever used it (the -k option) seriously.
There is no way to tell from within the script (command) that the environment variables were specified solely for this command; they are simply environment variables in the environment of that script.
This is the closest approach I know of to what you are asking for. I do not think anything equivalent exists for the C shell family. I don't know of any other argument parser that sets variables from name=value pairs on the command line.
With some fairly major caveats (it is relatively easy to do for simple values, but hard to deal with values containing shell meta-characters), you can do:
case $1 in
(*=*) eval $1;;
esac
This is not the C shell family. The eval effectively does the shell assignment.
arg=name1=value1
echo $name1
eval $arg
echo $name1
env action=build module=core myscript
You said you're using tcsh. For Bourne-based shells, you can drop the "env", though it's harmless to leave it there. Note that this applies to the shell from which you run the command, not to the shell used to implement myscript.
If you specifically want the name=value pairs to follow the command name, you'll need to do some work inside myscript.
It's quite an old question, but still valid
I have not found the cookie cut solution. I combined the above answers. For my needs I created this solution; this works even with white space in the argument's value.
Save this as argparse.sh
#!/bin/bash
: ${1?
'Usage:
$0 --<key1>="<val1a> <val1b>" [ --<key2>="<val2a> <val2b>" | --<key3>="<val3>" ]'
}
declare -A args
while [[ "$#" > "0" ]]; do
case "$1" in
(*=*)
_key="${1%%=*}" && _key="${_key/--/}" && _val="${1#*=}"
args[${_key}]="${_val}"
(>&2 echo -e "key:val => ${_key}:${_val}")
;;
esac
shift
done
(>&2 echo -e "Total args: ${#args[#]}; Options: ${args[#]}")
## This additional can check for specific key
[[ -n "${args['path']+1}" ]] && (>&2 echo -e "key: 'path' exists") || (>&2 echo -e "key: 'path' does NOT exists");
#Example: Note, arguments to the script can have optional prefix --
./argparse.sh --x="blah"
./argparse.sh --x="blah" --yy="qwert bye"
./argparse.sh x="blah" yy="qwert bye"
Some interesting use cases for this script:
./argparse.sh --path="$(ls -1)"
./argparse.sh --path="$(ls -d -1 "$PWD"/**)"
Above script created as gist, Refer: argparse.sh
Extending on Jonathan's answer, this worked nicely for me:
#!/bin/bash
if [ "$#" -eq "0" ]; then
echo "Error! Usage: Remind me how this works again ..."
exit 1
fi
while [[ "$#" > "0" ]]
do
case $1 in
(*=*) eval $1;;
esac
shift
done

BASH Variables with multiple commands and reentrant

I have a bash script that sources contents from another file. The contents of the other file are commands I would like to execute and compare the return value. Some of the commands are have multiple commands separated by either a semicolon (;) or by ampersands (&&) and I can't seem to make this work. To work on this, I created some test scripts as shown:
test.conf is the file being sourced by test
Example-1 (this works), My output is 2 seconds in difference
test.conf
CMD[1]="date"
test.sh
. test.conf
i=2
echo "$(${CMD[$i]})"
sleep 2
echo "$(${CMD[$i]})"
Example-2 (this does not work)
test.conf (same script as above)
CMD[1]="date;date"
Example-3 (tried this, it does not work either)
test.conf (same script as above)
CMD[1]="date && date"
I don't want my variable, CMD, to be inside tick marks because then, the commands would be executed at time of invocation of the source and I see no way of re-evaluating the variable.
This script essentially calls CMD on pass-1 to check something, if on pass-1 I get a false reading, I do some work in the script to correct the false reading and re-execute & re-evaluate the output of CMD; pass-2.
Here is an example. Here I'm checking to see if SSHD is running. If it's not running when I evaluate CMD[1] on pass-1, I will start it and re-evaluate CMD[1] again.
test.conf
CMD[1]=`pgrep -u root -d , sshd 1>/dev/null; echo $?`
So if I modify this for my test script, then test.conf becomes:
NOTE: Tick marks are not showing up but it's the key below the ~ mark on my keyboard.
CMD[1]=`date;date` or `date && date`
My script looks like this (to handle the tick marks)
. test.conf
i=2
echo "${CMD[$i]}"
sleep 2
echo "${CMD[$i]}"
I get the same date/time printed twice despite the 2 second delay. As such, CMD is not getting re-evaluate.
First of all, you should never use backticks unless you need to be compatible with an old shell that doesn't support $() - and only then.
Secondly, I don't understand why you're setting CMD[1] but then calling CMD[$i] with i set to 2.
Anyway, this is one way (and it's similar to part of Barry's answer):
CMD[1]='$(date;date)' # no backticks (remember - they carry Lime disease)
eval echo "${CMD[1]}" # or $i instead of 1
From the couple of lines of your question, I would have expected some approach like this:
#!/bin/bash
while read -r line; do
# munge $line
if eval "$line"; then
# success
else
# fail
fi
done
Where you have backticks in the source, you'll have to escape them to avoid evaluating them too early. Also, backticks aren't the only way to evaluate code - there is eval, as shown above. Maybe it's eval that you were looking for?
For example, this line:
CMD[1]=`pgrep -u root -d , sshd 1>/dev/null; echo $?`
Ought probably look more like this:
CMD[1]='`pgrep -u root -d , sshd 1>/dev/null; echo $?`'

Preserving whitespaces in a string as a command line argument

I'm facing a small problem here, I want to pass a string containing whitespaces , to another program such that the whole string is treated as a command line argument.
In short I want to execute a command of the following structure through a bash shell script:
command_name -a arg1 -b arg2 -c "arg with whitespaces here"
But no matter how I try, the whitespaces are not preserved in the string, and is tokenized by default. A solution please,
edit: This is the main part of my script:
#!/bin/bash
#-------- BLACKRAY CONFIG ---------------#
# Make sure the current user is in the sudoers list
# Running all instances with sudo
BLACKRAY_BIN_PATH='/opt/blackray/bin'
BLACKRAY_LOADER_DEF_PATH='/home/crozzfire'
BLACKRAY_LOADER_DEF_NAME='load.xml'
BLACKRAY_CSV_PATH='/home/crozzfire'
BLACKRAY_END_POINT='default -p 8890'
OUT_FILE='/tmp/out.log'
echo "The current binary path is $BLACKRAY_BIN_PATH"
# Starting the blackray 0.9.0 server
sudo "$BLACKRAY_BIN_PATH/blackray_start"
# Starting the blackray loader utility
BLACKRAY_INDEX_CMD="$BLACKRAY_BIN_PATH/blackray_loader -c $BLACKRAY_LOADER_DEF_PATH/$BLACKRAY_LOADER_DEF_NAME -d $BLACKRAY_CSV_PATH -e "\"$BLACKRAY_END_POINT\"""
sudo time $BLACKRAY_INDEX_CMD -a $OUT_FILE
#--------- END BLACKRAY CONFIG ---------#
You're running into this problem because you store the command in a variable, then expand it later; unless there's a good reason to do this, don't:
sudo time $BLACKRAY_BIN_PATH/blackray_loader -c $BLACKRAY_LOADER_DEF_PATH/$BLACKRAY_LOADER_DEF_NAME -d $BLACKRAY_CSV_PATH -e "$BLACKRAY_END_POINT" -a $OUT_FILE
If you really do need to store the command and use it later, there are several options; the bash-hackers.org wiki has a good page on the subject. It looks to me like the most useful one here is to put the command in an array rather than a simple variable:
BLACKRAY_INDEX_CMD=($BLACKRAY_BIN_PATH/blackray_loader -c $BLACKRAY_LOADER_DEF_PATH/$BLACKRAY_LOADER_DEF_NAME -d $BLACKRAY_CSV_PATH -e "$BLACKRAY_END_POINT")
sudo time "${BLACKRAY_INDEX_CMD[#]}" -a $OUT_FILE
This avoids the whole confusion between spaces-separating-words and spaces-within-words because words aren't separated by spaces -- they're in separate elements of the array. Expanding the array in double-quotes with the [#] suffix preserves that structure.
(BTW, another option would be to use escaped quotes rather like you're doing, then run the command with eval. Don't do this; it's a good way to introduce weird parsing bugs.)
Edit:
Try:
BLACKRAY_END_POINT="'default -p 8890'"
or
BLACKRAY_END_POINT='"default -p 8890"'
or
BLACKRAY_END_POINT="default\ -p\ 8890"
or
BLACKRAY_END_POINT='default\ -p\ 8890'
and
BLACKRAY_INDEX_CMD="$BLACKRAY_BIN_PATH/blackray_loader -c $BLACKRAY_LOADER_DEF_PATH/$BLACKRAY_LOADER_DEF_NAME -d $BLACKRAY_CSV_PATH -e $BLACKRAY_END_POINT"
Original answer:
Is blackray_loader a shell script?
Here is a demonstration that you have to deal with this issue both when specifying the parameter and when handling it:
A text file called "test.txt" (include the line numbers):
1 two words
2 two words
3 two
4 words
A script called "spacetest":
#!/bin/bash
echo "No quotes in script"
echo $1
grep $1 test.txt
echo
echo "With quotes in script"
echo "$1"
grep "$1" test.txt
echo
Running it with ./spacetest "two--------words" (replace the hyphens with spaces):
No quotes in script
two words
grep: words: No such file or directory
test.txt:1 two words
test.txt:2 two words
test.txt:3 two
With quotes in script
two words
2 two words
You can see that in the "No quotes" section it tried to do grep two words test.txt which interpreted "words" as a filename in addition to "test.txt". Also, the echo dropped the extra spaces.
When the parameter is quoted, as in the second section, grep saw it as one argument (including the extra spaces) and handled it correctly. And echo preserved the extra spaces.
I used the extra spaces, by the way, merely to aid in the demonstration.
I have a suggestion:
# iterate through the passed arguments, save them to new properly quoted ARGS string
while [ -n "$1" ]; do
ARGS="$ARGS '$1'"
shift
done
# invoke the command with properly quoted arguments
my_command $ARGS
probably you need to surround the argument by double quotes (e.g. "${6}").
Following OP comment it should be "$BLACKRAY_END_POINT"
Below is my example of restarting a script via exec su USER or exec su - USER. It accommodates:
being called from a relative path or current working directory
spaces in script name and arguments
single and double-quotes in arguments, without crazy escapes like: \\"
#
# This script should always be run-as a specific user
#
user=jimbob
if [ $(whoami) != "$user" ]; then
exec su -c "'$(readlink -f "$0")' $(printf " %q" "$#")" - $user
exit $?
fi
A post on other blog saved me for this whitespaces problem: http://logbuffer.wordpress.com/2010/09/23/bash-scripting-preserve-whitespaces-in-variables/
By default, whitespaces are trimed:
bash> VAR1="abc def gh ijk"
bash> echo $VAR1
abc def gh ijk
bash>
"The cause of this behaviour is the internal shell variable $IFS (Internal Field Separator), that defaults to whitespace, tab and newline.
To preserve all contiguous whitespaces you have to set the IFS to something different"
With IFS bypass:
bash> IFS='%'
bash> echo $VAR1
abc def gh ijk
bash>unset IFS
bash>
It works wonderfully for my command case:
su - user1 -c 'test -r "'${filepath}'"; ....'
Hope this helps.

Resources