My bash script writes an another bash script using printf.
printf "#!/bin/bash
HOME=${server}
file=gromacs*
file_name=\$(basename "\${file}")
date=\$(date +"\%m_\%d_\%Y")
for sim in \${HOME}/* ; do
if [[ -d \$sim ]]; then
simulation=$(basename "\$sim")
pushd \${sim}
cp \$file \${server}/\${results}/\${file_name}.\${simulation}.\${date}
echo "\${file_name}\ from\ \${simulation}\ has\ been\ collected!"
popd
fi
done" > ${output}/collecter.sh
Here there is a problem in escappiong of the elements within date variable
date=\$(date +"\%m_\%d_\%Y")
where the below part did not work properly
"\%m_\%d_\%Y"
it results in incomplete of a new bash script produced by printf.
How it should be fixed?
Thanks!
Use a quoted heredoc.
{
## print the header, and substitute our own value for HOME
printf '#!/bin/bash\nHOME=%q\n' "$server"
## EVERYTHING BELOW HERE UNTIL THE EOF IS LITERAL
cat <<'EOF'
file=( gromacs* )
(( ${#file[#]} == 1 )) && [[ -e $file ]] || {
echo "ERROR: Exactly one file starting with 'gromacs' should exist" >&2
exit 1
}
file_name=$(basename "${file}")
date=$(date +"%m_%d_%Y")
for sim in "$HOME"/* ; do
if [[ -d $sim ]]; then
simulation=$(basename "$sim")
(cd "${sim}" && exec cp "$file" "${server}/${results}/${file_name}.${simulation}.${date}")
echo "${file_name} from ${simulation} has been collected!"
fi
done
EOF
} >"${output}/collecter.sh"
Note:
Inside a quoted heredoc (cat <<'EOF'), no substitutions are performed, so no escaping is needed. We're thus able to write our code exactly as we want it to exist in the generated file.
When generating code, use printf %q to escape values in such a way as to evaluate back to their original values. Otherwise, a variable containing $(rm -rf ~) could cause the given command to be run (if it were substituted inside literal single quotes, making the contents $(rm -rf ~)'$(rm -rf ~)' would escape them).
Glob expansions return a list of results; the proper data type in which to store their results is an array, not a string. Thus, file=( gromacs* ) makes the storage of the result in an array explicit, and the following code checks for both the case where we have more than one result, and the case where our result is the original glob expression (meaning no matches existed).
All expansions need to be quoted to prevent string-splitting. This means "$HOME"/*, not $HOME/* -- otherwise you'll have problems whenever a user has a home directory containing whitespace (and yes, this does happen -- consider Windows-derived platforms where you have /Users/Firstname Lastname, or sites where you've mounted a volume for home directories off same).
pushd and popd are an interactive extension, as opposed to a tool intended for writing scripts. Since spawning an external program (such as cp) involves a fork() operation, and any directory change inside a subshell terminates when that subshell does, you can avoid any need for them by spawning a subshell, cd'ing within that subshell, and then using the exec builtin to replace the subshell's PID with that of cp, thus preventing the fork that would otherwise have taken place to allow cp to be started in a separate process from the shell acting as its parent.
You have to escapes in printf with esscaps, e. g.:
date=\$(date +"%%m_%%d_%%Y")
should print date=\$(date +"%m_%d_%Y"). But you should avoid using printf, because you don't use it's capabilities. Instead you could cat the string to the file:
cat > ${output}/collecter.sh <<'END'
HOME=${server}
...
done
END
This would allow you to avoid many escapes, and make the code more readable.
Try this
date=\$(date +\"%%m_%%d_%%Y\")"
Related
I have this function in my ~/.zshrc
async () {
if ! [[ $# -gt 0 ]];then
echo "Not enough arguments!"
fi
local busus="$IFS"
export IFS=" "
echo "$* &"
command "$* &"
export IFS="$busus"
}
and an alias
alias word='async "libreoffice --writer"'
The echo "$* &" line is used only for debugging.
When I run word, libreoffice --writer & is shown on the screen (no extra spaces or newlines), but nothing happens.
I also tried executing command libreoffice --writer & and it worked perfectly.
(My current shell is zsh)
What is wrong?
Thanks
Usually (especially in bash), the problem is that people aren't using enough double-quotes; in this case, it's the opposite: you're using too many double-quotes. The basic problem is that the command name and each of the arguments to it must be a separate "word" (in shell syntax), but double-quoting something will (usually) make the shell treat it as all one word. Here's a quick demo:
% echo foo
foo
% "echo foo"
zsh: command not found: echo foo
Here, the double-quotes make the shell treat " foo" as part of the command name, rather than as a delimiter and an argument after the command name. Similarly, when you use "$* &", the double-quotes tell the shell to treat the entire thing (including even the ampersand) as a single long word (and pass it as an argument to command). (BTW, the command isn't needed, but isn't causing any harm either.)
The standard way to do this is to use "$#" instead -- here the $# acts specially within double-quotes, making each argument into a separate word. In zsh, you could omit the double-quotes, but that can cause trouble in other shells so I recommend using them anyway.
Also, don't mess with IFS. You don't need to, and it opens a can of worms that's best left closed. And if there are no arguments, you should return immediately, rather than continuing and trying to run an empty command.
But there's another problem: in the alias, you double-quote "libreoffice --writer", which is going to have pretty much the same effect again. So remove those double-quotes. But keep the single-quotes around the alias, so it'll be defined as a single alias.
So here's my proposed correction:
async () {
if ! [[ $# -gt 0 ]];then
echo "Not enough arguments!"
return 1 # Do not continue if there's no command to run!
fi
echo "$* &" # Here quoting is appropriate, so it's a single argument to echo
"$#" &
}
alias word='async libreoffice --writer'
Using "$#" directly is more reliable:
async () { [ "$#" -gt 0 ] && "$#" & }
alias word='async libreoffice --writer'
I came across a script that is supposed to set up postgis in a docker container, but it references this "${psql[#]}" command in several places:
#!/bin/sh
# Perform all actions as $POSTGRES_USER
export PGUSER="$POSTGRES_USER"
# Create the 'template_postgis' template db
"${psql[#]}" <<- 'EOSQL'
CREATE DATABASE template_postgis;
UPDATE pg_database SET datistemplate = TRUE WHERE datname = 'template_postgis';
EOSQL
I'm guessing it's supposed to use the psql command, but the command is always empty so it gives an error. Replacing it with psql makes the script run as expected. Is my guess correct?
Edit: In case it's important, the command is being run in a container based on postgres:11-alpine.
$psql is supposed to be an array containing the psql command and its arguments.
The script is apparently expected to be run from here, which does
psql=( psql -v ON_ERROR_STOP=1 --username "$POSTGRES_USER" --no-password )
and later sources the script in this loop:
for f in /docker-entrypoint-initdb.d/*; do
case "$f" in
*.sh)
# https://github.com/docker-library/postgres/issues/450#issuecomment-393167936
# https://github.com/docker-library/postgres/pull/452
if [ -x "$f" ]; then
echo "$0: running $f"
"$f"
else
echo "$0: sourcing $f"
. "$f"
fi
;;
*.sql) echo "$0: running $f"; "${psql[#]}" -f "$f"; echo ;;
*.sql.gz) echo "$0: running $f"; gunzip -c "$f" | "${psql[#]}"; echo ;;
*) echo "$0: ignoring $f" ;;
esac
echo
done
See Setting an argument with bash for the reason to use an array rather than a string.
The #!/bin/sh and the [#] are incongruous. This is a bash-ism, where the psql variable is an array. This literal quote dollarsign psql bracket at bracket quote is expanded into "psql" "array" "values" "each" "listed" "and" "quoted" "separately." It's the safer way, e.g., to accumulate arguments to a command where any of them might have spaces in them.
psql=(/foo/psql arg arg arg) is the best way to define the array you need there.
It might look obscure, but it would work like so...
Let's say we have a bash array wc, which contains a command wc, and an argument -w, and we feed that a here document with some words:
wc=(wc -w)
"${wc[#]}" <<- words
one
two three
four
words
Since there are four words in the here document, the output is:
4
In the quoted code, there needs to be some prior point, (perhaps a calling script), that does something like:
psql=(psql -option1 -option2 arg1 arg2 ... )
As to why the programmer chose to invoke a command with an array, rather than just invoke the command, I can only guess... Maybe it's a crude sort of operator overloading to compensate for different *nix distros, (i.e. BSD vs. Linux), where the local variants of some necessary command might have different names from the same option, or even use different commands. So one might check for BSD or Linux or a given version, and reset psql accordingly.
The answer from #Barmar is correct.
The script was intended to be "sourced" and not "executed".
I faced the same problem and came to the same answer after I read that it had been reported here and fixed by "chmod".
https://github.com/postgis/docker-postgis/issues/119
Therefore, the fix is to change the permissions.
This can be done either in your git repository:
chmod -x initdb-postgis.sh
or add a line to your docker file.
RUN chmod -x /docker-entrypoint-initdb.d/10_postgis.sh
I like to do both so that it is clear to others.
Note: if you are using git on windows then permission can be lost. Therefore, "chmod" in the docker file is needed.
(This question is a follow-up on this comment, in an answer about git hooks)
I'm far too unskilled in bash (so far) to understand fully the remark and how to act accordingly. More specifically, I've been advised to avoid using bash command cat this way :
echo "$current_branch" $(cat "$1") > "$1"
because the order of operations depends on the specific shell and it could end up destroying the contents of the passed argument, so the commit message itself if I got it right?
Also, how to "save the contents in a separate step"?
Would the following make any sense?
tmp = "$1"
echo "$current_branch" $(cat $tmp) > "$1"
The proposed issue is not about overwriting variables or arguments, but about the fact that both reading from and writing to a file at the same time is generally a bad idea.
For example, this command may look like it will just write a file to itself, but instead it truncates it:
cat myfile > myfile # Truncates the file to size 0
However, this is not a problem in your specific command. It is guaranteed to work in a POSIX compliant shell because the order of operations specify that redirections will happen after expansions:
The words that are not variable assignments or redirections shall be expanded. If any fields remain following their expansion, the first field shall be considered the command name and remaining fields are the arguments for the command.
Redirections shall be performed as described in Redirection.
Double-however, it's still a bit fragile in the sense that seemingly harmless modifications may trigger the problem, such as if you wanted to run sed on the result. Since the redirection (> "$1") and command substitution $(cat "$1") are now in separate commands, the POSIX definition no longer saves you:
# Command may now randomly result in the original message being deleted
echo "$current_branch $(cat "$1")" | sed -e 's/(c)/©/g' > "$1"
Similarly, if you refactor it into a function, it will also suddenly stop working:
# Command will now always delete the original message
modify_message() {
echo "$current_branch $(cat "$1")"
}
modify_message "$1" > "$1"
You can avoid this by writing to a temporary file, and then replace your original.
tmp=$(mktemp) || exit
echo "$current_branch $(cat "$1")" > "$tmp"
mv "$tmp" "$1"
In my opinion, it's better to save to another file.
You may try something like
echo "$current_branch" > tmp
cat "$1" >> tmp # merge these into
# echo "$current_branch" $(cat "$1") > tmp
# may both OK
mv tmp "$1"
However I am not sure if my understanding is right, or there are some better solutions.
This is what I considered as the core of question. It is hard to decide the "precedence" of $() block and >. If > is executed "earlier", then echo "$current_branch" will rewrite "$1" file and drop the original content of "$1", which is a disaster. If $() is executed "earlier", then everything works as expected. However, there exists a risk, and we should avoid it.
A command group would be far better than a command substitution here. Note the similarity to Geno Chen's answer.
{
echo "$current_branch"
cat "$1"
} > tmp && mv tmp "$1"
First of all, hi to everyone, that's my first post here.
I swear I have checked the site for similar questions to avoid the "double post about same argument" issue but none of them answered exactly to my question.
The problem is that in the code below I always get the "There are no files with this extension" message when I call the script passing it an extension as first argument.
#!/bin/bash
if [ "$1" ];
then
file=*."$1";
if [ -f "$file" ];
then
for i in "$file";
[...do something with the each file using "$i" like echo "$i"]
else
echo "There are no files with this extension";
fi;
else
echo "You have to pass an extension"
fi;
I tried using the double parenthesis, using and not using the quotes in the nested if, using *."$1" directly in the if, but none of this solution worked.
One problem is that you're not quoting a variable when you first assign a value to file. In this statement:
file=*."$1";
The * will be interpreted by the shell, so for example if you passed in .py on the command line, file might end up with the value file1.py file2.py, which will throw off your file existence test later on.
Another problem, as #sideshowbarker points out, is that you can't use wildcards with the [ -f ... ].
Another variable quoting issue is that quoting inhibits wildcard expansion, such that even without the file existence test, if $file is, e.g., *.txt, then this:
for x in "$file"; do ...
Will loop over a single argument with the literal value *.txt, while this:
for x in $file; do ...
Will loop over all files that end with a .txt extension (unless there are none, in which case it will loop once with $x set to the literal value *.txt).
Typically, you would write your script to expect a list of arguments, and allow the user to call it like myscript *.txt...that is, leave wildcard handling to the interactive shell, and just let your script process a list of arguments. Then it becomes simply:
for i in "$#"; do
echo do something with this file named "$x"
done
If you really want to handle the wildcard expansion in your script, something like this might work:
#!/bin/bash
if [ "$1" ];
then
ext="$1"
for file in *.$ext; do
[ -f "$file" ] || continue
echo $file
done
else
echo "You have to pass an extension"
fi;
The statement [ -f "$file" ] || continue is necessary there because of the case I mentioned earlier: if there are no files, the loop will still execute once with the literal expansion of *.$ext.
I'm trying to write a simple script that will tell me if a filename exist in $Temp that starts with the string "Test".
For example, I have these files
Test1989.txt
Test1990.txt
Test1991.txt
Then I just want to echo that a file was found.
For example, something like this:
file="home/edward/bank1/fiche/Test*"
if test -s "$file"
then
echo "found one"
else
echo "found none"
fi
But this doesn't work.
One approach:
(
shopt -s nullglob
files=(/home/edward/bank1/fiche/Test*)
if [[ "${#files[#]}" -gt 0 ]] ; then
echo found one
else
echo found none
fi
)
Explanation:
shopt -s nullglob will cause /home/edward/bank1/fiche/Test* to expand to nothing if no file matches that pattern. (Without it, it will be left intact.)
( ... ) sets up a subshell, preventing shopt -s nullglob from "escaping".
files=(/home/edward/bank1/fiche/Test*) puts the file-list in an array named files. (Note that this is within the subshell only; files will not be accessible after the subshell exits.)
"${#files[#]}" is the number of elements in this array.
Edited to address subsequent question ("What if i also need to check that these files have data in them and are not zero byte files"):
For this version, we need to use -s (as you did in your question), which also tests for the file's existence, so there's no point using shopt -s nullglob anymore: if no file matches the pattern, then -s on the pattern will be false. So, we can write:
(
found_nonempty=''
for file in /home/edward/bank1/fiche/Test* ; do
if [[ -s "$file" ]] ; then
found_nonempty=1
fi
done
if [[ "$found_nonempty" ]] ; then
echo found one
else
echo found none
fi
)
(Here the ( ... ) is to prevent file and found_file from "escaping".)
You have to understand how Unix interprets your input.
The standard Unix shell interpolates environment variables, and what are called globs before it passes the parameters to your program. This is a bit different from Windows which makes the program interpret the expansion.
Try this:
$ echo *
This will echo all the files and directories in your current directory. Before the echo command acts, the shell interpolates the * and expands it, then passes that expanded parameter back to your command. You can see it in action by doing this:
$ set -xv
$ echo *
$ set +xv
The set -xv turns on xtrace and verbose. Verbose echoes the command as entered, and xtrace echos the command that will be executed (that is, after the shell expansion).
Now try this:
$ echo "*"
Note that putting something inside quotes hides the glob expression from the shell, and the shell cannot expand it. Try this:
$ foo="this is the value of foo"
$ echo $foo
$ echo "$foo"
$ echo '$foo'
Note that the shell can still expand environment variables inside double quotes, but not in single quotes.
Now let's look at your statement:
file="home/edward/bank1/fiche/Test*"
The double quotes prevent the shell from expanding the glob expression, so file is equal to the literal home/edward/bank1/finche/Test*. Therefore, you need to do this:
file=/home/edward/bank1/fiche/Test*
The lack of quotes (and the introductory slash which is important!) will now make file equal to all files that match that expression. (There might be more than one!). If there are no files, depending upon the shell, and its settings, the shell may simply set file to that literal string anyway.
You certainly have the right idea:
file=/home/edward/bank1/fiche/Test*
if test -s $file
then
echo "found one"
else
echo "found none"
fi
However, you still might get found none returned if there is more than one file. Instead, you might get an error in your test command because there are too many parameters.
One way to get around this might be:
if ls /home/edward/bank1/finche/Test* > /dev/null 2>&1
then
echo "There is at least one match (maybe more)!"
else
echo "No files found"
fi
In this case, I'm taking advantage of the exit code of the ls command. If ls finds one file it can access, it returns a zero exit code. If it can't find one matching file, it returns a non-zero exit code. The if command merely executes a command, and then if the command returns a zero, it assumes the if statement as true and executes the if clause. If the command returns a non-zero value, the if statement is assumed to be false, and the else clause (if one is available) is executed.
The test command works in a similar fashion. If the test is true, the test command returns a zero. Otherwise, the test command returns a non-zero value. This works great with the if command. In fact, there's an alias to the test command. Try this:
$ ls -li /bin/test /bin/[
The i prints out the inode. The inode is the real ID of the file. Files with the same ID are the same file. You can see that /bin/test and /bin/[ are the same command. This makes the following two commands the same:
if test -s $file
then
echo "The file exists"
fi
if [ -s $file ]
then
echo "The file exists"
fi
You can do it in one line:
ls /home/edward/bank1/fiche/Test* >/dev/null 2>&1 && echo "found one" || echo "found none"
To understand what it does you have to decompose the command and have a basic awareness of boolean logic.
Directly from bash man page:
[...]
expression1 && expression2
True if both expression1 and expression2 are true.
expression1 || expression2
True if either expression1 or expression2 is true.
[...]
In the shell (and in general in unix world), the boolean true is a program that exits with status 0.
ls tries to list the pattern, if it succeed (meaning the pattern exists) it exits with status 0, 2 otherwise (have a look at ls man page for details).
In our case there are actually 3 expressions, for the sake of clarity I will put parenthesis, although they are not needed because && has precedence on ||:
(expression1 && expression2) || expression3
so if expression1 is true (ie: ls found the pattern) it evaluates expression2 (which is just an echo and will exit with status 0). In this case expression3 is never evaluate because what's on the left site of || is already true and it would be a waste of resources trying to evaluate what's on the right.
Otherwise, if expression1 is false, expression2 is not evaluated but in this case expression3 is.
for entry in "/home/loc/etc/"/*
do
if [ -s /home/loc/etc/$entry ]
then
echo "$entry File is available"
else
echo "$entry File is not available"
fi
done
Hope it helps
The following script will help u to go to a process if that script exist in a specified variable,
cat > waitfor.csh
#!/bin/csh
while !( -e $1 )
sleep 10m
end
ctrl+D
here -e is for working with files,
$1 is a shell variable,
sleep for 10 minutes
u can execute the script by ./waitfor.csh ./temp ; echo "the file exits"
One liner to check file exist or not -
awk 'BEGIN {print getline < "file.txt" < 0 ? "File does not exist" : "File Exists"}'
Wildcards aren't expanded inside quoted strings. And when wildcard is expanded, it's returned unchanged if there are no matches, it doesn't expand into an empty string. Try:
output="$(ls home/edward/bank1/fiche/Test* 2>/dev/null)"
if [ -n "$output" ]
then echo "Found one"
else echo "Found none"
fi
If the wildcard expanded to filenames, ls will list them on stdout; otherwise it will print an error on stderr, and nothing on stdout. The contents of stdout are assigned to output.
if [ -n "$output" ] tests whether $output contains anything.
Another way to write this would be:
if [ $(ls home/edward/bank1/fiche/Test* 2>/dev/null | wc -l) -gt 0 ]