BASH - 'exit 1' failed in loop inside another loop [duplicate] - bash

This question already has answers here:
Exit bash script within while loop
(2 answers)
Closed last month.
The following code doesn't exit at the first exit 1 from the call of error_exit. What am I missing?
#!/bin/bash
THIS_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
JINJANG_DIR="$(cd "$THIS_DIR/../.." && pwd)"
DATAS_DIR="$THIS_DIR/datas"
error_exit() {
echo ""
echo "ERROR - Following command opens the file that has raised an error."
echo ""
echo " > open \"$1\""
exit 1
}
cd "$DATAS_DIR"
find . -name 'datas.*' -type f | sort | while read -r datafile
do
localdir="$(dirname $datafile)"
echo " * Testing ''$localdir''."
filename=$(basename "$datafile")
ext=${filename##*.}
if [ "$ext" == "py" ]
then
unsafe="-u"
else
unsafe=""
fi
datas="$DATAS_DIR/$datafile"
find . -name 'template.*' -type f | sort | while read -r template
do
filename=$(basename "$template")
ext=${filename##*.}
template="$DATAS_DIR/$template"
outputfound="$DATAS_DIR/$localdir/output_found.$ext"
cd "$JINJANG_DIR"
python -m src $UNSAFE "$DATA" "$TEMPLATE" "$OUTPUTFOUND" || error_exit "$localdir"
done
cd "$DATAS_DIR"
done
Here is the output I obtain.
ERROR - Following command opens the file that has raised an error.
> open "./html/no-param-1"
* Testing ''./html/no-param-2''.
ERROR - Following command opens the file that has raised an error.
> open "./html/no-param-2"
* Testing ''./latex/no-param-1''.
ERROR - Following command opens the file that has raised an error.
> open "./latex/no-param-1"
* Testing ''./latex/no-param-2''.
ERROR - Following command opens the file that has raised an error.

In my bash environment invoking exit in a subprocess does not abort the parent process, eg:
$ echo "1 2 3" | exit # does not exit my console but instead ...
$ # presents me with the command prompt
In your case you have the pipeline: find | sort | while, so the python || error_exit is being called within a subprocess which in turn means the exit 1 will apply to the subprocess but not the (parent) script.
One solution that insures the (inner) while (and thus the exit 1) is not run in a subprocess:
while read -r template
do
... snip ...
python ... || error_exit
... snip ...
done < <(find . -name 'template.*' -type f | sort)
NOTES:
I'd recommend getting used to this structure as it also addresses another common issue ...
values assigned to variables in a subprocess are not passed 'up' to the parent process
subprocess behavior may differ in other shells
Of course, this same issue applies to the parent/outer while loop so, if the objective is for the exit 1 to apply to the entire script then this same structure will need to be implemented for the parent/outer find | sort | while, too:
while read -r datafile
do
... snip ...
while read -r template
do
... snip ...
python ... || error_exit
done < <(find . -name 'template.*' -type f | sort)
cd "$DATAS_DIR"
done < <(find . -name 'datas.*' -type f | sort)
Additional note copied from GordonDavisson's edit of this answer:
Note that the <( ) construct ("process substitution") is not
available in all shells, or even in bash when it's in sh-compatibility
mode (i.e. when it's invoked as sh or /bin/sh). So be sure to use
an explicit bash shebang (like #!/bin/bash or #!/usr/bin/env bash)
in your script, and don't override it by running the script with the
sh command.

Related

How to pass bash commands as function arguments in bash?

Here is my bash script where I am trying to pass the bash command as function arguments to the function. However, I get an error. I am assuming it is trying to run the command.
./test1.sh: line 19: find /etc -type f > /dev/null: No such file or directory
here is my entire script. why am I getting this error what is the reason behind this?
#!/bin/bash
run_time_cmd() {
cmd="$1"
TIMEFORMAT='%3lR'
exec 3>&1 4>&2
echo "print cmd"
echo $cmd
echo "running cmd = $cmd"
time_val=$( { time $cmd 2>&1; 1>&3- 2>&4-; } 2>&1 ) # Captures time only.
return time_val
}
test_find_time() {
echo 'find test starting'
path="/etc"
cmd="find $path -type f > /dev/null"
echo $cmd
total_time=run_time_cmd "${cmd}"
echo $'Time to access each file average in seconds = ' $total_time
echo 'find test stopped'
find_results=$total_time
}
test_find_time
You never call the function. The line
total_time=run_time_cmd "${cmd}"
is non-sense. In general, a line
X=Y Z
in bash runs the program Z in an environment, which is like the environment of the caller, but the variable X is set to Y.
In your case, you take the whole content of cmd (i.e. the string find /etc -type f > /dev/null) as a single command name (including all the spaces) and try to execute it in an environment, where the variable toatl_time has been set to the string run_time_cmd. Since there is no executable file in your path with the funny name 'find /etc -type f > /dev/null', you get the error message which you mention in your question.
In order to have your function invoked, you have to actually call it, for instance by doing
total_time=$(run_time_cmd "$cmd")

Bash complete function - Separating completion parts with character other than space

I've written a bash completion script to essentially do file/directory completion, but using . as the separator instead of /. However, it's not behaving as I expect it to.
Before I dive further, does anyone know of any options for this, or something that's already been written that can do this? The motivation for this is to enable completion when calling python with the -m flag. It seems crazy that this doesn't exist yet, but I was unable to find anything relevant.
My issue is that bash doesn't recognize . as a separator for completion options, and won't show the next options until I add an additional space to the end of the current command.
Here's a few concrete examples, given this directory structure.
/module
/script1.py
/script2.py
For instance, when I use the ls command, it works like this
$ ls mo<TAB>
$ ls module/<TAB><TAB>
script1.py script2.py
However, with my function, it's working like this:
$ python -m mod<TAB>
$ python -m module.<TAB><TAB>
module.
So instead of showing the next entries, it just shows the finished string again. However, if I add a space, it then works, but I don't want it to include the space:
$ python -m mod<TAB>
$ python -m module. <TAB><TAB> # (note the space here after the dot)
script1 script2 # (Note, I'm intentionally removing the file extension here).
I'd like the completion to act just like the bottom example, except not be forced to include the space to go to the next set of options
I've got about 50 tabs open and I've tried a bunch of recommendations, but nothing seems to be able to solve this how I'd like. There are a few other caveats here that would take a lot of time to go through, so I'm happy to expand on any other points if I've skipped something important. I've attached my code below, any help would be greatly appreciated. Thanks!
#!/bin/bash
_python_target() {
local cur opts cur_path
# Retrieving the current typed argument
cur="${COMP_WORDS[COMP_CWORD]}"
# Preparing an array to store available list for completions
# COMREPLY will be checked to suggest the list
COMPREPLY=()
# Here, we'll only handle the case of "-m"
# Hence, the classic autocompletion is disabled
# (ie COMREPLY stays an empty array)
if [[ "${COMP_WORDS[1]}" != "-m" ]]
then
return 0
fi
# add each path component to the current path to check for additional files
cur_path=""
for word in ${COMP_WORDS[#]:2:COMP_CWORD-2}; do
path_component=$(echo ${word} | sed 's/\./\//g')
cur_path="${cur_path}${path_component}"
done
cur_path="./${cur_path}"
if [[ ! -f "$cur_path" && ! -d "$cur_path" ]]; then
return 0
fi
# this is not very pretty, but it works. Open to comments on this too
file_opts="$(find ${cur_path} -name "*.py" -type f -maxdepth 1 -print0 | xargs -0 basename -a | sed 's/\.[^.]*$//')"
dir_opts="$(find ${cur_path} ! -path ${cur_path} -type d -maxdepth 1 -print0 | xargs -0 basename -a | xargs -I {} echo {}.)"
opts="${file_opts} ${dir_opts}"
# We store the whole list by invoking "compgen" and filling
# COMREPLY with its output content.
COMPREPLY=($(compgen -W "$opts" -- "$cur"))
[[ $COMPREPLY == *\. ]] && compopt -o nospace
}
complete -F _python_target python
Here's a draft example:
_python_target()
{
local cmd=$1 cur=$2 pre=$3
if [[ $pre != -m ]]; then
return
fi
local cur_slash=${cur//./\/}
local i arr arr2
arr=( $( compgen -f "$cur_slash" ) )
arr2=()
for i in "${arr[#]}"; do
if [[ -d $i ]]; then
arr2+=( "$i/" )
elif [[ $i == *.py ]]; then
arr2+=( "${i%.py}" )
fi
done
arr2=( "${arr2[#]//\//.}" )
COMPREPLY=( $( compgen -W "${arr2[*]}" -- "$cur" ) )
}
complete -o nospace -F _python_target python
Try with the python-2.7.18 source code directory:

Bash: Check if a directory contains only files with a specific suffix

I am trying to write a script that will check if a directory contains only
a specific kind of file (and/or folder) and will return 1 for false, 0 for true.
IE: I want to check if /my/dir/ contains only *.gz files and nothing else.
This is what i have so far, but it doesn't seem to be working as intended:
# Basic vars
readonly THIS_JOB=${0##*/}
readonly ARGS_NBR=1
declare dir_in=$1
dir_in=$1"/*.gz"
#echo $dir_in
files=$(shopt -s nullglob dotglob; echo ! $dir_in)
echo $files
if (( ${#files} ))
then
echo "Success: Directory contains files."
exit 0
else
echo "Failure: Directory is empty (or does not exist or is a file)"
exit 1
fi
I want to check if /my/dir/ contains only *.gz files and nothing else.
Use find instead of globulation. It's really easier to use find and to parse find output. Globulation are simple for simple scripts, but once you want to parse "all files in a directory" and do some filtration and such, it's way easier (and safer) to use find:
find "$1" -mindepth 1 -maxdepth 1 \! -name '*.gz' -o \! -type f | wc -l | xargs test 0 -eq
This finds all "things" that are not named *.gz inside the directory or are not files (so mkdir a.gz is accounted for), counts them, and then tests if they're count is equal to 0. If the count is equal to 0, xargs test 0 -eq will return 0, if not, it will return status between 1 - 125. You can handle the nonzero return status with a simple || return 1 if you wish.
You can remove xargs with a simple bash substitution and use the method from this thread for a little speedup and get test return value, which is 0 or 1:
[ 0 -eq "$(find "$1" -mindepth 1 -maxdepth 1 \! -name '*.gz' -o \! -type f -print '.' | wc -c)" ]
Remember that the exit status of a script is the exit status of the last command executed. So you don't need anything else in your script if you wish, only a shebang and this oneliner will suffice.
Using Bash's extglob, !(*.gz) and grep:
$ if grep -qs . path/!(*.gz) ; then echo yes ; else echo nope ; fi
man grep:
-q, --quiet, --silent
Quiet; do not write anything to standard output. Exit
immediately with zero status if any match is found, even if an
error was detected. Also see the -s or --no-messages option.
-s, --no-messages
Suppress error messages about nonexistent or unreadable files.
Since you are using bash, there is another setting you can use: GLOBIGNORE
#!/bin/bash
containsonly(){
dir="$1"
glob="$2"
if [ ! -d "$dir" ]; then
echo 1>&2 "Failure: directory does not exist"
return 2
fi
local res=$(
cd "$dir"
GLOBIGNORE=$glob"
shopt -s nullglob dotglob
echo *
)
if [ ${#res} = 0 ]; then
echo 1>&2 "Success: directory contains no extra files"
return 0
else
echo 1>&2 "Failure: directory contains extra files"
return 1
fi
}
# ...
containsonly myfolder '*.gz'
Some have suggested to count all files which do not match the globbing pattern *.gz. This might be quite inefficient depending on the the number of files. For you job it is sufficient to find just one file, which does not match your globbing pattern. Use the -quite action of find to exit after the first match:
if [ -z "$(find /usr/share/man/man1/* -not -name '*.gz' -print -quit)" ]
then echo only gz
fi

Error in attempting to parallel task of a bash script

I am trying to parallel the task of rpw_gen_features in the following bash script:
#!/bin/bash
maxjobs=8
jobcounter=0
MYDIR="/home/rasoul/workspace/world_db/journal/for-training"
DIR=$1
FILES=`find $MYDIR/${DIR}/${DIR}\_*.hpl -name *.hpl -type f -printf "%f\n" | sort -n -t _ -k 2`
for f in $FILES; do
fileToProcess=$MYDIR/${DIR}/$f
# construct .pfl file name
filebasename="${f%.*}"
fileToCheck=$MYDIR/${DIR}/$filebasename.pfl
# check if the .pfl file is already generated
if [ ! -f $fileToCheck ];
then
echo ../bin/rpw_gen_features -r $fileToProcess &
jobcounter=jobcounter+1
fi
if [jobcounter -eq maxjobs]
wait
jobcounter=0
fi
done
but it generates some error at runtime:
line 20: syntax error near unexpected token `fi'
I'm not an expert in bash programming, so please feel free to comment on the whole code.
I am curious why you don't just use GNU Parallel:
MYDIR="/home/rasoul/workspace/world_db/journal/for-training"
DIR=$1
find $MYDIR/${DIR}/${DIR}\_*.hpl -name *.hpl -type f |
parallel '[ ! -f {.}.pfl ] && echo ../bin/rpw_gen_features -r {}'
Or even:
MYDIR="/home/rasoul/workspace/world_db/journal/for-training"
parallel '[ ! -f {.}.pfl ] && echo ../bin/rpw_gen_features -r {}' ::: $MYDIR/$1/$1\_*.hpl
It seems to be way more readable, and it will automatically scale when you move from an 8-core to a 64-core machine.
Watch the intro video for a quick introduction:
https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1
Walk through the tutorial (man parallel_tutorial). You command line
with love you for it.
You are missing a then, spaces and ${} around the variables:
if [jobcounter -eq maxjobs]
wait
jobcounter=0
fi
Should be
if [ ${jobcounter} -eq ${maxjobs} ]; then
wait
jobcounter=0
fi
Further, you need to double check your script as I can see many missing ${} for example:
jobcounter=jobcounter+1
Even if you use the variables correctly this still will not work:
jobcounter=${jobcounter}+1
Will yield:
1
1+1
1+1+1
And not what you expect. You need to use:
jobcounter=`expr $jobcounter + 1`
With never versions of BASH you should be able to do:
(( jobcounter++ ))

If grep finds what it is looking for do X else Y [duplicate]

This question already has answers here:
How do I use a file grep comparison inside a bash if/else statement?
(5 answers)
Closed 2 years ago.
In this statement I am trying to match if a version ($var2) exist in the path /app/$var1 (application name)
if
find /app/$var1 -maxdepth 1 -type l -o -type d | grep $var2 #results in a nice list where i can manually get a true match.
# if match is found then execute command.
$realcmd "$#"
rc=$?
exit $rc
else
echo "no match, the version you are looking for does not exist"
fi
current code:
this include all my code (not cleaned).
command I run: "./xmodule load firefox/3.6.12"
this version does exit
#!/bin/bash
# hook for some commands
#echo $# #value of sting that is entered after "xmodule"
cmd=$(basename "$0")
#echo "called as $cmd"
if [[ $cmd = "xmodule" ]]
then
realcmd='/app/modules/0/bin/modulecmd tcsh'
# verify parameters
fi
# check if $# contains value "/" to determine if a specific version is requested.
case "$#" in
*/*)
echo "has slash"
var1=$(echo "$#" | grep -Eio '\s\w*') # Gets the aplication name and put it into var1
echo $var1 # is not printed should be "firefox"
var2=$(echo "$#" | grep -o '[^/]*$') # Gets version name and put it into var2
echo $var2
# Checking if version number exist in /app/appname/
if find /app/$var1 -noleaf -maxdepth 1 -type l -o -type d | grep $var2; then
$realcmd "$#"
exit $?
else
echo "no match, the version you are looking for does not exist"
# Should there be an exit here?
fi
;;
*)
echo "doesn't have a slash"
;;
esac
output:
mycomputer [9:55am] [user/Desktop/script] -> ./xmodule load firefox/3.6.12
'has slash
3.6.12
no match, the version you are looking for does not exist
Where there is a blank (above 3.6.1) there should be the application name. I am now realizing that this must be my problem sins the path that it uses i likely just /app.
But I do not think I changed anything in that part of the code.
You can use the entire grep pipeline as the condition of the if statement. Use grep -q to keep it from printing the match it finds (unless you want that printed). I also simplified the exit (there's no need to store $? in a variable if you're just going to use it immediately). Here's the result:
if find "/app/$var1" -maxdepth 1 -type l -o -type d | grep -q "$var2"; then
$realcmd "$#"
exit $?
else
echo "no match, the version you are looking for does not exist"
# Should there be an exit here?
fi
BTW, since you're going to exit immediately after $realcmd, you could use exec $realcmd "$#" to replace the shell with $realcmd instead of running $realcmd as a subprocess.
From the grep manpage:
The exit status is 0 if selected lines are found, and 1 if not found. If an error occurred the exit status is 2.
In other words, immediately following your blah blah | grep $var2, simply check the return value.
Since the exit code for a pipeline is the exit code for the last process in that pipeline, you can use something like:
find /app/$var1 -maxdepth 1 -type l -o -type d | grep $var2 ; greprc=$?
if [[ $greprc -eq 0 ]] ; then
echo Found
else
if [[ $greprc -eq 1 ]] ; then
echo Not found
else
echo Some sort of error
fi
fi

Resources