How to iterate on different lists according to a parameter - bash

I have a script that can get a list of folders or get them automatically with find. I'd like to write a single loop that handles both cases: if there are parameters, use them, otherwise, create the list automatically.
So far, I have something like:
if [ "$#" -gt 0 ]
then
for value in "$#"
do
v1=$(process_data "$value")
<do things with $v1>
done
else
for value in "$(find ...)"
do
v1=$(process_data "$value")
<do things with $v1>
done
fi
I'd like to do something like:
if [ "$#" -gt 0 ]
then
data="$#"
else
data="$(find ...)"
fi
for value in "${data[#]}"
do
v1=$(process_data "$value")
<do things with $v1>
done
but this data assignment can be potentially big. In a language with pointers this should not be a problem, but in bash it doesn't look good.
Is there a better way to achieve my needs?

These don't look right:
data="$#"
data="$(find ...)"
If you want data to be an array, you need
data=("$#")
data=($(find ...))
What you can do is assign to the positional parameters:
#!/bin/bash
if [ "$#" = 0 ]
then
# no arguments; default to all matching files
set -- $(find ...)
fi
for value in "$#"
do
# things
done
This does mean that the loop can't start until the find command finishes. To solve that, an alternative might be to turn it around and make using "$#" more like the output of find, i.e. stream it to a pipe, like this:
#!/bin/bash
args_or_find() {
if [ "$#" -gt 0 ]
then
printf '%s\n' "$#"
else
find ...
fi
}
while read -r value
do
# things
done < <(args_or_find)
The natural extension to improve robustness using \0 instead of \n as separator is left as an exercise.

You may use a script like this to have processing loop only once:
if (($#)); then # arguments are passed
arr=("$#")
else # build array from find command's result
arr=()
while IFS= read -rd '' f; do
arr+=("$f")
done < <(find . -name '*.txt' -print0)
fi
# main processing loop
for line in "${arr[#]}"; do
echo "processing... <$line>"
done

Related

How to set an arbitrary positional argument, while still preserving the rest?

I would like to do something like this, but preserve every argument after $i:
for i in "$#"; do
if [[ $i == "--" ]]; then
set $i "-S --"
break
fi
done
ls "$#"
In this example, I want to make a simple wrapper over ls where -S is always the final option that is applied.
This is simple if the arguments do not have "--":
ls "$#" -S
However, this breaks whenever there is a "--" as an argument.
To work around this, I would like to find the first occurrence of -- and place an -S before it.
EDIT:
The reason why I do not use:
ls -S "$#"
is because I want the output to be sorted by size LAST. So if -t is passed into the arguments, the output should be sorted by modification time THEN by size. That use case fails here:
ls -S -t
Create a second array by iterating over the first one and inserting -S where needed.
#! /bin/bash
arr=()
for arg in "$#" ; do
if [[ $arg == -- ]] ; then
arr+=(-S --)
else
arr+=("$arg")
fi
done
ls "${arr[#]}"
You might need to insert it just once to be utterly correct:
#! /bin/bash
arr=()
inserted=
for arg in "$#" ; do
if [[ $arg == -- && ! $inserted ]] ; then
arr+=(-S --)
inserted=1
else
arr+=("$arg")
fi
done
If you really need to set the positional arguments, use
set "${arr[#]}"
to set positional arguments to the members of ${arr[#]}.

sh: Is it safe to use a variable as a command if the command contains only letters, number and underscores?

I'm writing a POSIX compliant script in dash so I am having to get creative with using fake arrays.
Contents of fake_array.sh
fake_array_job() {
array="$1"
job_name="$2"
comma_count="$(echo "$array" | grep -o -F ',' | wc -l)"
if [ "$comma_count" -lt '1' ]; then
echo 'You gave a fake array to fake_array_job that does not contain at least one comma. Exiting...'
exit
fi
array_count="$(( comma_count + 1 ))"
position=1
while [ "$position" -le "$array_count" ]; do
item="$(echo "$array" | cut -d ',' -f "$position")"
"$job_name" || exit
position="$(( position + 1 ))"
done
}
Contents of script.sh
#!/bin/sh
. fake_array.sh
job_to_do() {
echo "$item"
}
fake_array_job 'goat,pig,sheep' 'job_to_do'
second_job() {
echo "$item"
}
fake_array_job 'apple,orange' 'second_job'
I am aware that it may seem silly to use a unique name for each job I pass to fake_array_job, but I like that I have to type it twice because it helps to reduce human error.
I keep reading that it is a bad idea to use a variable as a command. Does my use of "$job_name" to run a function have any negative implications as it concerns stability, security or efficiency?
(Read to the end for a good suggestion by Charles Duffy. I'm too lazy to completely rewrite my answer to mention it earlier...)
You can iterate over the "array" using simple parameter expansions without requiring multiple elements in the array.
fake_array_job() {
args=${1%,}, # Ensure the array ends with a comma
job_name=$2
while [ -n "$args" ]; do
item=${args%%,*}
"$job_name" || exit
args=${args#*,}
done
}
One problem with the above is that assures that the array is comma-terminated by assuming that foo,bar, is not a comma-delimited array with an empty last element. A better (though uglier) solution is to use read to break up the array.
fake_array_job () {
args=$1
job_name=$2
rest=$args
while [ -n "$rest" ]; do
IFS=, read -r item rest <<EOF
$rest
EOF
"$job_name" || exit
done
}
(You can use <<-EOF and make sure the here doc is indented with tabs, but it's hard to convey that here, so I'll just leave the ugly version.)
There's also Charles Duffy's good suggestion of using case to pattern match on the array to see if there are any commas left or not:
while [ -n "$args" ]; do
case $var in
*,*) next=${args%%,*}; var=${args#*,}; "$cmd" "$next";;
*) "$cmd" "$var"; break;;
esac;
done

find -iname not working in script

Following find command.
find Work/Linux4/test/test/test_goal/spyglass_reports/clock-reset/Ac_coherency06/ -iname "Ac_coherency*.csv"
is working fine when run on shell.
But in perl script it return nothing.
#!/bin/bash
REPORT_DIR=$1
FIND_CMD=$2
echo "##";
echo $REPORT_DIR ;
echo $FIND_CMD ;
LIST_OF_CSV=$(find $REPORT_DIR $FIND_CMD)
echo $LIST_OF_CSV
if [ "X" == "X${LIST_OF_CSV}" ]; then
echo "No files Found for : '$FIND_CMD' in directory ";
echo " '$REPORT_DIR' " | sed -e 's;Work/.*/test_reports;Work/PLATFORM/test_reports;g';
echo;
exit 0;
fi
Output of script:
##
Work/$PLATFORM_SPECIES/test_reports/clock-reset/Ac_coherency06 -iname "Ac_coherency06*.csv"
No files Found for : '-iname "Ac_coherency06*.csv"' in directory 'Work/PLATFORM/test_reports/clock-reset/Ac_coherency06'
If you're allowing a list of find predicates to be passed, keep them in list form, one argument to find per argument to your script. As an example implemented in this manner:
#!/bin/bash
# read report_dir off the command line, and shift it from arguments
report_dir=$1; shift
# generate a version of report_dir for human consumption
re='Work/.*/test_reports'
replacement='Work/PLATFORM/test_reports'
if [[ $report_dir =~ $re ]]; then
report_dir_name=${report_dir//${BASH_REMATCH[0]}/$replacement}
else
report_dir_name=$report_dir
fi
# read results from find -- stored NUL-delimited -- into an array
# using NUL-delimited inputs ensure that even unusual filenames work correctly
declare -a list_of_csv
while IFS= read -r -d '' filename; do
list_of_csv+=( "$filename" )
done < <(find "$report_dir" '(' "$#" ')' -print0)
# Use the length of that array to determine whether we found contents
echo "Found ${#list_of_csv[#]} files" >&2
if (( ${#list_of_csv[#]} == 0 )); then
echo "No files found in $report_dir_name" >&2
fi
Here, shift consumes the first argument from your list, and "$#" refers to all the others that remain after that point. This means that the items you want to have passed as separate, individual arguments to find can (and must) be passed as separate, individual arguments to your script.
Thus, with usage yourscript "/path/to/report/dir" -name '*.txt', initially, $1 will be /path/to/report/dir, $2 will be -name, and $3 will be *.txt. However, after shift is run, $1 will be -name, and $2 will be *.txt; and "$#" will refer to both of those, each passed as a separate word.
For details on the use of a while read loop to read items off of a stream, see BashFAQ #001.
For details on the syntax used for bash-native string replacement, see BashFAQ #100 or http://wiki.bash-hackers.org/syntax/pe
For details on shell arrays, including ${#arrayname[#]} to check their length or "${arrayname[#]}" to expand to their contents, see BashFAQ #005.
If you have a command that is running well on the shell but not on your script, the first thing I would try would be to specify Bash on the command being called, see if this works:
bash -c 'find Work/Linux4/test/test/test_goal/spyglass_reports/clock-reset/Ac_coherency06/ -iname "Ac_coherency*.csv"'
Or even better:
/bin/bash -c 'find Work/Linux4/test/test/test_goal/spyglass_reports/clock-reset/Ac_coherency06/ -iname "Ac_coherency*.csv"'
You could also store the result on a variable or other data structure as needed, and pass it later to the script, for example:
ResultCommand="$(bash -c 'find Work/Linux4/test/test/test_goal/spyglass_reports/clock-reset/Ac_coherency06/ -iname "Ac_coherency*.csv"')"
Edit: this answer was edited more than once to fix possible issues.

bash picking arguments

I want to write a function for when I have something like the following
echo 1 2 3|pick
Pick will then take the arguments and I will do something with them.
How do I do this?
Are you looking for xargs?
pick() {
read -r arg1 arg2 remainder
echo first arg is $arg1
echo The remaining args are $remainder
}
--EDIT (response to question in comment)
One way to loop through the arguments:
pick() {
read args;
set $args;
while test $# -ne 0; do
echo $1
shift
done
}
On each iteration of the loop, $1 refers to an argument.
If I'm not mistaken, the OP wants the same thing I do: you feed it a string, and if the string containes multiple {words,lines}, it presents you a menu, and you pick one, and it returns the one you pick on stdout.
If there's only one item, it just returns it.
This is useful for--to use my particular use-case--a log file viewer script: you give it a substring of a filename, and it greps through find /var/log -name \*$arg\* -print to see what it can find. If it gets a unique hit, it hands it back to your script, which runs less against it. If it gets more than one hit, it shows you a menu, and lets you pick one.
ISTR that KSH has a builtin for this, but that I wasn't all that impressed with it; I don't recall if bash has one.
I am here because I was searching to see if someone had already written it before writing it myself. :-)
UPDATE: Nope; I wrote it myself:
Here's some example code:
/usr/local/bin/msg:
PATH=$PATH:/usr/local/bin
[ $UID = 0 ] || exec sudo su root -c "$0 $*"
FILE=/var/log/messages
[ $# -eq 1 ] &&
FILE=`find /var/log/ -name \*$1\* -print |
egrep -v '2011|.[0-9]$' |
pick`
echo "$FILE"
less +F $FILE
Since I'm piping the name to less +F I want to grep out archived log files; this is for interactive log viewing.
/usr/local/bin/pick:
# Present the user a bash Select menu, and let them pick
# Try to be smart about multi-line responses
# must take input on stdin if it might be multiline
# get multiline input from stdin
while read LINE </dev/stdin
do
CHOICES+=( $LINE )
done
# add on anything specified as arguments
while [ $# -gt 0 ]
do
CHOICES+=( $1 )
shift
done
# if only one thing to pick, just pick it
if [ ${#CHOICES[*]} -eq 1 ]
then
echo $CHOICES
exit
fi
# eval set $CHOICES
select CHOSEN in ${CHOICES[#]}
do
echo $CHOSEN
exit
done </dev/tty

How to test filename expansion result in bash?

I want to check whether a directory has files or not in bash.
My code is here.
for d in {,/usr/local}/etc/bash_completion.d ~/.bash/completion.d
do
[ -d "$d" ] && [ -n "${d}/*" ] &&
for f in $d/*; do
[ -f "$f" ] && echo "$f" && . "$f"
done
done
The problem is that "~/.bash/completion.d" has no file.
So, $d/* is regarded as simple string "~/.bash/completion.d/*", not empty string which is result of filename expansion.
As a result of that code, bash tries to run
. "~/.bash/completion.d/*"
and of course, it generates error message.
Can anybody help me?
If you set the nullglob bash option, through
shopt -s nullglob
then globbing will drop patterns that don't match any file.
# NOTE: using only bash builtins
# Assuming $d contains directory path
shopt -s nullglob
# Assign matching files to array
files=( "$d"/* )
if [ ${#files[#]} -eq 0 ]; then
echo 'No files found.'
else
# Whatever
fi
Assignment to an array has other benefits, including desirable (correct!) handling of filenames/paths containing white-space, and simple iteration without using a sub-shell, as the following code does:
find "$d" -type f |
while read; do
# Process $REPLY
done
Instead, you can use:
for file in "${files[#]}"; do
# Process $file
done
with the benefit that the loop is run by the main shell, meaning that side-effects (such as variable assignment, say) made within the loop are visible for the remainder of script. Of course, it's also way faster, if performance is an issue.
Finally, an array can also be inserted in command line arguments (without splitting arguments containing white-space):
$ md5sum fileA "${files[#]}" fileZ
You should always attempt to correctly handle files/paths containing white-space, because one day, they will happen!
You could use find directly in the following way:
for f in $(find {,/usr/local}/etc/bash_completion.d ~/.bash/completion.d -maxdepth 1 -type f);
do echo $f; . $f;
done
But find will print a warning if some of the directory isn't found, you can either put a 2> /dev/null or put the find call after testing if the directories exist (like in your code).
find() {
for files in "$1"/*;do
if [ -d "$files" ];then
numfile=$(ls $files|wc -l)
if [ "$numfile" -eq 0 ];then
echo "dir: $files has no files"
continue
fi
recurse "$files"
elif [ -f "$files" ];then
echo "file: $files";
:
fi
done
}
find /path
Another approach
# prelim stuff to set up d
files=`/bin/ls $d`
if [ ${#files} -eq 0 ]
then
echo "No files were found"
else
# do processing
fi

Resources