I'm running this in bash and even though there is a .txt file it prints out "no new folders to create" in the terminal.
Am I missing something?
FILES=cluster-02/*/*
for f in $FILES
do
if [[ $f == *."txt" ]]
then
cat $f | xargs mkdir -p
else
echo "No new folders to create"
fi
done;
As mentioned in the first comment, the behaviour is indeed as you might expect from your script: you run through all files, text files and other ones. In case your file is a text file, you perform the if-case and in case your file is another type of file, you perform the else-case.
In order to solve this, you might decide not to take the other files into account (only handle text files), I think you might do this as follows:
FILES=cluster-02/*/*.txt
You're looping over multiple files, so the first result may trigger the if and the second can show the else.
You could save the wildcard result in an array, check if there's something in it, and loop if so:
shopt -s nullglob
FILES=( foo/* )
if (( ${#FILES[#]} )); then
for f in "${FILES[#]}"; do
if [[ $f == *."txt" ]]; then
echo $f
fi
done
else
echo "No new folders to create"
fi
#!/usr/bin/env bash
# Create an array containing a list of files
# This is safer to avoid issues with files having special characters such
# as spaces, glob-characters, or other characters that might be cumbersome
# Note: if no files are found, the array contains a single element with the
# string "cluster-02/*/*"
file_list=( cluster-02/*/* )
# loop over the content of the file list
# ensure to quote the list to avoid the same pitfalls as above
for _file in "${file_list[#]}"
do
[ "${_file%.txt}" == "${_file}" ] && continue # skip, not a txt
[ -f "${_file}" ] || continue # check if the file exists
[ -r "${_file}" ] || continue # check if the file is readable
[ -s "${_file}" ] || continue # check if the file is empty
< "${_file}" xargs mkdir -p -- # add -- to avoid issues with entries starting with -
_c=1
done;
[ "${_c}" ] || echo "No new folders to create"
Related
I'm hoping this is a simple question, since I've never done shell scripting before. I'm trying to filter certain files out of a list of results. While the script executes and prints out a list of files, it's not filtering out the ones I don't want. Thanks for any help you can provide!
#!/bin/bash
# Purpose: Identify all *md files in H2 repo where there is no audit date
#
#
#
# Example call: no_audits.sh
#
# If that call doesn't work, try ./no_audits.sh
#
# NOTE: Script assumes you are executing from within the scripts directory of
# your local H2 git repo.
#
# Process:
# 1) Go to H2 repo content directory (assumption is you are in the scripts dir)
# 2) Use for loop to go through all *md files in each content sub dir
# and list all file names and directories where audit date is null
#
#set counter
count=0
# Go to content directory and loop through all 'md' files in sub dirs
cd ../content
FILES=`find . -type f -name '*md' -print`
for f in $FILES
do
if [[ $f == "*all*" ]] || [[ $f == "*index*" ]] ;
then
# code to skip
echo " Skipping file: " $f
continue
else
# find audit_date in file metadata
adate=`grep audit_date $f`
# separate actual dates from rest of the grepped line
aadate=`echo $adate | awk -F\' '{print $2}'`
# if create date is null - proceed
if [[ -z "$aadate" ]] ;
then
# print a list of all files without audit dates
echo "Audit date: " $aadate " " $f;
count=$((count+1));
fi
fi
done
echo $count " files without audit dates "
First, to address the immediate issue:
[[ $f == "*all*" ]]
is only true if the exact contents of f is the string *all* -- with the wildcards as literal characters. If you want to check for a substring, then the asterisks shouldn't be quoted:
[[ $f = *all* ]]
...is a better-practice solution. (Note the use of = rather than == -- this isn't essential, but is a good habit to be in, as the POSIX test command is only specified to permit = as a string comparison operator; if one writes [ "$f" == foo ] by habit, one can get unexpected failures on platforms with a strictly compliant /bin/sh).
That said, a ground-up implementation of this script intended to follow best practices might look more like the following:
#!/usr/bin/env bash
count=0
while IFS= read -r -d '' filename; do
aadate=$(awk -F"'" '/audit_date/ { print $2; exit; }' <"$filename")
if [[ -z $aadate ]]; then
(( ++count ))
printf 'File %q has no audit date\n' "$filename"
else
printf 'File %q has audit date %s\n' "$filename" "$aadate"
fi
done < <(find . -not '(' -name '*all*' -o -name '*index*' ')' -type f -name '*md' -print0)
echo "Found $count files without audit dates" >&2
Note:
An arbitrary list of filenames cannot be stored in a single bash string (because all characters that might otherwise be used to determine where the first name ends and the next name begins could be present in the name itself). Instead, read one NUL-delimited filename at a time -- emitted with find -print0, read with IFS= read -r -d ''; this is discussed in [BashFAQ #1].
Filtering out unwanted names can be done internal to find.
There's no need to preprocess input to awk using grep, as awk is capable of searching through input files itself.
< <(...) is used to avoid the behavior in BashFAQ #24, wherein content piped to a while loop causes variables set or modified within that loop to become unavailable after its exit.
printf '...%q...\n' "$name" is safer than echo "...$name..." when handling unknown filenames, as printf will emit printable content that accurately represents those names even if they contain unprintable characters or characters which, when emitted directly to a terminal, act to modify that terminal's configuration.
Nevermind, I found the answer here:
bash script to check file name begins with expected string
I tried various versions of the wildcard/filename and ended up with:
if [[ "$f" == *all.md ]] || [[ "$f" == *index.md ]] ;
The link above said not to put those in quotes, and removing the quotes did the trick!
As an exercise I have set myself the task of recursively listing files using bash builtins. I particularly don't want to use ls or find and I would prefer not to use setopt extendedglob. The following appears to work but I cannot see how to extend it with /.* to list hidden files. Is there a simple workaround?
g() { for k in "$1"/*; do # loop through directory
[[ -f "$k" ]] && { echo "$k"; continue; }; # echo file path
[[ -d "$k" ]] && { [[ -L "$k" ]] && { echo "$k"; continue; }; # echo symlinks but don't follow
g "$k"; }; # start over with new directory
done; }; g "/Users/neville/Desktop" # original directory
Added later: sorry - I should have said: 'bash-3.2 on OS X'
Change
for k in "$1"/*; do
to
for k in "$1"/* "$1"/.[^.]* "$1"/..?*; do
The second glob matches all files whose names start with a dot followed by anything other than a dot, while the third matches all files whose names start with two dots followed by something. Between the two of them, they will match all hidden files other than the entries . and ...
Unfortunately, unless the shell option nullglob is set, those (like the first glob) could remain as-is if there are no files whose names match (extremely likely in the case of the third one) so it is necessary to verify that the name is actually a file.
An alternative would be to use the much simpler glob "$1"/.*, which will always match the . and .. directory entries, and will consequently always be substituted. In that case, it's necessary to remove the two entries from the list:
for k in "$1"/* "$1"/.*; do
if ! [[ $k =~ /\.\.?$ ]]; then
# ...
fi
done
(It is still possible for "$1"/* to remain in the list, though. So that doesn't help as much as it might.)
Set the GLOBIGNORE file to exclude . and .., which implicitly turns on "shopt -u dotglob". Then your original code works with no other changes.
user#host [/home/user/dir]
$ touch file
user#host [/home/user/dir]
$ touch .dotfile
user#host [/home/user/dir]
$ echo *
file
user#host [/home/user/dir]
$ GLOBIGNORE=".:.."
user#host [/home/user/dir]
$ echo *
.dotfile file
Note that this is bash-specific. In particular, it does not work in ksh.
You can specify multiple arguments to for:
for k in "$1"/* "$1"/.*; do
But if you do search for .* in directories , you should be aware that it also gives you the . and .. files. You may also be given a nonexistent file if the "$1"/* glob matches, so I would check that too.
With that in mind, this is how I would correct the loop:
g() {
local k subdir
for k in "$1"/* "$1"/.*; do # loop through directory
[[ -e "$k" ]] || continue # Skip missing files (unmatched globs)
subdir=${k##*/}
[[ "$subdir" = . ]] || [[ "$subdir" = .. ]] && continue # Skip the pseudo-directories "." and ".."
if [[ -f "$k" ]] || [[ -L "$k" ]]; then
printf %s\\n "$k" # Echo the paths of files and symlinks
elif [[ -d "$k" ]]; then
g "$k" # start over with new directory
fi
done
}
g ~neville/Desktop
Here the funky-looking ${k##*/} is just a fast way to take the basename of the file, while local was put in so that the variables don't modify any existing variables in the shell.
One more thing I've changed is echo "$k" to printf %s\\n "$k", because echo is irredeemably flawed in its argument handling and should be avoided for the purpose of echoing an unknown variable. (See Rich's sh tricks for an explanation of how; it boils down to -n and -e throwing a spanner in the works.)
By the way, this will NOT print sockets or fifos - is that intentional?
I have a bunch of files that need to be renamed and the new name is in a text file.
Example file name:
ASBC_Fishbone_Ia.pdf
Example entry in text file:
Ia. Propagation—Design Considerations
Expected new file name:
Ia. Propagation—Design Considerations.pdf
or
Ia._Propagation—Design_Considerations
What would be a good way of going about this using typical linux cli tools? I'm thinking some combination of ls, grep and rename?
You can try:
#!/bin/bash
# Do not allow the script to run if it's not Bash or Bash version is < 4.0 .
[ -n "$BASH_VERSION" ] && [[ BASH_VERSINFO -ge 4 ]] || exit 1
# Do not allow presenting glob pattern if no match is found.
shopt -s nullglob
# Use an associative array.
declare -A MAP=() || exit 1
while IFS=$'\t' read -r CODE NAME; do
# Maps name with code e.g. MAP['Ia']='Propagation—Design Considerations'
MAP[${CODE%.}]=$NAME
done < /path/to/text_file
# Change directory. Not needed if files are in current directory.
cd "/path/to/dir/containing/files" || exit 1
for FILE in *_*.pdf; do
# Get code from filename.
CODE=${FILE##*_} CODE=${CODE%.pdf}
# Skip if no code was extracted from file.
[[ -n $CODE ]] || continue
# Get name from map based from code.
NAME=${MAP[$CODE]}
# Skip if no new name was registered based on code.
[[ -n $NAME ]] || continue
# Generate new name.
NEW_NAME="${CODE}. $NAME.pdf"
# Replace spaces with _ at your preference. Uncomment if wanted.
# NEW_NAME=${NEWNAME// /_}
# Rename file. Remove echo if you find it correct already.
echo mv -- "$FILE" "$NEW_NAME"
done
Background:
I have a bunch of filenames named username.sub in single letter directories under script_testing (first letter of username is the folder name). For every username.sub, I need to check if the line user.$username.contacts exists and, if not, append the line followed by a real tab.
Question:
Given the code I have below, why is it not appending to the file? I think I am missing something simple. I keep getting "contacts already subscribed" even if that line is not there.
#!/bin/bash
Path_to_files=/home/user/script_testing/^[A-z]+$/
FULLNAME="${Path_to_files##*/}"
NAME="${FULLNAME%.*}"
if grep 'contacts' $NAME.sub; then
echo 'contacts already subscribed'
else
echo "subscribing to contacts"
echo -e user.$NAME.Contacts \t >> $NAME.sub
fi
You're grepping for the word contacts - which, depending on what else you have in those files, may always be present.
Instead, use grep -q "^user\.$NAME\.Contacts" to look for your line.
Fixed with the following:
#!/bin/bash
#testing directory
#p=$HOME/script_testing
for f in "$p"/*/*.sub ; do
# if this is a file
if [ -f "$f" ]; then
# define variables
F="${f##*/}"
u="${F%%.*}"
cont=$(grep "user.$u.Contacts" "$f")
cal=$(grep "user.$u.Calendar" "$f")
# if our file doesn't contain Contacts subscription
if [ -z "$cont" ]; then
# add Contacts subscription
echo -e "user.$u.Contacts\t" >> "$f"
#fi
# if our file doesn't contain Calendar subscription
elif [ -z "$cal" ]; then
# add Calendar subscription
echo -e "user.$u.Calendar\t" >> "$f"
fi
fi
done
Also added extra line(s) to append. Please, let me know if there is an issue with this so I can learn, but I haven't encountered any problems.
I want to check whether a directory has files or not in bash.
My code is here.
for d in {,/usr/local}/etc/bash_completion.d ~/.bash/completion.d
do
[ -d "$d" ] && [ -n "${d}/*" ] &&
for f in $d/*; do
[ -f "$f" ] && echo "$f" && . "$f"
done
done
The problem is that "~/.bash/completion.d" has no file.
So, $d/* is regarded as simple string "~/.bash/completion.d/*", not empty string which is result of filename expansion.
As a result of that code, bash tries to run
. "~/.bash/completion.d/*"
and of course, it generates error message.
Can anybody help me?
If you set the nullglob bash option, through
shopt -s nullglob
then globbing will drop patterns that don't match any file.
# NOTE: using only bash builtins
# Assuming $d contains directory path
shopt -s nullglob
# Assign matching files to array
files=( "$d"/* )
if [ ${#files[#]} -eq 0 ]; then
echo 'No files found.'
else
# Whatever
fi
Assignment to an array has other benefits, including desirable (correct!) handling of filenames/paths containing white-space, and simple iteration without using a sub-shell, as the following code does:
find "$d" -type f |
while read; do
# Process $REPLY
done
Instead, you can use:
for file in "${files[#]}"; do
# Process $file
done
with the benefit that the loop is run by the main shell, meaning that side-effects (such as variable assignment, say) made within the loop are visible for the remainder of script. Of course, it's also way faster, if performance is an issue.
Finally, an array can also be inserted in command line arguments (without splitting arguments containing white-space):
$ md5sum fileA "${files[#]}" fileZ
You should always attempt to correctly handle files/paths containing white-space, because one day, they will happen!
You could use find directly in the following way:
for f in $(find {,/usr/local}/etc/bash_completion.d ~/.bash/completion.d -maxdepth 1 -type f);
do echo $f; . $f;
done
But find will print a warning if some of the directory isn't found, you can either put a 2> /dev/null or put the find call after testing if the directories exist (like in your code).
find() {
for files in "$1"/*;do
if [ -d "$files" ];then
numfile=$(ls $files|wc -l)
if [ "$numfile" -eq 0 ];then
echo "dir: $files has no files"
continue
fi
recurse "$files"
elif [ -f "$files" ];then
echo "file: $files";
:
fi
done
}
find /path
Another approach
# prelim stuff to set up d
files=`/bin/ls $d`
if [ ${#files} -eq 0 ]
then
echo "No files were found"
else
# do processing
fi