I'm trying to write a generic shell script to archive X days older files matching pattern passed as parameter. I'm having tough time making the boolean parameter, regex parameter and regex variable in script work across mac and ubuntu. I'm new to shell scripting. Any suggestion related to the problem or best practices are welcome. Following is the script:
#!/usr/bin/env bash
#Default source dir
SOURCE_DIR=./logs/
# Delete by default
DELETE=YES
# Archive files older by these many days
OLD=7
# Pattern to archive
PATTERN="*.log*"
# Use -gt 1 to consume two arguments per pass in the loop
# (each argument has a corresponding value to go with it).
while [[ $# -gt 1 ]]; do
key="$1"
case $key in
-s|--source_dir)
SOURCE_DIR="$2"
shift # past argument
;;
-d|--dest_dir)
DEST_DIR="$2"
shift # past argument
;;
-o|--days)
OLD="$2"
shift # past argument
;;
-p|--pattern)
PATTERN="$2"
shift # past argument
;;
-n|--no-delete)
DELETE=NO
;;
*)
# unknown option
;;
esac
shift # past argument or value
done
if [[ ! -d "$SOURCE_DIR" ]]; then
echo 'Archive source does not exist'
exit 1
fi
SOURCE_DIR=${SOURCE_DIR%/}
if [[ -z "$DEST_DIR" ]]; then
DEST_DIR="${SOURCE_DIR%/}/backup"
fi
DEST_DIR=${DEST_DIR%/}
if [[ ! -d "$DEST_DIR" ]]; then
echo 'Creating destination '$DEST_DIR
mkdir -p -- "$DEST_DIR"
fi
echo $SOURCE_DIR
echo $DEST_DIR
echo $OLD
echo $PATTERN
echo $DELETE
files=$(find $SOURCE_DIR -mtime +$OLD -type f -name $PATTERN)
echo $files
if [[ $DELETE = YES ]]; then
echo "Delete files"
else
echo "Don't delete files"
fi
Outpout on Mac:
(mysql30):recon-etl anshuc$ ./archive.sh -s junk/ -p *.py -o 10 -n
junk
junk/backup
10
*.py
YES
junk/__init__.py junk/client.py
Delete files
(mysql30):recon-etl anshuc$
Output on ubuntu 14.04
anshuc:~/workspace/xyz$ ./archive.sh -s ae/tools/ -d ae/logs/backup/ -p *.py -n -o 420
ae/tools
ae/logs/backup
420
*.py
NO
ae/tools/services/__init__.py ae/tools/__init__.py
Don't delete files
anshuc:~/workspace/xyz$
DELETE not working on mac is the concern. Also, I was having problem with PATTERN argument before. Though on trial and error I have come across a way to do. But am not sure of side-effects in case someone doesn't use quotes or any other intricacies that may be involved. A li'l input on that would make me more knowledged. :-)
TIA
Related
On MacOS Catalina, I have a bash script with
if [[ ! -f $CR/home/files/Recovery_*.txt ]]
then
echo "File does not exists in /home/files directory. Exiting" >> $log
echo "Aborted - $CR/home/files/Recovery_*txt not exist"
exit
fi
Even though there are 2 files in the directory the script exits. If I list directory contents beforehand there are 2 files. If I change it as follows the if is skipped:
if [[ `ls -la $CR/home/files/Recovery_*.txt | wc -l` -eq 0 ]]
then
echo "No files are :"
exit
fi
I am wanting to use the if -f conditional.
Any suggestions please?
Cheers, C
If you use nullglob, a glob expression that doesn't match returns an empty string. This lets us count files in bash without spawning other processes. Create an array with the expression, then check its length.
shopt -s nullglob
files=($CR/home/files/Recovery_*.txt)
if [[ ${#files[#]} -eq 0 ]]
then
echo "No files"
exit
fi
[Edited]
The error was not the variables, but the missing shebang as the script has come across from W2K3 SFU.
The tip about shellchecker.net was awesome and I will use that from now.
Thanks.
Trying to write a script that use vlc to create a playlist.
#!/bin/bash
filename=/media/*/*.mp3
while [ "$1" != "" ]; do
case $1 in
-f | --filepath ) shift
filename=$1
;;
-h | --help ) usage
exit
;;
* ) usage
exit 1
esac
shift
done
#echo $filename
vlc $filename --novideo --quiet
This script is working but it only finds mp3 files in the root of any usb device. So i want to change the filename variable. This code gives similar results but it lists evething.
filename=$(find /media/* -name *.mp3 -print)
filename=$(tr '\n' ' ' <<<$filename)
Now the problem is that i can't pass it as an argument. I tried:
vlc $filename --novideo --quiet
or
vlc $*filename --novideo --quiet
or
vlc "$filename" --novideo --quiet
nothing worked. Any suggestions?
UPDATE:
Guys the problem I want help with is how to make vlc accept the filename variable as argument or arguments of files to use in the playlist. filename contains
/media/MULTIBOOT/Linkin Park - In The End.mp3 /media/MULTIBOOT/Man with a
Mission ft. Takuma - Database.mp3 /media/MULTIBOOT/Sick Puppies - You're
Going Down.mp3 /media/MULTIBOOT/Skillet - Rise.mp3 /media/MULTIBOOT/Song
Riders - Be.mp3 /media/MULTIBOOT/30 Seconds to Mars - This is War.mp3
/media/MULTIBOOT/Fade - One Reason.mp3
Now this is a string how to use it file path arguments?
I would use bash's recursive globbing and arrays:
#!/bin/bash
shopt -s globstar nullglob
files=()
while [[ $1 ]]; do
case $1 in
-f | --filepath ) shift
files+=("$1")
;;
-h | --help ) usage
exit
;;
* ) usage
exit 1
esac
shift
done
if [[ ${#files[#]} -eq 0 ]]; then
files=( /media/**/*.mp3 )
if [[ ${#files[#]} -eq 0 ]]; then
echo "no mp3 files found"
exit 1
fi
fi
#printf "%s\n" "${files[#]}"
vlc "${files[#]}" --novideo --quiet
With this code, you can specify -f filename multiple times to play a few songs.
You need to quote *.mp3. Otherwise it will be expanded in the current directory.
filename=$(find /media/* -name '*.mp3' -print)
You also don't need to remove the newlines. When you use the variable without quoting it, all whitespace, including newlines, will be converted to word delimiters.
rather than storing all filenames in a variable, you can tell find to call an application with all the files. this will prevent problems with whitespace, newlines and the like:
find /media -name '*.mp3' -exec vlc --novideo --quiet \{\} \+
A better way to handle options in your script might be to use getopts, if you don't mind losing the option of long options. For example:
#!/usr/bin/env bash
while getopts vqt opt; do
case "$opt" in
f) filename=($OPTARG) ;;
h) usage; exit 0 ;;
*) usage; exit 1 ;;
esac
done
shift $((OPTIND - 1))
filename=($(find /media/ -name \*.mp3 -type f))
vlc --novideo --quiet "${filename[#]}"
I don't know this usage of VLC, but the effect of this script is to create a command line with all the files found by the find command, which were stored in array called $filename.
An advantage of handling things in an array is that it lends itself to use in for loops.
for thisfile in "${filename[#]}"; do
vlc "$thisfile" # with options to convert just one file
done
NOTE that since you're using bash, you may not need to use find at all.
shopt -s globstar
filelist=(/media/**/*.mp3)
Check man bash for discussion of globstar.
I haven't found anything to deal with this particular situation. Maybe there is a easy way that I'm overlooking instead of checking for a string to catch this scenario. When I check an input for existence of a file, if the input is ~/filecheck , this won't work. I get negative results while the file is in my home folder. Any suggestions for improvement to any part of the script I will definitely appreciate. I also have to use an input instead of a argument. Thanks for any help.
my test script
read -p "Enter: " input
echo $input
if [ -f $input ]; then
read -p "Do you REALLY want to delete this file?:" input2
if [[ $input2='y' || $input2 = 'Y' ]]
then
rm -f $input
elif [[ $input2='n' || $input2='N' ]]
then
exit
else
echo "Invaild Option"
exit
fi
else
echo Invaild Option!
exit
fi
Since you are entering input string as ~/filecheck shell doesn't expand tilde while using condition with -f in [ -f $input ]
You can use it this way but it is not recommended and potentially dangerous as arbitrary commands can be run by user:
if [[ -f $(bash -c "echo $input") ]]; then
echo "file exists"
fi
EDIT: As per the comments below to avoid risky bash -c you can use:
if [[ -f "${input/\~/$HOME}" ]]; then
echo "file exists"
fi
You can't have tilde expansion in this part of the program without using something based on eval—and you don't want to do that with user input. So, your poor-man solution will be to substitute any potential leading ~/ with the expansion of $HOME/. Here's the adaptation of your script in an arguably better style:
#!/bin/bash
read -e -p "Enter: " input
input=${input/#~\//$HOME/} # <--- this is the main idea of this answer (and it's rather poor)
echo "$input"
if [[ -f $input ]]; then
read -e -p "Do you REALLY want to delete this file? " input2
if [[ ${input2,,} = y ]]; then
rm -f -- "$input"
elif [[ ${input2,,} = n ]]; then
exit
else
echo "Invalid Option"
exit
fi
else
echo "Invalid Option!"
fi
exit
Now, out of curiosity, why are you spending time to make a wrapper around rm? you're making a clunky interface to an already existing program, without adding anything to it, only rendering it less powerful and less easy to use.
If all what you want it's to ask the user before deleting, you can use:
rm -i
This will give you appropriate error in the case file does not exist.
I'm writing a script in bash. It will receive from 2 to 5 arguments. For example:
./foo.sh -n -v SomeString Type Directory
-n, -v and Directory are optional.
If script doesn't receive argument Directory it will search in current directory for a string.
Otherwise it will follow received path and search there. If this directory doesn't exist it will send a message.
The question is: Is there a way to check if the last arg is a path or not?
You can get last argument using variable reference:
numArgs=$#
lastArg="${!numArgs}"
# check if last argument is directory
if [[ -d "$lastArg" ]]; then
echo "it is a directory"
else
echo "it is not a directory"
fi
you can use this:
#!/bin/bash
if [[ -d ${!#} ]]
then
echo "DIR EXISTS"
else
echo "dosen't exists"
fi
First, use getopts to parse the options -n and -v (they will have to be used before any non-options, but that's not usually an issue).
while getopts nv opt; do
case $opt in
n) nflag=1 ;;
v) vflag=1 ;;
*) printf >&2 "Unrecognized option $opt\n"; exit 1 ;;
esac
done
shift $((OPTIND-1))
Now, you will have only your two required arguments, and possibly your third optional argument, in $#.
string_arg=$1
type_arg=$2
dir_arg=$3
if [ -d "$dir_arg" ]; then
# Do something with valid directory
fi
Note that this code will work in any POSIX-compliant shell, not just bash.
I need some help with a shell code. Now I have this code:
find $dirname -type f -exec md5sum '{}' ';' | sort | uniq --all-repeated=separate -w 33 | cut -c 35-
This code finds duplicated files (with same content) in a given directory. What I need to do is to update it - find out latest (by date) modified file (from duplicated files list), print that file name and also give opportunity to delete that file in terminal.
Doing this in pure bash is a tad awkward, it would be a lot easier to write
this in perl or python.
Also, if you were looking to do this with a bash one-liner, it might be feasible,
but I really don't know how.
Anyhoo, if you really want a pure bash solution below is an attempt at doing
what you describe.
Please note that:
I am not actually calling rm, just echoing it - don't want to destroy your files
There's a "read -u 1" in there that I'm not entirely happy with.
Here's the code:
#!/bin/bash
buffer=''
function process {
if test -n "$buffer"
then
nbFiles=$(printf "%s" "$buffer" | wc -l)
echo "================================================================================="
echo "The following $nbFiles files are byte identical and sorted from oldest to newest:"
ls -lt -c -r $buffer
lastFile=$(ls -lt -c -r $buffer | tail -1)
echo
while true
do
read -u 1 -p "Do you wish to delete the last file $lastFile (y/n/q)? " answer
case $answer in
[Yy]* ) echo rm $lastFile; break;;
[Nn]* ) echo skipping; break;;
[Qq]* ) exit;;
* ) echo "please answer yes, no or quit";;
esac
done
echo
fi
}
find . -type f -exec md5sum '{}' ';' |
sort |
uniq --all-repeated=separate -w 33 |
cut -c 35- |
while read -r line
do
if test -z "$line"
then
process
buffer=''
else
buffer=$(printf "%s\n%s" "$buffer" "$line")
fi
done
process
echo "done"
Here's a "naive" solution implemented in bash (except for two external commands: md5sum, of course, and stat used only for user's comfort, it's not part of the algorithm). The thing implements a 100% Bash quicksort (that I'm kind of proud of):
#!/bin/bash
# Finds similar (based on md5sum) files (recursively) in given
# directory. If several files with same md5sum are found, sort
# them by modified (most recent first) and prompt user for deletion
# of the oldest
die() {
printf >&2 '%s\n' "$#"
exit 1
}
quicksort_files_by_mod_date() {
if ((!$#)); then
qs_ret=()
return
fi
# the return array is qs_ret
local first=$1
shift
local newers=()
local olders=()
qs_ret=()
for i in "$#"; do
if [[ $i -nt $first ]]; then
newers+=( "$i" )
else
olders+=( "$i" )
fi
done
quicksort_files_by_mod_date "${newers[#]}"
newers=( "${qs_ret[#]}" )
quicksort_files_by_mod_date "${olders[#]}"
olders=( "${qs_ret[#]}" )
qs_ret=( "${newers[#]}" "$first" "${olders[#]}" )
}
[[ -n $1 ]] || die "Must give an argument"
[[ -d $1 ]] || die "Argument must be a directory"
dirname=$1
shopt -s nullglob
shopt -s globstar
declare -A files
declare -A hashes
for file in "$dirname"/**; do
[[ -f $file ]] || continue
read md5sum _ < <(md5sum -- "$file")
files[$file]=$md5sum
((hashes[$md5sum]+=1))
done
has_found=0
for hash in "${!hashes[#]}"; do
((hashes[$hash]>1)) || continue
files_with_same_md5sum=()
for file in "${!files[#]}"; do
[[ ${files[$file]} = $hash ]] || continue
files_with_same_md5sum+=( "$file" )
done
has_found=1
echo "Found ${hashes[$hash]} files with md5sum=$hash, sorted by modified (most recent first):"
# sort them by modified date (using quicksort :p)
quicksort_files_by_mod_date "${files_with_same_md5sum[#]}"
for file in "${qs_ret[#]}"; do
printf " %s %s\n" "$(stat --printf '%y' -- "$file")" "$file"
done
read -p "Do you want to remove the oldest? [yn] " answer
if [[ ${answer,,} = y ]]; then
echo rm -fv -- "${qs_ret[#]:1}"
fi
done
if((!has_found)); then
echo "Didn't find any similar files in directory \`$dirname'. Yay."
fi
I guess the script is self-explanatory (you can read it like a story). It uses the best practices I know of, and is 100% safe regarding any silly characters in file names (e.g., spaces, newlines, file names starting with hyphens, file names ending with a newline, etc.).
It uses bash's globs, so it might be a bit slow if you have a bloated directory tree.
There are a few error checkings, but many are missing, so don't use as-is in production! (it's a trivial but rather tedious taks to add these).
The algorithm is as follows: scan each file in the given directory tree; for each file, will compute its md5sum and store in associative arrays:
files with keys the file names and values the md5sums.
hashes with keys the hashes and values the number of files the md5sum of which is the key.
After this is done, we'll scan through all the found md5sum, select only the ones that correspond to more than one file, then select all files with this md5sum, then quicksort them by modified date, and prompt the user.
A sweet effect when no dups are found: the script nicely informs the user about it.
I would not say it's the most efficient way of doing things (might be better in, e.g., Perl), but it's really a lot of fun, surprisingly easy to read and follow, and you can potentially learn a lot by studying it!
It uses a few bashisms and features that only are in bash version ≥ 4
Hope this helps!
Remark. If on your system date has the -r switch, you can replace the stat command by:
date -r "$file"
Remark. I left the echo in front of rm. Remove it if you're happy with how the script behaves. Then you'll have a script that uses 3 external commands :).