Bash: Archiving 30 days or older files - bash

I'm having the hardest time finding out how to do a fairly simple task succesfully:
Archive files and folders to a different location (eg cp -rp /source/file /destination/file)
Remove them once coppied (rm -rf /source/file)
But only if they are 30+ days or older, which I know requires find -mtime 30. But I do not know how to make the following code work with find:
#!/bin/bash
#functions
help()
{
echo "archiveer <doelmap>
archiveer /home/goedvoorbeeld/testmap
Dit commando zorgt ervoor dat gebackupte users van meer dan 30 dagen oud worden gearchiveerd. Alleen de meest recente data wordt opgeslagen.
De doelmap parameter is verplicht.";
exit
}
doemaar()
{
echo "Doe maar wordt uitgevoerd"
for f in *; do
if [[ "${f}" -nt "${doelmap}${f}" ]]
then
echo "file $f is nieuwer of niet aanwezig, gebacked up"
cp -rp "$f" "${doelmap}${f}"
rm -rf "$f"
else
echo "file $f is ouder"
fi
done
}
#check for parameters
if [[ $1 = "help" ]]
then
help
elif [[ $1 = "" ]]
then
echo "Missende parameter!"
help
fi
#set variables
doelmap=$1
#script gaat runnen
echo "${doelmap}" && sleep 1
if [ ! -d "${doelmap}" ]
then
mkdir -p "${doelmap}"
echo "${doelmap} is gemaakt"
else
echo "doelmap bestaat al"
fi
doemaar
echo "Script end"

Note that * can sometimes cause problems if filenames have spaces or special characters in them. That is why I always redirect any lists to a file, from which I work from. Also, the results list from * might exceed shell limits, causing different issues.
Also, "-rp" implies recursive, so directory contents. But earlier you stated that you were only looking at files. So the assumption is that there are no sub-directories under the directory where you are trying to find old files.
I would process the file archiving, then I would check the parent directories to purge directories that are left empty after the archiving. This is a safety measure in case of power loss.
You could consider adapting the following into your script:
cd "${LIVE_DIR}"
TOPIC=`basename "${LIVE_DIR}" `
find . -type f -mtime 30+ -print >joblist.files
mkdir "${ARCHIVE_ROOT}/${TOPIC}"
if [ $? -ne 0 ] ; then echo "unable to create dir" ; exit 1 ; fi
while read file
do
ARC_FILE="${ARCHIVE_ROOT}/${TOPIC}/${file}"
cp -pv "${file}" "${ARC_FILE}"
if [ $? -eq 0 ]
then rm -f "${file}"
if [ $? -ne 0 ]
then echo "PURGE|rm -f \"${file}\"" >&2
fi
else rm -fv "${ARC_FILE}"
echo "ARCHIVE|cp -pv \"${file}\" \"${ARC_FILE}\"" >&2
fi
echo "LAST|${file}" >joblist.last
done <joblist.files 2>joblist.failed
joblist.failed contains any cleanup that needs to be performed, commands that need to be repeated to complete the initial job.
joblist.last contains the last file processed if you need to kill the job and keep track of where you left off.

Related

Alternative method in Bash for checking if file is present

I am currently running a script (check_files.sh) that may or may not produce match* files (eg. match1.txt, match2.txt, etc. I then want to go on to run an R script on the produced match* files.
However, before I run the R script, I have tried a line in my script which checks if the files are present.
if [ -s match* ]
then
for f in match*
do
Rscript --vanilla format_matches.R ${f}
rm match*
done
else
echo "No matches present"
fi
However, I keep getting an error message (as there are often a lot of match files producted):
./check_files.sh: line 52: [: too many arguments
Is there an alternative to [ -s match* ] which would not throw up the error message? I take it that the error message appears as there are multiple match* files produced.
If you expect filenames with spaces, this is a bit ugly but robust:
found=0
find -maxdepth 1 -type f -name 'match*' -print0 | while IFS= read -rd $'\0' f
do
found=1
Rscript --vanilla format_matches.R "$f"
rm "$f"
done
if [ "$found" -eq 0 ]; then
>&2 echo "No matches present"
fi
You could change your logic to the following:
found=0
for f in match*; do
[ -e "$f" ] || continue
found=1
Rscript --vanilla format_matches.R "$f"
rm "$f"
done
if [ "$found" -eq 0 ]; then
>&2 echo "No matches present"
fi
Is there an alternative to [ -s match* ] which would not throw up the error message?
The following works for my example file match1.txt
if [ -f ~/match* ]; then
echo "yeha"
fi

Getting stuck in a logic statement

I'm writing a BASH script to purge the cache from a web server. The script is designed to take arguments from positional parameters. "ShellCheck.net" is telling me that my script is functionally correct, but when I test it I'm getting error where I shouldn't ... so I thought I'd ask for some folks to put fresh eyes on it. Take a look, I'll continue below and describe my problem:
#!/bin/bash
#
# Verify the user running is root, if not, fail.
if [[ "$UID" -ne "0" ]]; #added-1438279711
then
echo 'Only ROOT may run this script.';
exit 1;
fi
#
# Set the variables
BASE="/path/to/folder/foo/bar/" #added-1438279711
DOMAINOPT="$1" #added 1438451428
PATHOPT="$2" #added 1438451428
#
# Define Functions
function usage() { #added-1438382631
echo -en "Proper Usage:\n\n"
echo -en "\tSpecify the domain to be used\n"
echo -en "\tUsage: \"cleancache.sh abc.com\"\n"
echo -en "\t\tNote: This option will search for files and folders, recursively, within the domain folder, and remove them.\n\n"
echo -en "\tSpecify the URI you'd like to act upon within the domain\n"
echo -en "\tUsage: \"cleancache.sh abc.com /path/to/folder/\"\n"
echo -en "\t\tNote: This option will search for files and folders, recursively,\n\t\twithin the specified path, and remove them. Removing a single file is not currently supported with this script.\n\n"
}
#
# Validate the input
if [[ ! -z "$DOMAINOPT" ]] && [[ "$DOMAINOPT" != "^[A-Za-z0-9-]*[\.][a-z]*$" ]] #added-1438462778
then
clear
echo -en "Please follow the proper format for the DOMAIN option\n\n"
usage
exit 1
elif [[ ! -z "$DOMAINOPT" ]] && [[ "$DOMAINOPT" = "^[A-Za-z0-9-]*[\.][a-z]*$" ]]
then
DOMAINOPT="$DOMAINOPT"
else
clear
echo -en "Please enter a domain!\n\n"
usage
exit 1
fi
if [[ ! -z "$PATHOPT" ]] && [[ "$PATHOPT" != "^[\/][\S]*[\/]$" ]] #added-1438456371
then
clear
echo "Please follow the proper format for the PATH option"
usage
exit 1
elif [[ ! -z "$PATHOPT" ]] && [[ "$PATHOPT" = "^[\/][\S]*[\/]$" ]]
then
PATHOPT="$PATHOPT"
else
echo ""
fi
#
# Doing Stuff
if [[ "$#" -gt "2" ]]
then
echo -en "Too many arguments!\n\n"
usage
exit 1
elif [[ "$#" -eq "2" ]]
then
echo "Purging Cache in \"$BASE$DOMAINOPT$PATHOPT\""
find "$BASE""$DOMAINOPT""$PATHOPT" -type d -exec rm -rf {} \;
find "$BASE""$DOMAINOPT""$PATHOPT" -type f -exec rm -f {} \;
echo "Purging Complete"
exit 0
else
echo "Purging Cache in \"$BASE$DOMAINOPT\""
find "$BASE" -type d -name "$DOMAINOPT" -exec rm -rf {} \;
mkdir -p "$BASE$DOMAINOPT" && chown apache:apache "$BASE$DOMAINOPT" && chmod 755 "$BASE$DOMAINOPT"
echo "Purging Complete!"
echo "Creating \".stat\" file"
echo "" > "$BASE""$DOMAINOPT""/.stat"
if [[ -f "$BASE""$DOMAINOPT""/.stat" ]] #added-1438387045
then
echo "$BASE$DOMAINOPT/.stat file created!"
fi
fi
echo "All Operations Complete, exiting now!"
Everything responds normally if you run the script without any arguments (Please enter a domain), It responds normally if you try to enter a path before a domain ... but when I do it correctly, when I type: "cleancache.sh abc.com", I get an error like i haven't met the required pattern ("Please follow the proper format for the DOMAIN option") ... when that is exactly write! ... I don't understand what I'm missing, been banging my head all day, no joy.
PLEASE HELP!
Use this to match a regex:
[[ "$DOMAINOPT" =~ ^[A-Za-z0-9-]*[\.][a-z]*$ ]]
or this:
[[ ! "$DOMAINOPT" =~ ^[A-Za-z0-9-]*[\.][a-z]*$ ]]
Don't quote the regex.

Bash: check if txt files exist in a dir and if yes, process them with aria2c

OK, so, I'm trying to create a script that...
a. Will look in a folder where I drop my txt notes, and if any of them starts with "a2c_" it will recognize it as an "aria2c download list". Which brings as to...
b. Will pass the first of the matching files to aria2c, together with a directory name similar to the txt file, so that it will download each URL found in the txt to the same-named directory.
I've ended up with this..:
#!/bin/bash
#PARAMETERS: _______________________________
workingdir="/home/username/downloads/"
#___________________________________________
echo Working dir is: $workingdir
mkdir -p $workingdir
echo "Making dir"
cd $workingdir
if [ -f a2c_* ]
then
echo "Found files"
mkdir -p !a2c_downloaded
echo "Making !a2c_downloaded dir"
counter=0
echo "Set counter to 0"
for f in a2c_*.txt
do
counter=$((counter+1))
echo Download List File is: $f
echo $counter file processing.
tempfile=${f%%.*}
tempfile="`echo "$tempfile" | sed ' s/a2c_//' `"
downdir=$tempfile
echo Download Dir is: $downdir
mkdir -p $downdir
echo ___________________________________________
echo $endfilename
aria2c --auto-file-renaming -i $f -d $downdir --force-sequential
echo "Will download $f to $downdir"
sleep 5
mv $f !a2c_downloaded/
done
else
echo "No files found"
fi
...that worked when I tested it. Today, one day later, I throw some a2c_*.txt files in the dir and I'm met with "unexpected operator" errors. Any ideas? And is there an easier way to accomplish what I'm trying to do?
Thanks.
_UPDATED: _________________________________
#!/bin/bash
#PARAMETERS: _______________________________
workingdir="/home/username/downloads/"
MatchPattern="a2c_*"
#___________________________________________
echo Working dir is: $workingdir
mkdir -p $workingdir
echo "Making dir"
cd $workingdir
#if [ -f a2c_* ];
#if [ find . -maxdepth 1 -type f -name "a2c_*.txt" 2>/dev/null | grep "a2c_*" ]
#if [ -f a2c_* ]
#if [ "$?" = "0" ];
echo "Match Pattern set as $MatchPattern"
echo "Now looking in $workingdir for $MatchPattern"
echo "Manual list:"
echo "_________________________________________"
MatchList=$(ls -1 "$MatchPattern")
echo "$MatchList"
echo "_________________________________________"
if ls -1 $MatchPattern >/dev/null 2>&1
then
echo "Found files"
mkdir -p !a2c_downloaded
echo "Making !a2c_downloaded dir"
counter=0
echo "Set counter to 0"
for f in a2c_*.txt
do
counter=$((counter+1))
echo "Download List File is: $f"
echo $counter file processing.
tempfile=${f%%.*}
tempfile="`echo "$tempfile" | sed ' s/a2c_//' `"
downdir=$tempfile
echo "Download Dir is: $downdir"
mkdir -p "$downdir"
echo ___________________________________________
echo "$endfilename"
aria2c --auto-file-renaming -i "$f" -d "$downdir" --force-sequential
echo "Will download $f to $downdir"
sleep 5
mv "$f" !a2c_downloaded/
done
else
echo "No files found"
fi
...The above works, after some fixes http://www.shellcheck.net/ told me to do. Problem is, it advises me to use double quotes in line:
if ls -1 $MatchPattern >/dev/null 2>&1
like
if ls -1 "$MatchPattern" >/dev/null 2>&1
..."Double quote to prevent globbing and word splitting.", but when I do, the script stops working for me. Should I leave it as it is? It seems to be working fine - for now.
Instead of
if [ -f a2c_* ]
then
You can try this:
file_exists() {
for _i do
[ -f "$_i" ] && break
done
}
and then
if file_exists a2c_*
then

sh: Test for existence of files

How does one test for the existence of files in a directory using bash?
if ... ; then
echo 'Found some!'
fi
To be clear, I don't want to test for the existence of a specific file. I would like to test if a specific directory contains any files.
I went with:
(
shopt -s dotglob nullglob
existing_files=( ./* )
if [[ ${#existing_files[#]} -gt 0 ]] ; then
some_command "${existing_files[#]}"
fi
)
Using the array avoids race conditions from reading the file list twice.
From the man page:
-f file
True if file exists and is a regular file.
So:
if [ -f someFileName ]; then echo 'Found some!'; fi
Edit: I see you already got the answer, but for completeness, you can use the info in Checking from shell script if a directory contains files - and lose the dotglob option if you want hidden files ignored.
I typically just use a cheap ls -A to see if there's a response.
Pseudo-maybe-correct-syntax-example-ahoy:
if [[ $(ls -A my_directory_path_variable ) ]] then....
edit, this will work:
myDir=(./*) if [ ${#myDir[#]} -gt 1 ]; then echo "there's something down here"; fi
You can use ls in an if statement thus:
if [[ "$(ls -a1 | egrep -v '^\.$|^\.\.$')" = "" ]] ; then echo empty ; fi
or, thanks to ikegami,
if [[ "$(ls -A)" = "" ]] ; then echo empty ; fi
or, even shorter:
if [[ -z "$(ls -A)" ]] ; then echo empty ; fi
These basically list all files in the current directory (including hidden ones) that are neither . nor ...
If that list is empty, then the directory is empty.
If you want to discount hidden files, you can simplify it to:
if [[ "$(ls)" = "" ]] ; then echo empty ; fi
A bash-only solution (no invoking external programs like ls or egrep) can be done as follows:
emp=Y; for i in *; do if [[ $i != "*" ]]; then emp=N; break; fi; done; echo $emp
It's not the prettiest code in the world, it simply sets emp to Y and then, for every real file, sets it to N and breaks from the for loop for efficiency. If there were zero files, it stays as Y.
Try this
if [ -f /tmp/foo.txt ]
then
echo the file exists
fi
ref: http://tldp.org/LDP/abs/html/fto.html
you may also want to check this out: http://tldp.org/LDP/abs/html/fto.html
How about this for whether directory is empty or not
$ find "/tmp" -type f -exec echo Found file {} \;
#!/bin/bash
if [ -e $1 ]; then
echo "File exists"
else
echo "Files does not exist"
fi
I don't have a good pure sh/bash solution, but it's easy to do in Perl:
#!/usr/bin/perl
use strict;
use warnings;
die "Usage: $0 dir\n" if scalar #ARGV != 1 or not -d $ARGV[0];
opendir my $DIR, $ARGV[0] or die "$ARGV[0]: $!\n";
my #files = readdir $DIR;
closedir $DIR;
if (scalar #files == 2) { # . and ..
exit 0;
}
else {
exit 1;
}
Call it something like emptydir and put it somewhere in your $PATH, then:
if emptydir dir ; then
echo "dir is empty"
else
echo "dir is not empty"
fi
It dies with an error message if you give it no arguments, two or more arguments, or an argument that isn't a directory; it's easy enough to change if you prefer different behavior.
# tested on Linux BASH
directory=$1
if test $(stat -c %h $directory) -gt 2;
then
echo "not empty"
else
echo "empty"
fi
For fun:
if ( shopt -s nullglob ; perl -e'exit !#ARGV' ./* ) ; then
echo 'Found some!'
fi
(Doesn't check for hidden files)

Shell programming, looping through files

I am trying to loop through files in a specified directory. But I can't seem to figure out the logic. I am looping through each file and asking if they want to delete that file.
#!/bin/bash
dirpath=$1
y=y
Y=Y
echo "changing directory '$dirpath' `cd $dirpath`"
for f in $1/*
do
#####################################
if test -f `ls -1 $1`
then
echo -n "remove file '$f' `ls -1` ?"
read answer
##########################
if test $answer = $y || test $answer = $Y
then
echo "Processing $f file..."
echo `rm $f`
echo "file '$f' deleted "
else
echo "file '$f' not removed"
fi#2nd if loop
############################
else
echo 'not a file'
fi#1st if loop
#######################################
done
Your code seems much more complicated that it should be. Does this fulfill your needs or are you doing some shell practice?
rm -iv DIRECTORY/*
There's no need for ls, you already have the filename. Change this:
if test -f `ls -1 $1`
to:
if test -f "$f"
Why are you using echo and backticks here? Change
echo `rm $f`
to:
rm "$f"
Here's another place you're using backticks unnecessarily. Change this:
echo "changing directory '$dirpath' `cd $dirpath`"
to:
echo "changing directory '$dirpath'"
cd "$dirpath"
Always quote variables that contain filenames.
You can have rm do the "asking" for you via its -i flag to prompt user before removal. I am assuming you want to consider only files, not directories, and not recurse any sub-directories.
#!/bin/bash
for f in $1/* ; do
if [ -f $f ] ; then
rm -i $f ;
fi
done
Without the error, can't really help, but it could be written like this, not as verbose though
rm -i *
If $1 is a relative path, then once you've cd'd into $1, the wildcard in your for loop will be meaningless. I'd recommend something more like
cd $1
for f in *; do
...
done
Since it will accept both relative and absolute paths.
Moreover, the arguments to the first test are wrong. Each time through the loop, $f will hold one filename, so your test should be like
if (test -f $f); then
You also repeat this in your echo arguments.
The following does basically what you want, with only slight modifications from your script.
#!/bin/bash
dirpath=$1
y=y
Y=Y
echo "changing directory '$dirpath' `cd $dirpath`"
for f in ./*; do
if (test -f $f); then
echo -n "remove file '$f' ?"
read answer
if (test $answer == $y) || (test $answer == $Y); then
echo "Processing $f file..."
rm $f
echo "file '$f' deleted "
else
echo "file '$f' not removed"
fi
else
echo 'not a file'
fi
done

Resources