I have a folder that contains database backups, but i want to automate it using cron to delete old backups.
so i created the following script
#Get the current year
YEAR=$(date +'%Y')
#Get the current month
MONTH=$(date +'%m')
#Delete data from previous months
deleteOldData() { ls /root/copy/dbbackup/smpp_credits/ | awk -F "-" -v m="$MONTH" '$2 < m' | xargs -d "\n" rm -rf ;}
#Delete data from previous years ( if any )
deletePrevYearData() { ls /root/copy/dbbackup/smpp_credits/ | awk -F "-" -v y="$YEAR" '$3 < y' | xargs -d "\n" rm -rf ;}
deleteOldData
deletePrevYearData
Executing ls /root/copy/dbbackup/smpp_credits/ | awk -F "-" -v m="$MONTH" '$2 < m' in the terminal works as expected, (it lists the required files).
but upon appending | xargs -d "\n" rm -rf the code runs and returns without any output, and checking the directory reveals that the files are still there. By the way, this code is stored and executed from a .sh file
Assuming GNU find and date, -newermt can be used to compare a file's modification time against a specific date given as an argument:
delete_older_than_date="$(date +'%Y-%m-01')"
find /root/copy/dbbackup/smpp_credits \
-maxdepth 1 \
-type f \
'!' -newermt "$delete_older_than_date" \
-exec rm -rf -- '{}' +
Parsing ls output is widely considered to be a bad idea. I would try a find command, which should be cleaner.
find /root/copy/dbbackup/smpp_credits/ -maxdepth 1 -mtime +365 -exec rm -rf {} \;
from here. You can use -mtime +30 for files that are older than one month.
Related
Below is the command I am using for moving files from dir a to dir b
ls /<someloc>/a/* | tail -2000 | xargs -I{} mv {} /<someloc>/b/
-bash: /usr/bin/ls: Argument list too long
folder a has files in millions ..
Need your help to fix this please.
If the locations of both directories are on the same disk/partition and folder b is originally empty, you can do the following
$ rmdir /path/to/b
$ mv /other/path/to/a /path/to/b
$ mkdir /other/path/to/a
If folder b is not empty, then you can do something like this:
find /path/to/a/ -type f -exec mv -t /path/to/b {} +
If you just want to move 2000 files, you can do
find /path/to/a/ -type f -print | tail -2000 | xargs mv -t /path/to/b
But this can be problematic with some filenames. A cleaner way would be is to use -print0 of find, but the problem is that head and tail can't process those, so you have to use awk for this.
# first 2000 files (mimick head)
find /path/to/a -type f -print0 \
| awk 'BEGIN{RS=ORS="\0"}(NR<=2000)' \
| xargs -0 mv -t /path/to/b
# last 2000 files (mimick tail)
find /path/to/a -type f -print0 \
| awk 'BEGIN{RS=ORS="\0"}{a[NR%2000]=$0}END{for(i=1;i<=2000;++i) print a[i]}' \
| xargs -0 mv -t /path/to/b
The ls in the code in the question does nothing useful. The glob (/<someloc>/a/*) produces a sorted list of files, and ls just copies it (after re-sorting it), if it works at all. See “Argument list too long”: How do I deal with it, without changing my command? for the reason why ls is failing.
One way to make the code work is to replace ls with printf:
printf '%s\n' /<someloc>/a/* | tail -2000 | xargs -I{} mv {} /<someloc>/b/
printf is a Bash builtin, so running it doesn't create a subprocess, and the "Argument list too long" problem doesn't occur.
This code will still fail if any of the files contains a newline character in its name. See the answer by kvantour for alternatives that are not vulnerable to this problem.
my command was this
ls -l|grep "\-[r,-][w,-]x*"|tr -s " " | cut -d" " -f9
but for the result I get all the files, not only the ones for which user has a right to execute ( the first x bit is set on).
I'm running linux ubuntu
You can use find with the -perm option:
find . -maxdepth 1 -type f -perm -u+x
OK -- if you MUST use grep:
ls -l | grep '^[^d]..[sx]' | awk '{ print $9 }'
Don't use grep. If you want to know if a file is executable, use test -x. To check all files in the current directory, use find or a for loop:
for f in *; do test -f "$f" -a -x "$f" && echo "$f"; done
or
find . -maxdepth 1 -type f -exec test -x {} \; -print
Use awk with match
ls -l|awk 'match($1,/^...x/) {print $9}'
match($1,/^...x/): match first field for the regular expression ^...x, ie search for owner permission ending in x.
I have a script that finds log files older than x days within a specified directory and removes them.
find $LOG_ARCHIVE/* -mtime +$DAYS_TO_KEEP_LOGS -exec rm -f {} \;
This is working as expected but I would like to have the option to print the processing to the screen and log file so I know what files (if any) have been deleted. I've tried appending tee at the end but have had no success.
find $LOG_ARCHIVE/* -mtime +$DAYS_TO_KEEP_LOGS -exec rm -fv {} \; | tee -a $LOG
There are multiple ways the task can be done.
One possibility is to simply run find twice:
find "$LOG_ARCHIVE" -mtime +"$DAYS_TO_KEEP_LOGS" -print > "$LOG"
find "$LOG_ARCHIVE" -mtime +"$DAYS_TO_KEEP_LOGS" -exec rm -f {} +
Another possibility is to use tee along with (GNU extensions) -print0 to find and -0 to xargs:
find "$LOG_ARCHIVE" -mtime +"$DAYS_TO_KEEP_LOGS" -print0 |
tee "$LOG" |
xargs -0 rm -f
With this version, the log file will have null bytes at the end of each file name. You can arrange to replace those with newlines if you don't mind the possible ambiguity:
find "$LOG_ARCHIVE" -mtime +"$DAYS_TO_KEEP_LOGS" -print0 |
tee >(tr '\0' '\n' >"$LOG") |
xargs -0 rm -f
This uses Bash (and Korn shell) process substitution to pass the log file through tr to map null bytes '\0' to newlines '\n'.
Another way of doing it is to write a tiny custom script (call it remove-log.sh):
printf '%s\n' "$#" >> "$LOG"
rm -f "$#"
and then use:
find "$LOG_ARCHIVE" -mtime +"$DAYS_TO_KEEP_LOGS" -exec bash remove-log.sh {} +
Note that the script needs to see the value of $LOG, so that must be exported as an environment variable. You could avoid that by passing the log name explicitly:
logfile="$1"
shift
printf '%s\n' "$#" >> "$logfile"
rm -f "$#"
plus:
find "$LOG_ARCHIVE" -mtime +"$DAYS_TO_KEEP_LOGS" -exec bash remove-log.sh "$LOG" {} +
Note that both of these use >> to append because the script might be invoked more than once (though it probably won't be). The onus is on you to ensure that the log file is empty before you run the find command.
Note that I dropped the /* from the path argument for find; it wasn't really needed. You might want to add -type f to ensure that only files are removed. The + is a feature from the POSIX 2008 specification of find which makes find act rather like xargs without needing to explicitly use xargs.
find $LOG_ARCHIVE/* -mtime +$DAYS_TO_KEEP_LOGS -exec sh -c 'echo {} |tee -a "$LOG"; rm -f {}' \;
Try and see if it works.
I have a script (executed periodically via cron) that downloads the latest version of WhatsApp from their server. I would like to retain just the latest version with the filename WhatsApp_x.x.xx in my server with a softlink latest.apk.
#!/bin/bash
# Get the local version
oldVer=$(ls -v1 | grep -v latest | tail -n 1 | awk -F "_" '{print $2}' | grep -oP ".*(?=.apk)")
# Get the server version
newVer=$(wget -q -O - "$#" whatsapp.com/android | grep -oP '(?<=Version )([\d.]+)')
# Check if the server version is newer
newestVer=$(echo -e "$oldVer\n$newVer" | sort -n | tail -n 1)
#Download the newer versino
[ "$newVer" = "$newestVer" ] && [ "$oldVer" != "$newVer" ] && wget -O WhatsApp_${newVer}_.apk http://www.whatsapp.com/android/current/WhatsApp.apk || echo "The newest version already downloaded"
#Delete all files that not is a new version
find ! -name "*$newVer*" ! -type d -exec rm -f {} \;
# set the link to the latest
ln -sf $(ls -v1 | grep -v latest| tail -n1) latest.apk
This is how my /var/www/APK looks like:
/var/www/APK$ tree
.
├── latest.apk -> WhatsApp_2.12.96_.apk
├── script.sh
└── WhatsApp_2.12.96_.apk
But this command:
find ! -name "*$newVer*" ! -type d -exec rm -f {} \;
It's also deleting the script.sh file. How can I modify the statement to not affect other files? I can't think of anything.
This is the cronjob, if that helps:
* * * * * sh /var/www/APK/script.sh
With find, you can chain together multiple conditions of the same type. This leaves you with a couple of options:
You could blacklist other specific files, like this:
find ! -name "*$newVer*" ! -name 'script.sh' ! -type d -delete
Or just whitelist the .apk extension:
find -name '*.apk' ! -name "*$newVer*" ! -type d -delete
find /var/www/APK -type f -name '*.apk' -print |
sort -V |
tail -n +2 |
xargs echo rm
That finds the .apk files under the particular directory,
sorts them by version (may require GNU sort),
removes all but the newest version from the list,
and shows you which ones will be removed.
If you're satisfied it's finding the right files, take out echo
Taking a different approach: I'm going to assume that all the APK files are in the same directory, and that the filenames do not contain whitespace.
#!/bin/bash
shopt -s extglob nullglob
cd /var/www/APK
apk_files=( printf "%s\n" !(latest).apk | sort -V )
newest=${apk_files[0]}
for (( i=1; i < ${#apk_files[#]}; i++ )); do
echo rm "${apk_files[i]}"
done
ln -f -s "$newest" latest.apk
Is it possible to get the modification date and time of a folder?
I know you can use stat -f "%m" folder, but it doesn't reflect sub-files/folders changes.
Things that doesn't work:
ls -l folder - doesn't reflect changes inside the folder
stat -f "%m" folder - same as above
date -r folder - same again
find foo bar baz -printf - the printf option doesn't exist on my version of find
Versions of things:
OS: Mac OS X 10.7.1
Bash: GNU bash, version 3.2.48(1)-release (x86_64-apple-darwin11)
Solution:
find . -exec stat -f "%m" \{} \; | sort -n -r | head -1
Explanation:
the find command traverses the current directory (.) and for each file encountered executes (-exec) the command stat -f "%m". stat -f "%m" prints the last modification unix timestamp of the file.
sort -n -r sorts the output of the find command numerically (-n) in reverse order (-r). This will list the latest modification timestamp first.
head -1 then extracts the first line of the output from sort. This is the latest modification unix timestamp of all the files.
You could try 'date -r folder' to give you a date last modified
You could always get it from ls :
ls -ld mydir | awk -F' ' '{ print $6 " "$7 }'
if you need to clear cache after build . then you can check the age of the last change and delete it like this
sh("find ~/Library/Developer/Xcode/DerivedData/ -type d -maxdepth 1 -mmin +360 -name 'Cache-folder-*' -print0 | xargs -0 -I {} /bin/rm -rf '{}' || echo 'There is no such folder! Start script execution' ; exit 0")
sh("find ~/Library/Developer/Xcode/DerivedData/ -type d -maxdepth 1 -mtime 0 -name 'Cache-folder-*' -ls -exec rm -r {} \\;")