If file modification date is older than N days - bash

This question pertains to taking action if a file has a modification date older than so many days. I'm sure it would be similar for creation date or access date, but for modification date, if I have:
file=path-name-to-some-file
N=100 # for example, N is number of days
How would I do:
if file modification time is older than N days
then
fi

Several approaches are available. One is just to ask find to do the filtering for you:
if [[ $(find "$filename" -mtime +100 -print) ]]; then
echo "File $filename exists and is older than 100 days"
fi
Another is to use GNU date to do the math:
# collect both times in seconds-since-the-epoch
hundred_days_ago=$(date -d 'now - 100 days' +%s)
file_time=$(date -r "$filename" +%s)
# ...and then just use integer math:
if (( file_time <= hundred_days_ago )); then
echo "$filename is older than 100 days"
fi
If you have GNU stat, you can ask for a file's timestamp in seconds-since-epoch, and do some math yourself (though this will potentially be a bit off on the boundary cases, since it's counting seconds -- and not taking into account leap days and such -- and not rounding to the beginning of a day):
file_time=$(stat --format='%Y' "$filename")
current_time=$(( date +%s ))
if (( file_time < ( current_time - ( 60 * 60 * 24 * 100 ) ) )); then
echo "$filename is older than 100 days"
fi
Another option, if you need to support non-GNU platforms, is to shell out to Perl (which I'll leave it to others to demonstrate).
If you're interested more generally in getting timestamp information from files, and portability and robustness constraints surrounding same, see also BashFAQ #87.

I'm surprised no one mentioned this method - it's hiding right there in man test, implied by -nt and -ot, so I suspect it's been there for a long time:
N_DAYS_AGO=/tmp/n-days-ago.$$
touch -d "$N days ago" $N_DAYS_AGO
if [ "$myfile" -ot "$N_DAYS_AGO" ]; then
...
fi

I found another great answer to the question. We use find with the -newerXY option. Here is an example:
if [[ ! $(find "$file" -newermt "8 hours ago") ]]; then
echo "File can't be older than 8 hours: $file"
exit 1
fi
Found it here: https://superuser.com/a/1106706/1090383

Create a temporary file, then set it's time to N days ago. Compare your file with the temporary file.
N=100
rf=/tmp/temp1234
echo "temporary file" &> $rf
touch -d "$N days ago" $rf
for myfile in $(find . -type f); do
if [[ $rf -nt $myfile ]]; then
echo "$myfile is older than $N days."
else
echo "$myfile is not too old."
fi
done

Related

existing bash script for deleting old backup files - how does it work?

I'm a total beginner with scripts. I have some questings regarding an old script, which should delete old backup files.
KEEP_FULL=7
KEEP_DAILY=14
KEEP_WEEKLY=30
DB_FULL_PATH=/Volumes/path
LAST_DAY=0
LAST_WEEK=0
LAST_MONTH=0
find $DB_FULL_PATH -type f| while read f; do
< <(stat -f %Sm -t "%m %V %d" $f) read -s MONTH WEEK DAY
if [ $DAY -eq $LAST_DAY ]; then
find $f -mtime +$KEEP_FULL | xargs rm
else if [ $WEEK -eq $LAST_WEEK ]; then
find $f -mtime +$KEEP_DAILY | xargs rm
else if [ $MONTH -eq $LAST_MONTH ]; then
find $f -mtime +$KEEP_WEEKLY | xargs rm
fi fi fi
export LAST_DAY=$DAY
export LAST_WEEK=$WEEK
export LAST_MONTH=$MONTH
done
Could someone explain (for dummies) what happens within the while-loop?
I understand that for each file within the folder the information (day, week, month of creation) is written into $MONTH $WEEK $DAY.
But the following logic I don't understand correctly.
This answer won't be complete because I lack informations. Also It seems overly complex for the task it should do.
The loop below process all files under /Volumes/path and stores them in the variable f.
find $DB_FULL_PATH -type f| while read f; do
# [...]
done
For each file f, this command is first performed:
< <(stat -f %Sm -t "%m %V %d" $f) read -s MONTH WEEK DAY
stat is invoked on the file f
According to the man page of stat (OSX):
-f format Display information using the specified format. See the FORMATS section for a description of valid formats.
-t timefmt Display timestamps using the specified format. This format is passed directly to strftime(3).
According to the FORMATS section of the manual page, %Sm is the combination of S and m, which stand respectively for 1) "the [last] time file was [...] modified" (m) and 2) formatted, "if applicable, should be in string format" (S, it should be the case here).
According to the manual page of strftime, %m %V %d stand for the month, the week number of the year and the day of the month as a decimal number
The output is read and stored in variables MONTH WEEK and DAY (-s stands for silent mode)
At this point, you have the month, the week number of the year and the day of the month of the time the file was modifed.
Next the hardest part:
If the day of modification of the previously processed file equals the current processed file one, then check if it was modified (-mtime) one week ago (KEEP_FULL=7) ; if so, delete it (f is passed to rm with xargs):
if [ $DAY -eq $LAST_DAY ]; then
find $f -mtime +$KEEP_FULL | xargs rm
fi
Otherwise, if the week number of the year of modification of the previously processed file (breathes) equals the current processed file one, then check if it was modified two weeks ago (KEEP_DAILY=14) ; if so, delete it:
else if [ $WEEK -eq $LAST_WEEK ]; then
find $f -mtime +$KEEP_DAILY | xargs rm
Otherwise, if the month of modification of the previously processed file (sips) equals the current processed file one, then check if it was modified 30 days ago (KEEP_WEEKLY=30) ; if so, delete it:
else if [ $MONTH -eq $LAST_MONTH ]; then
find $f -mtime +$KEEP_WEEKLY | xargs rm
fi fi fi
Retrieve the month, the week and the day of the month of the current file to process the next one with them (will be used as parts of the modification time of said "previously processed" file):
export LAST_DAY=$DAY
export LAST_WEEK=$WEEK
export LAST_MONTH=$MONTH
done
That said, to understand why such comparisons are performed, you need to know the order the first find outputs files injected to the standard input of the while loop.

How to set date range in shell script

I am writing a code in a shell script to load data from specific range but it does not stops at the data I want and instead goes past beyond that. Below is my code of shell script.
j=20180329
while [ $j -le 20180404]
do
i have problem that my loop run after the date 20180331 till 20180399 then it go to 20180401.
i want it to go from 20180331 to 20180401. not 20180332 and so on
One simple question, 3+ not so short answer...
As your request stand for shell
1. Compatible answer first
j=20180329
while [ "$j" != "20180405" ] ;do
echo $j
j=`date -d "$j +1 day" +%Y%m%d`
done
Note I used one day after as while condition is based on equality! Of course interpreting YYYYMMDD date as integer will work too:
Note 2 Care about timezone set TZ=UTC see issue further...
j=20180329
while [ $j -le 20180404 ] ;do
echo $j
j=`TZ=UTC date -d "$j +1 day" +%Y%m%d`
done
But I don't like this because if time format change, this could become an issue.
Tested under bash and shell as dash and busybox.
(using date (GNU coreutils) 8.26.
1.2 Minimize fork under POSIX shell
Before using bash bashisms, here is a way of doing this under any POSIX shell:
The power of POSIX shell is that we could use very simple converter like date and do condition over result:
#!/usr/bin/env sh
tempdir=$(mktemp -d)
datein="$tempdir/datein"
dateout="$tempdir/dateout"
mkfifo "$datein" "$dateout"
exec 5<>"$datein"
exec 6<>"$dateout"
stdbuf -i0 -o0 date -f - <"$datein" >"$dateout" +'%Y%m%d' &
datepid=$!
echo "$2" >&5
read -r end <&6
echo "$1" >&5
read -r crt <&6
while [ "$crt" -le "$end" ];do
echo $crt
echo "$crt +1 day" >&5
read -r crt <&6
done
exec 5>&-
exec 6<&-
kill "$datepid"
rm -fR "$tempdir"
Then
daterange.sh 20180329 20180404
20180329
20180330
20180331
20180401
20180402
20180403
20180404
2. bash date via printf
Under bash, you could use so-called bashisms:
Convert date to integer Epoch (Unix time), but two dates via one fork:
{
read start;
read end
} < <(date -f - +%s <<eof
20180329
20180404
eof
)
or
start=20180329
end=20180404
{ read start;read end;} < <(date -f - +%s <<<$start$'\n'$end)
Then using bash builtin printf command (note: there is $[24*60*60] -> 86400 seconds in a regular day)
for (( i=start ; i<=end ; i+=86400 )) ;do
printf "%(%Y%m%d)T\n" $i
done
3. Timezone issue!!
Warning there is an issue around summer vs winter time:
As a function
dayRange() {
local dR_Start dR_End dR_Crt
{
read dR_Start
read dR_End
} < <(date -f - +%s <<<${1:-yesterday}$'\n'${2:-tomorrow})
for ((dR_Crt=dR_Start ; dR_Crt<=dR_End ; dR_Crt+=86400 )) ;do
printf "%(%Y%m%d)T\n" $dR_Crt
done
}
Showing issue:
TZ=CET dayRange 20181026 20181030
20181026
20181027
20181028
20181028
20181029
Replacing printf "%(%Y%m%d)T\n" $dR_Crt by printf "%(%Y%m%dT%H%M)T\n" $dR_Crt could help:
20181026T0000
20181027T0000
20181028T0000
20181028T2300
20181029T2300
In order to avoid this issue, you just have to localize TZ=UTC at begin of function:
local dR_Start dR_End dR_Crt TZ=UTC
Final step for function: Avoiding useless forks
In order to improve performances, I try to reduce forks, avoiding syntax like:
for day in $(dayRange 20180329 20180404);do ...
# or
mapfile range < <(dayRange 20180329 20180404)
I use ability of function to directly set submited variables:
There is my purpose:
dayRange() { # <start> <end> <result varname>
local dR_Start dR_End dR_Crt dR_Day TZ=UTC
declare -a dR_Var='()'
{
read dR_Start
read dR_End
} < <(date -f - +%s <<<${1:-yesterday}$'\n'${2:-tomorrow})
for ((dR_Crt=dR_Start ; dR_Crt<=dR_End ; dR_Crt+=86400 )) ;do
printf -v dR_Day "%(%Y%m%d)T\n" $dR_Crt
dR_Var+=($dR_Day)
done
printf -v ${3:-dRange} "%s" "${dR_Var[*]}"
}
Then quick little bug test:
TZ=CET dayRange 20181026 20181030 bugTest
printf "%s\n" $bugTest
20181026
20181027
20181028
20181029
20181030
Seem fine. This could be used like:
dayRange 20180329 20180405 myrange
for day in $myrange ;do
echo "Doing something with string: '$day'."
done
2.2 Alternative using shell-connector
There is a shell function for adding background command in order to reduce forks.
wget https://f-hauri.ch/vrac/shell_connector.sh
. shell_connector.sh
Initiate background date +%Y%m%d and test: #0 must answer 19700101
newConnector /bin/date '-f - +%Y%m%d' #0 19700101
Then
j=20190329
while [ $j -le 20190404 ] ;do
echo $j; myDate "$j +1 day" j
done
3.++ Little bench
Let's try little 3 years range:
j=20160329
time while [ $j -le 20190328 ] ;do
echo $j;j=`TZ=UTC date -d "$j +1 day" +%Y%m%d`
done | wc
1095 1095 9855
real 0m1.887s
user 0m0.076s
sys 0m0.208s
More than 1 second on my system... Of course, there are 1095 forks!
time { dayRange 20160329 20190328 foo && printf "%s\n" $foo | wc ;}
1095 1095 9855
real 0m0.061s
user 0m0.024s
sys 0m0.012s
Only 1 fork, then bash builtins -> less than 0.1 seconds...
And with newConnector function:
j=20160329
time while [ $j -le 20190328 ] ;do echo $j
myDate "$j +1 day" j
done | wc
1095 1095 9855
real 0m0.109s
user 0m0.084s
sys 0m0.008s
Not as quick than using builtin integer, but very quick anyway.
Store the max and min dates using seconds since epoch. Don't use dates - they are not exact (GMT? UTC? etc.). Use seconds since epoch. Then increment your variable with the number of seconds in a day - ie. 24 * 60 * 60 seconds. In your loop, you can convert the number of seconds since epoch back to human readable date using date --date=#<number>. The following will work with POSIX shell and GNU's date utlity:
from=$(date --date='2018/04/04 00:00:00' +%s)
until=$(date --date='2018/04/07 00:00:00' +%s)
counter="$from"
while [ "$counter" -le "$until" ]; do
j=$(date --date=#"$counter" +%Y%m%d)
# do somth with j
echo $j
counter=$((counter + 24 * 60 * 60))
done
GNU's date is a little strange when parsing it's --date=FORMAT format string. I suggest to always feed it with %Y/%m/%d %H/%M/%S format string so that it always knows how to parse it.

Bash on macOS - Get a list of dates for every Saturday on a given year

In bash on macOS, I would like to write a small script with dates (or any other program that would do) that gives me a list of dates in the format yyyymmdd of every Saturday of a given year and saves it to a variable.
For example, if I wanted to have a list of dates for all Saturdays of the year 1850, it should somehow look like this:
var = [ 18500105, 18500112, 18500119, …, 18501228 ]
with the below code:
list=()
for month in `seq -w 1 12`; do
for day in `seq -w 1 31`; do
list=( $(gdate -d "1850$month$day" '+%A %Y%m%d' | grep 'Saturday' | egrep -o '[[:digit:]]{4}[[:digit:]]{2}[[:digit:]]{2}' | tee /dev/tty) )
done
done
However, the above command does not write anything in the array list although it gives me the right output with tee.
How can I fix these issues?
Modifying Dennis Williamson's answer slightly to suit your requirement and to add the results into the array. Works on the GNU date and not on FreeBSD's version.
#!/usr/bin/env bash
y=1850
for d in {0..6}
do
# Identify the first day of the year that is a Saturday and break out of
# the loop
if (( $(date -d "$y-1-1 + $d day" '+%u') == 6))
then
break
fi
done
array=()
# Loop until the last day of the year, increment 7 days at a
# time and append the results to the array
for ((w = d; w <= $(date -d "$y-12-31" '+%j'); w += 7))
do
array+=( $(date -d "$y-1-1 + $w day" '+%Y%m%d') )
done
Now you can just print the results as
printf '%s\n' "${array[#]}"
To set up the GNU date on MacOS you need to do brew install coreutils and access the command as gdate to distinguish it from the native version provided.
Argh, just realised you need it for MacOS date.
I will leave the answer for others that do not have that restriction, but it will not work for you.
This is not quite what you want, but close:
year=1850
firstsat=$(date -d $year-01-01-$(date -d $year-01-01 +%w)days+6days +%Y%m%d)
parset a 'date -d '$firstsat'+{=$_*=7=}days +%Y%m%d' ::: {0..52}
echo ${a[#]}
It has the bug, that it finds the next 53 Saturdays, and the last of those may not be in current year.
parset is part of GNU Parallel.
I didn't do much error checking, but here's another implementation.
Takes day of week and target year as arguments.
Gets the julian day of the first matching weekday requested -
gets the epoch seconds of noon on that day -
as long as the year matches what was requested, adds that date to the array and adds a week's worth of seconds to the tracking variable.
Lather, rinse, repeat until no longer in that year.
$: typeset -f alldays
alldays () { local dow=$1 year=$2 julian=1;
until [[ "$dow" == "$( date +%a -d $year-01-0$julian )" ]]; do (( julian++ )); done;
es=$( date +%s -d "12pm $year-01-0$julian" );
allhits=( $( while [[ $year == "$( date +%Y -d #$es )" ]]; do date +%Y%m%d -d #$es; (( es+=604800 )); done; ) )
}
$: time alldays Sat 1850
real 0m9.931s
user 0m1.025s
sys 0m6.695s
$: printf "%s\n" "${allhits[#]}"
18500105
18500112
18500119
18500126
18500202
18500209
18500216
18500223
18500302
18500309
18500316
18500323
18500330
18500406
18500413
18500420
18500427
18500504
18500511
18500518
18500525
18500601
18500608
18500615
18500622
18500629
18500706
18500713
18500720
18500727
18500803
18500810
18500817
18500824
18500831
18500907
18500914
18500921
18500928
18501005
18501012
18501019
18501026
18501102
18501109
18501116
18501123
18501130
18501207
18501214
18501221
18501228

Problems with "while" bash

I need to control files in a folder... The script has to wait the file until it exists...
These files have the name... The format is file_d01_YYYY-MM-DD_HH:00:00. For example:
file_d01_2018-11-12_00:00:00
file_d01_2018-11-12_01:00:00
And so on, for 7 days ahead.
!/bin/bash
ZZ=`date +%k`
date=$(date +%Y%m%d)
if [[ $ZZ -ge 2 ]] && [[ $ZZ -lt 14 ]] ; then #03:45 UTC
ZZ=00
PARAM=`date +%Y%m%d`$ZZ
PARAM2=`date +%Y-%m-%d`
elif [[ $ZZ -ge 14 ]] && [[ $ZZ -lt 23 ]] ; then #15:45 UTC
ZZ=12
PARAM=`date +%Y%m%d`$ZZ
PARAM2=`date +%Y-%m-%d`
fi
rundir=/home/$PARAM/wrfv3
dir=/home/$PARAM
data=$(date +%Y-%m-%d)
data1=$(date -d "1 day" +%Y-%m-%d)
data2=$(date -d "2 day" +%Y-%m-%d)
data3=$(date -d "3 day" +%Y-%m-%d)
data4=$(date -d "4 day" +%Y-%m-%d)
data5=$(date -d "5 day" +%Y-%m-%d)
data6=$(date -d "6 day" +%Y-%m-%d)
days=( "$data" "$data1" "$data2" "$data3" "$data4" "$data5" "$data6" ) #array of days
values=( {00..23} ) #array of 24 hours
echo ${#values[#]}
# Here, using to loops, I check if files exist...for every day and hour
for day in "${days[#]}"; do
for value in "${values[#]}"; do
echo file_d01_${day}_${value}:00:00
while [ ! -f $rundir/file_d01_2018-11-15_20:00:00 ] # while file doesn't exist...wait...and repeat checking till it exists...
do
echo "waiting for the file"
sleep 10
done
echo "file exists"
sleep 5
done
done
I receive always "waiting for the file"...and they exist... where is the problema in the code?
You should add the double quotes "" to protect the path. It's a good practice. Also bash expansion escapes the : character, so maybe it is an issue in your context (not in the one i did the test).
while [ ! -e "$rundir/file_d01_2018-11-15_20:00:00" ]
I would suggest to follow those steps:
Protect the path with double quotes "" (not simple ones, otherwise $rundir won't be expanded)
Write echo "waiting for the file $rundir/file_d01_2018-11-15_20:00:00" to see what path you're testing
Additionally, use -e to see any changes (-e checks for a path existence, not only a regular file one)
Note: the brackets [ ] invokes in fact test. So, man test will give you the operators you can use and their meanings. Also nowadays bash has double brackets [[ ]] as built-in operators, more powerful, which can be used instead.
The code in the question contains:
echo file_d01_${day}_${value}:00:00
while [ ! -f $rundir/file_d01_2018-11-15_20:00:00 ]
It's calculating a file name, echoing it, and then checking for a (probably different) fixed, unchanging, file name. It should check for the calculated file name. For example:
file=file_d01_${day}_${value}:00:00
echo "$file"
while [ ! -f "$rundir/$file" ]
To make debugging easier, it would be better to have:
filepath=$rundir/file_d01_${day}_${value}:00:00
echo "$filepath"
while [ ! -f "$filepath" ]
The full code in the question has many issues (starting with a broken shebang line). It's a good idea to make Bash code Shellcheck-clean.

Test a file date with bash

I am trying to test how old ago a file was created (in seconds) with bash in an if statement. I need creation date, not modification.
Do you have any idea how to do this, without using a command like find with grep?
I'm afraid I cann't answer the question for creation time, but for last modification time you can use the following to get the epoch date, in seconds, since filename was last modified:
date --utc --reference=filename +%s
So you could then so something like:
modsecs=$(date --utc --reference=filename +%s)
nowsecs=$(date +%s)
delta=$(($nowsecs-$modsecs))
echo "File $filename was modified $delta secs ago"
if [ $delta -lt 120 ]; then
# do something
fi
etc..
Update
A more elgant way of doing this (again, modified time only): how do I check in bash whether a file was created more than x time ago?
Here is the best answer I found at the time being, but it's only for the modification time :
expr `date +%s` - `stat -c %Y /home/user/my_file`
If your system has stat:
modsecs=$(stat --format '%Y' filename)
And you can do the math as in Joel's answer.
you can use ls with --full-time
file1="file1"
file2="file2"
declare -a array
i=0
ls -go --full-time "$file1" "$file2" | { while read -r a b c d time f
do
time=${time%.*}
IFS=":"
set -- $time
hr=$1;min=$2;sec=$3
hr=$(( hr * 3600 ))
min=$(( min * 60 ))
totalsecs=$(( hr+min+sec ))
array[$i]=$totalsecs
i=$((i+1))
unset IFS
done
echo $(( ${array[0]}-${array[1]} ))
}

Resources