I want to create a bash program that lists only files that were modified in the last year.
Hint: the -l output of ls will be helpful
This is from my class exercise but I don't know how to start. Please help.
Thank you in advance.
For testing purposes, you can create a file dated last year with touch command:
touch oldFile -d "Mar 9 2017"
Then you can use grep and date commands as follows:
ls -l -1 | grep "$(date "+%Y" --date="1 year ago")"
where date "+%Y" --date="1 year ago" will expand into the previous year, that is 2017. Result:
-rw-rw-r-- 1 userX userX 0 Mar 9 2017 oldFile
Following Cyrus' link of not parsing ls output, I suggest using:
find . -type f -newermt 2017-01-01 ! -newermt 2017-12-31 -ls
Not -l, but -ls.
Or, to get a real answer to the question that also works next year:
#!/bin/bash
last_year=$(date +%Y -d "1 year ago")
yearstart=$(date +%Y-%m-%d -d "1 january")
find . -type f -newermt $last_year-01-01 ! -newermt $yearstart -ls
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Closed 9 years ago.
Improve this question
How do I use the UNIX command find to search for files created on a specific date?
As pointed out by Max, you can't, but checking files modified or accessed is not all that hard. I wrote a tutorial about this, as late as today. The essence of which is to use -newerXY and ! -newerXY:
Example: To find all files modified on the 7th of June, 2007:
$ find . -type f -newermt 2007-06-07 ! -newermt 2007-06-08
To find all files accessed on the 29th of september, 2008:
$ find . -type f -newerat 2008-09-29 ! -newerat 2008-09-30
Or, files which had their permission changed on the same day:
$ find . -type f -newerct 2008-09-29 ! -newerct 2008-09-30
If you don't change permissions on the file, 'c' would normally correspond to the creation date, though.
Use this command to search for files and folders on /home/ add a time period of time according to your needs:
find /home/ -ctime time_period
Examples of time_period:
More than 30 days ago: -ctime +30
Less than 30 days ago: -ctime -30
Exactly 30 days ago: -ctime 30
It's two steps but I like to do it this way:
First create a file with a particular date/time. In this case, the file is 2008-10-01 at midnight
touch -t 0810010000 /tmp/t
Now we can find all files that are newer or older than the above file (going by file modified date). You can also use -anewer for accessed and -cnewer file status changed.
find / -newer /tmp/t
find / -not -newer /tmp/t
You could also look at files between certain dates by creating two files with touch
touch -t 0810010000 /tmp/t1
touch -t 0810011000 /tmp/t2
This will find files between the two dates & times
find / -newer /tmp/t1 -and -not -newer /tmp/t2
You could do this:
find ./ -type f -ls |grep '10 Sep'
Example:
[root#pbx etc]# find /var/ -type f -ls | grep "Dec 24"
791235 4 -rw-r--r-- 1 root root 29 Dec 24 03:24 /var/lib/prelink/full
798227 288 -rw-r--r-- 1 root root 292323 Dec 24 23:53 /var/log/sa/sar24
797244 320 -rw-r--r-- 1 root root 321300 Dec 24 23:50 /var/log/sa/sa24
You can't. The -c switch tells you when the permissions were last changed, -a tests the most recent access time, and -m tests the modification time. The filesystem used by most flavors of Linux (ext3) doesn't support a "creation time" record. Sorry!
#Max: is right about the creation time.
However, if you want to calculate the elapsed days argument for one of the -atime, -ctime, -mtime parameters, you can use the following expression
ELAPSED_DAYS=$(( ( $(date +%s) - $(date -d '2008-09-24' +%s) ) / 60 / 60 / 24 - 1 ))
Replace "2008-09-24" with whatever date you want and ELAPSED_DAYS will be set to the number of days between then and today. (Update: subtract one from the result to align with find's date rounding.)
So, to find any file modified on September 24th, 2008, the command would be:
find . -type f -mtime $(( ( $(date +%s) - $(date -d '2008-09-24' +%s) ) / 60 / 60 / 24 - 1 ))
This will work if your version of find doesn't support the -newerXY predicates mentioned in #Arve:'s answer.
With the -atime, -ctime, and -mtime switches to find, you can get close to what you want to achieve.
cp `ls -ltr | grep 'Jun 14' | perl -wne 's/^.*\s+(\S+)$/$1/; print $1 . "\n";'` /some_destination_dir
I found this scriplet in a script that deletes all files older than 14 days:
CNT=0
for i in $(find -type f -ctime +14); do
((CNT = CNT + 1))
echo -n "." >> $PROGRESS
rm -f $i
done
echo deleted $CNT files, done at $(date "+%H:%M:%S") >> $LOG
I think a little additional "man find" and looking for the -ctime / -atime etc. parameters will help you here.
This question already has answers here:
Delete all files older than 30 days, based on file name as date
(3 answers)
Closed 3 years ago.
I have CSV files get updated every day and we process the files and delete the files older than 30 days based on the date in the filename.
Example filenames :
XXXXXXXXXXX_xx00xx_**20171001**.000000_0.csv
I would like to schedule the job in crontab to delete 30 days older files daily.
Path could be /mount/store/
XXXXXXXXXXX_xx00xx_**20171001**.000000_0.csv
if [ $(date -d '-30 days' +%Y%m%d) -gt $D ]; then
rm -rf $D
fi
this above script doesn't seem to help me. Kindly help me on this.
I have been trying this for last two days.
Using CENTOS7
Thanks.
For all files:
Extract the date
touch the file with that date
delete files with the -mtime option
Do this in the desired dir for all files:
f=XXXXXXXXXXX_xx00xx_20171001.000000_0.csv
d=$(echo $f | sed -r 's/[^_]+_[^_]+_(20[0-9]{6})\.[0-9]{6}_.\.csv/\1/')
touch -d $d $f
After performing that for the whole dir, delete the older-thans:
find YourDir -type f -mtime +30 -name "*.csv" -delete
Gnu-sed has the -delete option. Other finds might need -exec rm ... .
Test before. Other pitfalls are different kind of dates, affected by touch (mtime, ctime, atime).
Test, manipulating the date with touch:
touch XXXXXXXXXXX_xx00xx_20171001.000000_0.csv
f=XXXXXXXXXXX_xx00xx_20171001.000000_0.csv; d=$(echo $f | sed -r 's/[^_]+_[^_]+_(20[0-9]{6})\.[0-9]{6}_.\.csv/\1/'); touch -d $d $f
ls -l $f
-rw-rw-r-- 1 stefan stefan 0 Okt 1 00:00 XXXXXXXXXXX_xx00xx_20171001.000000_0.csv
An efficient way to extract date from filename is to use variable expansions
f=XXXXXXXXXXX_xx00xx_20171001.000000_0.csv
d=${f%%.*} # removes largest suffix .*
d=${d##*_} # removes largest prefix *_
Or to use bash specific regex
if [[ $f =~ [0-9]{8} ]]; then echo "$BASH_REMATCH"; fi
Here is a solution if you have dgrep from dateutils.
ls *.csv | dateutils.dgrep -i '%Y%m%d' --le $(date -d "-30 day" +%F) | xargs -d '\n' rm
First we can use either ls or find to obtain a list of filenames. We can then pipe the results to dgrep to filter the filenames that contains a date string which matches our condition (in this case older than 30 days). Finally, we pipe the result to xargs rm to remove all the matched files.
-i '%Y%m%d' input date format as specified in your filename
--le $(date -d "-30 day" +%F) filter dates that are older than 30 days
You can change rm to printf "%s\n" to test the command before actually deleting it.
The following approach does not look at any generation time information of the file, it assumes the date in the filename is unrelated to the day the file is created.
#/usr/bin/env bash
d=$(date -d "-30 days" "+%Y%m%d")
for file in /yourdir/*csv; do
date=${file:$((${#file}-21)):8}
(( date < d )) && rm $file
done
I have a directory on unix server.
cd /home/client/files
It has multiple client folders like these below.
cd /home/client/files/ibm
cd /home/client/files/aol
cd /home/client/files/citi
All of them send us a file starting with either lower or upper case like below:
pre-ibm-03222017
PRE-aol-170322
Once we recieve the files, we process them and convert pre to pro as below:
pro-ibm-03222017
PRO-aol-170322
I want to count the files processed each day. Here is what I am looking for:
If I can just get the total count per client, that would be perfect. If not, then the total count overall.
Keep in mind it has all files as below:
cd /home/client/files/ibm
pre-ibm-03222017
pro-ibm-03222017
cd /home/client/files/aol
PRE-aol-170322
PRO-aol-170322
And I want to COUNT ONLY the PRO/pro that will either be lower or upper case. One folder can get more than 1 file per day.
I am using the below command:
find /home/client/files -type f -mtime -1 -exec ls -1 {} \;| wc -l
But it is giving me the total count of pre and pro files and also it is counting files for last 24 hours....and not the current day only.
For Example. It is currently 09:00 PM. The above command include files received yesterday between 09:00 PM and 12:00 AM as well. I don't wan't those. In other words if I run it at 01:00 AM....it should have all files for 1 hour only and not last 24 hours.
Thanks
---- Update -----
This works great for me.
touch -t 201703230000 first
touch -t 201703232359 last
find /home/client/files/ -newer first ! -newer last | grep -i pro | wc -l
Now, I was just wondering if I can pass the above as parameter.
For example, instead of using touch -t date and alias.....I want to type shortcuts and dates only to get the output. I have made the following aliases:
alias reset='touch -t `date +%m%d0000` /tmp/$$'
alias count='find /home/client/files/ -type f -newer /tmp/$$ -exec ls -1 {} \; | grep -i pro | wc -l'
This way as soon as I logon to the server, I type reset and then I type count and I get my daily number.
I was wondering if I can do something similar for any duration of days by setting date1 and date2 as aliases. If not, then perhaps a short script that would ask for parameters.
What about this?
touch -t `date +%m%d0000` /tmp/$$
find /home/client/files -type f -newer /tmp/$$ -exec ls -1 {} \; | grep -i pro | wc -l
rm /tmp/$$
Other options for finding a file created today can be found in this question:
How do I find all the files that were created today
Actually, a better way to do this is to just use this:
find /home/client/files -type f -m 0 | grep -i pro | wc -l
You can replace -m 0 with -m 5 to find files 5 days old.
For the same day issue u can use -daystart (GNU find)
The regex define a contains of /pre
find /home/client/files -regex '.*\/[pP][rR][eE].*' -type f -daystart -mtime -1 -exec ls -1 {} \;| wc -l
I am struggling with listing files modified in a specific month (for example, in February).
Here are several unsuccessful attempts:
I tried creating temporary files and setting their timestamp to the first time in the next month and the first time in the target month and use -newer in find, like this:
find -newer "$from" ! -newer "$to"
This lists files modified in the time interval ($from, $to], but I would like the time interval [$from, $to) (otherwise, there would be false positives on files created on the first second in the next month). Listing files modified in February is another problem, since this would require to set one of the timestamps to the greatest one still in February, but the number of days in February varies depending on whether it is a leap year or not, which requires extra checking.
If I use ls, I encounter a lot of complication when parsing, because of the possibility that user names or groups contain whitespace.
Is there an easy way and relatively portable way for doing this (so it works for any month, regardless of file names, etc.)?
date allows you to easily generate timestamps for purposes like that:
date -d "01-Mar-2011 -1 sec" # last second of Feb-2011
Fortunately, the same syntax is possible in find:
month="Mar-2010"
find . -newermt "01-$month -1 sec" -and -not -newermt "01-$month +1 month -1 sec"
will find all files modified in March 2010.
See the option -newerXY in find's man page.
Well, I can create files that have the minimum timestamp and the maximum timestamp in February, and files that are just beyond February in each direction.
$ touch -t 201102010000.01 from
$ touch -t 201102282359.59 to
$ touch -t 201103010000.01 march
$ touch -t 201101312359.59 january
$ ls -l
total 0
-rw-r--r-- 1 mike None 0 Feb 1 00:00 from
-rw-r--r-- 1 mike None 0 Jan 31 23:59 january
-rw-r--r-- 1 mike None 0 Mar 1 00:00 march
-rw-r--r-- 1 mike None 0 Feb 28 23:59 to
Then using GNU 'find' like this seems to show just the files whose timestamp is in February.
$ find -newermt '2011-02-01' ! -newermt '2011-03-01' -print
./from
./to
I don't know how portable these arguments are to other versions of 'find'.
Adding to Pumbaa80's answer:
In my pre-production environment, find does not support -newermt.
What I did instead was:
Get a list of all possible files (via find, ls etc.)
Generate the timestamps of the last second of last month and this month
LAST_MONTH=$(date -d "01-Jun-2015" -1 sec +%s)
THIS_MONTH=$(date -d "31-Jul-2015" +%s)
Iterate over the list from point 1 and compare the timestamp of each file with the timestamps from point 2
for file in $LIST_OF_FILES
do
TIMESTAMP=$(stat -c"%Y" $file)
if (( $LAST_MONTH < $TIMESTAMP ))
then
if (( $TIMESTAMP < $THIS_MONTH ))
then
echo "Your code here"
fi
fi
done
A workaround could be to use the -printf option to find:
find -printf "%Cm %p\\n"| egrep ^02 |cut -b4-
I don't think find can filter the -printf result itself, nor can it filter on date elements.
edit or if you really want the ls-like output:
find -printf "%Cm " -ls | egrep ^02 |cut -b4-
I'm writing a bash script that needs to delete old files.
It's currently implemented using :
find $LOCATION -name $REQUIRED_FILES -type f -mtime +1 -delete
This will delete of the files older than 1 day.
However, what if I need a finer resolution that 1 day, say like 6 hours old? Is there a nice clean way to do it, like there is using find and -mtime?
Does your find have the -mmin option? That can let you test the number of mins since last modification:
find $LOCATION -name $REQUIRED_FILES -type f -mmin +360 -delete
Or maybe look at using tmpwatch to do the same job. phjr also recommended tmpreaper in the comments.
Here is the approach that worked for me (and I don't see it being used above)
$ find /path/to/the/folder -name '*.*' -mmin +59 -delete > /dev/null
deleting all the files older than 59 minutes while leaving the folders intact.
You could to this trick: create a file 1 hour ago, and use the -newer file argument.
(Or use touch -t to create such a file).
-mmin is for minutes.
Try looking at the man page.
man find
for more types.
For SunOS 5.10
Example 6 Selecting a File Using 24-hour Mode
The descriptions of -atime, -ctime, and -mtime use the ter-
minology n ``24-hour periods''. For example, a file accessed
at 23:59 is selected by:
example% find . -atime -1 -print
at 00:01 the next day (less than 24 hours later, not more
than one day ago). The midnight boundary between days has no
effect on the 24-hour calculation.
If you do not have "-mmin" in your version of "find", then "-mtime -0.041667" gets pretty close to "within the last hour", so in your case, use:
-mtime +(X * 0.041667)
so, if X means 6 hours, then:
find . -mtime +0.25 -ls
works because 24 hours * 0.25 = 6 hours
If one's find does not have -mmin and if one also is stuck with a find that accepts only integer values for -mtime, then all is not necessarily lost if one considers that "older than" is similar to "not newer than".
If we were able to create a file that that has an mtime of our cut-off time, we can ask find to locate the files that are "not newer than" our reference file.
To create a file that has the correct time stamp is a bit involved because a system that doesn't have an adequate find probably also has a less-than-capable date command that could do things like: date +%Y%m%d%H%M%S -d "6 hours ago".
Fortunately, other old tools can manage this, albeit in a more unwieldy way.
To begin finding a way to delete files that are over six hours old, we first have to find the time that is six hours ago. Consider that six hours is 21600 seconds:
$ date && perl -e '#d=localtime time()-21600; \
printf "%4d%02d%02d%02d%02d.%02d\n", $d[5]+1900,$d[4]+1,$d[3],$d[2],$d[1],$d[0]'
> Thu Apr 16 04:50:57 CDT 2020
202004152250.57
Since the perl statement produces the date/time information we need, use it to create a reference file that is exactly six hours old:
$ date && touch -t `perl -e '#d=localtime time()-21600; \
printf "%4d%02d%02d%02d%02d.%02d\n", \
$d[5]+1900,$d[4]+1,$d[3],$d[2],$d[1],$d[0]'` ref_file && ls -l ref_file
Thu Apr 16 04:53:54 CDT 2020
-rw-rw-rw- 1 root sys 0 Apr 15 22:53 ref_file
Now that we have a reference file exactly six hours old, the "old UNIX" solution for "delete all files older than six hours" becomes something along the lines of:
$ find . -type f ! -newer ref_file -a ! -name ref_file -exec rm -f "{}" \;
It might also be a good idea to clean up our reference file...
$ rm -f ref_file
Here is what one can do for going on the way #iconoclast was wondering about in their comment on another answer.
use crontab for user or an /etc/crontab to create file /tmp/hour:
# m h dom mon dow user command
0 * * * * root /usr/bin/touch /tmp/hour > /dev/null 2>&1
and then use this to run your command:
find /tmp/ -daystart -maxdepth 1 -not -newer /tmp/hour -type f -name "for_one_hour_files*" -exec do_something {} \;
find $PATH -name $log_prefix"*"$log_ext -mmin +$num_mins -exec rm -f {} \;