Need to get the files between two given dates via a KornShell (ksh) script. If there are multiple files on one day get the latest of the files for that day.
I haven't tried it out, but there's a mailing list post about finding files between two dates. The relevant part:
Touch 2 files, start_date and
stop_date, like this: $ touch -t
200603290000.00 start_date $ touch -t 200603290030.00 stop_date
Ok, start_date is 03/29/06 midnight,
stop_date is 03/29/06 30 minutes after
midnight. You might want to do a ls
-al to check.
On to find, you can find -newer and
then ! -newer, like this: $ find /dir
-newer start_date ! -newer stop_date -print
Combine that with ls -l, you get: $
find /dir -newer start_date ! -newer
stop_date -print0 | xargs -0 ls -l
(Or you can try -exec to execute ls
-l. I am not sure of the format, so you have to muck around a little bit)
in bash shell, just an example, you can use the -nt test operator (korn shell comes with it also, if i am not wrong).
printf "Enter start date( YYYYMMDD ):"
read startdate
printf "Enter end date( YYYYMMDD ):"
read enddate
touch -t "${startdate}0000.00" sdummy
touch -t "${enddate}0000.00" edummy
for fi in *
do
if [ $fi -nt "sdummy" -a ! $fi -nt "edummy" ] ;then
echo "-->" $fi
fi
done
In a nut shell for ksh :
!/usr/bin/ksh
# main from_date to_date path
# date format: YYMMDDhhmmss
ls -l --time-style "+%y%m%d%H%M%S" $3 | awk '{ print $6 " " $7 }' | while read t n
do
if (( t > $1 )) && (( t < $2 )); then
echo $t $n
fi
done
Related
I am using the following code but the final 'echo $dirname' is giving empty output on console
for folderpath in find /u01/app/SrcFiles/commercial/ngdw/* -name "IQ*";
do
folder_count=ls -d $folderpath/* | wc -l
echo -e "Total Date folder created : $folder_count in $folderpath \n"
if [ $folder_count -ne 0 ];
then
for dirpath in `find $folderpath/* -name "2*" `;
do
dirname=${dirpath##*/}
(( dirname <= 20210106 )) || continue
echo $dirname
done
fi
done
First I would calculate the date it was 3 months ago with the date command:
# with GNU date (for example on Linux)
mindate=$(date -d -3month +%Y%m%d)
# with BSD date (for example on macOS)
mindate=$(date -v -3m +%Y%m%d)
Then I would use a shell arithmetic comparison for determining the directories to remove:
# for dirpath in "$directory"/*
for dirpath in "$directory"/{20220310,20220304,20220210,20220203,20210403,20210405}
do
dirname=${dirpath##*/}
(( dirname <= mindate )) || continue
echo "$dirpath"
# rm -rf "$dirpath"
done
== doesn't do wildcard matching. You should do that in the for statement itself.
There's also no need to put * at the beginning of the wildcard, since the year is at the beginning of the directory name, not in the middle.
for i in "$directory"/202104*; do
if [ -d "$i" ]; then
echo "$i"
rm -rf "$i"
fi
done
The if statement serves two purposes:
If there are no matching directories, the wildcard expands to itself (unless you set the nullglob option), and you'll try to remove this nonexistent directory.
In case there are matching files rather than directories, they're skipped.
Suggesting to find which are the directories that created before 90 days ago or older, with find command.
find . -type d -ctime +90
If you want to find which directories created 90 --> 100 days ago.
find . -type d -ctime -100 -ctime +90
Once you have the correct folders list. Feed it to the rm command.
rm -rf $(find . -type d -ctime +90)
I have written a bash script which I want to use to monitor backups on a synology via pushgateway.
The script should search for subfolders in the backup folder, write the newest file into a variable and write the age and size of the file into an array.
To give it finally to the push gateway, I list all metrics with indexes. All folders or files are available. If I execute the script, often one or more indexes are not found. If I execute the commands manually one by one, I get a correct output.
Here is the script:
#!/bin/bash
set -e
backup_dir=$1
for dir in $(find "$backup_dir" -maxdepth 1 -mindepth 1 -type d \( ! -name #eaDir \)); do
if compgen -G "${dir}/*.vib" > /dev/null; then
latest_vib=$(ls -t1 "$dir"/*.vib | head -1)
age_vib=$(( ( $(date +%s) - $(stat -c %Y "$latest_vib") ) ))
size_vib=$(stat -c %s "$latest_vib")
arrage_vib+=("${age_vib}")
arrsize_vib+=("${size_vib}")
fi
if compgen -G "${dir}/*.vbk" > /dev/null; then
latest_vbk=$(ls -t1 "$dir"/*.vbk | head -1)
age_vbk=$(( ( $(date +%s) - $(stat -c %Y "$latest_vbk") ) ))
size_vbk=$(stat -c %s "$latest_vbk")
arrage_vbk+=("${age_vbk}")
arrsize_vbk+=("${size_vbk}")
fi
min_dir=$(echo "$dir" | cut -d'/' -f4- | tr '[:upper:]' '[:lower:]')
sign_dir=${min_dir//_/-}
arrdir+=("${sign_dir}")
done
echo "${arrdir[4]}"
echo "${arrage_vib[4]}"
cat << EOF | curl -ks -u user:pw --data-binary #- https://pushgateway/metrics/job/backup/instance/instance_name
# HELP backup_age displays the age of backups in seconds
# TYPE backup_age gauge
backup_age_vib{dir="${arrdir[1]}"} ${arrage_vib[1]}
backup_age_vib{dir="${arrdir[2]}"} ${arrage_vib[2]}
backup_age_vib{dir="${arrdir[3]}"} ${arrage_vib[3]}
backup_age_vib{dir="${arrdir[4]}"} ${arrage_vib[4]}
backup_age_vbk{dir="${arrdir[1]}"} ${arrage_vbk[1]}
...
# HELP backup_size displays the size of backups in bytes
# TYPE backup_size gauge
backup_size_vib{dir="${arrdir[1]}"} ${arrsize_vib[1]}
...
EOF
I hope you can help me and point out where I made a mistake. I am also open for general optimizations of the script, because I assume that it can be solved better and more performant or optimal. I have a few lines of code from here ;-).
Many thanks in advance.
I'm working on a script that takes a date in the format YYYMMDD as an input:
#!/bin/bash
check_dir () {
d1=$2
d2=$((d1+1))
f1=`mktemp`
f2=`mktemp`
touch -d $d1 $f1
touch -d $d2 $f2
n=$(find $1 -mindepth 1 \( -name "*$d1*" \) -o \( -newer $f1 ! -newer $f2 \) | wc -l)
if [ $n != $3 ]; then echo $1 "=" $n ; fi
rm -f $f1 $f2
}
In this script d2=$((d1+1)) calculates the next day's date from the date provide i.e., d1. But, what if the date is 20151231, then it will not be able to handle this situation. Could anyone please help me how to handle this exception?
gdate does not work on my platform!!
With the date version from GNU coreutils (i.e. used in Linux):
$ date -d '20151231 +1 day' +'%Y%m%d'
20160101
I'm trying to use the newer command as follows:
touch $HOME/mark.start -d "$d1"
touch $HOME/mark.end -d "$d2"
SF=$HOME/mark.start
EF=$HOME/mark.end
find . -newer $SF ! -newer $EF
But this gives me an output like this:
.
./File5
and counts it as 2 files, however that directory only has 1 file i.e., File5. Why is this happening and how to solve it?
UPDATE:
I'm actually trying to run the following script:
#!/bin/bash
check_dir () {
d1=$2
d2=$((d1+1))
f1=`mktemp`
f2=`mktemp`
touch -d $d1 $f1
touch -d $d2 $f2
n=$(find $1 \( -name "*$d1*" \) -o \( -newer $f1 ! -newer $f2 \) | wc -l)
if [ $n != $3 ]; then echo $1 "=" $n ; fi
rm -f $f1 $f2
}
That checks if the directory has file that either has a particular date in the format YYYMMDD or if its last modification time was last 1 day.
check_dir ./dir1 20151215 4
check_dir ./dir2 20151215 3
where in dir1 there should be 4 such files and if it is not true then it will print the actual number of files that is there.
So, when the directory only has file with dates in their name, then it checks them fine, but when it checks with newer, it always gives 1 file extra (which is not even there in the directory). Why is this happening???
The question asks why there's an extra . in the results from find, even when no file or directory by that name exists. The answer is simple: . always exists, even when it's hidden. Use ls -a to show hidden contents, and you'll see that it's present.
Your existing find command doesn't exempt the target directory itself -- . -- from being a legitimate result, which is why you're getting more results than you expect.
Add the following filter:
-mindepth 1 # only include content **under** the file or directory specified
...or, if you only want to count files, use...
-type f # only include regular files
Assuming GNU find, by the way, this all can be made far more efficient:
check_dir() {
local d1 d2 # otherwise these variables leak into global scope
d1=$2
d2=$(gdate -d "+ 1 day $d1" '+%Y%m%d') # assuming GNU date is installed as gdate
n=$(find "$1" -mindepth 1 \
-name "*${d1}*" -o \
'(' -newermt "$d1" '!' -newermt "$d2" ')' \
-printf '\n' | wc -l)
if (( n != $3 )); then
echo "$1 = $n"
fi
}
I receive files with names constructed in the following format
[2letters e.g.AF][6-digit-number sequence][Date in ccyymmdd][Time in hhmmss]
For Example:
AF00010720120917144500.csv
I want to automate loading such files onto my database using the date part of the file.
something which may start like this:
#!/bin/bash
filename_datepart=$(echo `date -d "1 day ago" +"%d%m%Y"`)
filename="/home/hlosi/AF000107"$filename_datepart".csv"
But remember, the part 000107 changes with each new file.
You can use wildcards to fill in the unknown values
#!/bin/bash
file=/home/hlosi/AF??????`date -d "1 day ago" +"%d%m%Y"`??????.csv
echo $file
Here is a BASH solution:
#!/bin/bash
#The full name
fullname="/home/hlosi/AF00010720120917144500.csv"
#Rip off the directory
file=$(basename "$fullname")
#Now pull out just the characters that we want
extract=$(echo "$file" | cut -c3-8)
echo "You want: $extract"
I think you want this. In case you have to handle multiple files.
#!/bin/bash
fpath=/home/hlosi/
filename_datepart=$(echo `date -d "1 day ago" +"%d%m%Y"`)
files=$(find $fpath -iname "*$filename_datepart.csv")
for file in $files
do
echo "found file: " $file
done
forgive me for my ignorance there is -atime -ctime -mtime I think its -ctime
find -ctime 1 -name \*.csv -print
-mtime match ending of csv and are exactly 1 day old, the trouble of this is it works in 24 hour period so files less than 24 hours but still yesterday would not show
this would be a simpler way of doing things since and would not care about changes in file name formatting for future proofing.
cd pathtcsv;
d=`date -d "1 day ago" +"%d"`
d=$d find . -type f -name \*.csv -ctime 1 -exec ls -l {} \;|awk '$7 ~ d'|awk '{print $NF}'| awk '{ print substr( $0, length($0) - 1, length($0) ) }'
# D = $d which is set as yesterday's date, it finds files from yseterday that have csv ending it then does an ls, pipes into awk and checks out value 7 against the date of yesterday which $7 is the date value on ls -l, it finally prints the last field and pipes into a final awk which prints the string and splits from char position to char position which is what you wanted ? you need to figure out what chars you need here is another example of above for char positions of 0 to 10.
d=$d find . -type f -ctime 1 -name \*.csv -exec ls -l {} \;|awk '$7 ~ d'|awk '{print $NF}'| awk '{ print substr( $0, 0, 10)}'