BASH ERROR: syntax error: operand expected (error token is ") - bash

I am new to bash scripting, and I'm having an issue with one of my scripts. I'm trying to compose a list of Drivers Under 25 after reading their birthdates in from a folder filled with XML files and calculating their ages. Once I have determined they are under 25, the filename of the driver's data is saved to a text file. The script is working up until a certain point and then it stops. The error I'm getting is:
gdate: extra operand ‘+%s’
Try 'gdate --help' for more information.
DriversUnder25.sh: line 24: ( 1471392000 - )/60/60/24 : syntax error: operand expected (error token is ")/60/60/24 ")
Here is my code:
#!/bin/bash
# define directory to search and current date
DIRECTORY="/*.xml"
CURRENT_DATE=$(date '+%Y%m%d')
# loop over files in a directory
for FILE in $DIRECTORY;
do
# grab user's birth date from XML file
BIRTH_DATE=$(sed -n '/Birthdate/{s/.*<Birthdate>//;s/<\/Birthdate.*//;p;}' $FILE)
# calculate the difference between the current date
# and the user's birth date (seconds)
DIFFERENCE=$(( ( $(gdate -ud $CURRENT_DATE +'%s') - $(gdate -ud $BIRTH_DATE +'%s') )/60/60/24 ))
# calculate the number of years between
# the current date and the user's birth date
YEARS=$(($DIFFERENCE / 365))
# if the user is under 25
if [ "$YEARS" -le 25 ]; then
# save file name only
FILENAME=`basename $FILE`
# output filename to text file
echo $FILENAME >> DriversUnder25.txt
fi
done
I'm not sure why it correctly outputs the first 10 filenames and then stops. Any ideas why this may be happening?

You need to quote the expansion of $BIRTH_DATE to prevent word splitting on the whitespace in the value. (It is good practice to quote all your parameter expansions, unless you have a good reason not to, for this very reason.)
DIFFERENCE=$(( ( $(gdate -ud "$CURRENT_DATE" +'%s') - $(gdate -ud "$BIRTH_DATE" +'%s') )/60/60/24 ))
(Based on your comment, this would probably at least allow gdate to give you a better error message.)

A best-practices implementation would look something like this:
directory=/ # patch as appropriate
current_date_unix=$(date +%s)
for file in "$directory"/*.xml; do
while IFS= read -r birth_date; do
birth_date_unix=$(gdate -ud "$birth_date" +'%s')
difference=$(( ( current_date_unix - birth_date_unix ) / 60 / 60 / 24 ))
years=$(( difference / 365 ))
if (( years < 25 )); then
echo "${file%.*}"
fi
done < <(xmlstarlet sel -t -m '//Birthdate' -v . -n <"$file")
done >DriversUnder25.txt
If this script needs to be usable my folks who don't have xmlstarlet installed, you can generate an XSLT template and then use xsltproc (which is available out-of-the-box on modern opertaing systems).
That is to say, if you run this once, and bundle its output with your script:
xmlstarlet sel -C -t -m '//Birthdate' -v . -n >get-birthdays.xslt
...then the script can be modified to replace xmlstarlet with:
xsltproc get-birthdays.xslt - <"$file"
Notes:
The XML input files are being read with an actual XML parser.
When expanding for file in "$directory"/*.xml, the expansion is quoted but the glob is not (thus allowing the script to operate on directories with spaces, glob characters, etc. in their names).
The output file is being opened once, for the loop, rather than once per line of output (reducing overhead unnecessarily opening and closing files).
Lower-case variable names are in use to comply with POSIX conventions (specifying that variables with meaning to the operating system and shell have all-upper-case names, and that the set of names with at least one lower-case character is reserved for application use; while the docs in question are with respect to environment variables, shell variables share a namespace, making the convention relevant).

The issue was that there were multiple drivers in some files, thus importing multiple birth dates into the same string. My solution is below:
#!/bin/bash
# define directory to search and current date
DIRECTORY="/*.xml"
CURRENT_DATE=$(date '+%Y%m%d')
# loop over files in a directory
for FILE in $DIRECTORY;
do
# set flag for output to false initially
FLAG=false
# grab user's birth date from XML file
BIRTH_DATE=$(sed -n '/Birthdate/{s/.*<Birthdate>//;s/<\/Birthdate.*//;p;}' $FILE)
# loop through birth dates in file (there can be multiple drivers)
for BIRTHDAY in $BIRTH_DATE;
do
# calculate the difference between the current date
# and the user's birth date (seconds)
DIFFERENCE=$(( ( $(gdate -ud $CURRENT_DATE +'%s') - $(gdate -ud $BIRTHDAY +'%s') )/60/60/24))
# calculate the number of years between
# the current date and the user's birth date
YEARS=$(($DIFFERENCE / 365))
# if the user is under 25
if [ "$YEARS" -le 25 ]; then
# save file name only
FILENAME=`basename $FILE`
# set flag to true (driver is under 25 years of age)
FLAG=true
fi
done
# if there is a driver under 25 in the file
if $FLAG == true; then
# output filename to text file
echo $FILENAME >> DriversUnder25.txt
fi
done

Related

Identifying the files older than x-months by the filename only and deleting them

I have 4 different files with different fileName.date formats, having a date embedded as part of the name. I want to identify the files older than 3 months based on their name only because the files would be edited/changed later as well. I want to create a shell script and run it as a cron.
Here below are the file under the same directory:
fileone.log.2018-03-23
file_two_2018-03-23.log
filethree.log.2018-03-23
file_four_file_four_2018-03-23.log
I have checked the existing example but have not found what I am actually looking for!
Working on the premise that you mean 90 days - if you need specifically months, we can check that too, but it's different logic.
here's some code you could work from -
(you said you don't want to work from a list, so I edited to use the current directory.)
$: cat chkDates
# while read f # replaced with -
for f in *[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]*
do # first get the epoch timestamp of the file based on the sate string embedded in the name
filedate=$(
date +%s -d $(
echo $f | sed -E 's/.*([0-9]{4}-[0-9]{2}-[0-9]{2}).*/\1/'
) # this returns the date substring
) # this converts it to an epoch integer of seconds since 1/1/70
# now see if it's > 90 days ( you said 3 months. if you need *months* we have to do some more...)
daysOld=$(( ( $(date +%s) - $filedate ) / 86400 )) # this should give you an integer result, btw
if (( 90 < $daysOld ))
then echo $f is old
else echo $f is not
fi
done # < listOfFileNames # not reading list now
You can pass date a date to report, and a format to present it.
sed pattern explanation
Note the sed -E 's/.*([0-9]{4}-[0-9]{2}-[0-9]{2}).*/\1/' command. This assumes the date format will be consistently YYYY-MM-DD, and does no validations of reasonableness. It will happily accept any 4 digits, then 2, then 2, delimited by dashes.
-E uses expanded regexes, so parens () can denote values to be remembered, without needing \'s. . means any character, and * means any number (including zero) of the previous pattern, so .* means zero or more characters, eating up all the line before the date. [0-9] means any digit. {x,y} sets a minimum(x) and maximum(y) number of consecutive matches - with only one value {4} means only exactly 4 of the previous pattern will do. So, '.*([0-9]{4}-[0-9]{2}-[0-9]{2}).*' means ignore as many characters as you can until seeing 4 digits, then a dash, 2 digits, then a dash, then 2 digits; remember that pattern (the ()'s), then ignore any characters behind it.
In a substitution, \1 means the first remembered match, so
sed -E 's/.*([0-9]{4}-[0-9]{2}-[0-9]{2}).*/\1/'
means find and remember the date pattern in the filenames, and replace the whole name with just that part in the output. This assumes the date will be present - on a filename where there is no date, the pattern will not match, and the whole filename will be returned, so be careful with that.
(hope that helped.)
By isolating the date string from the filenames with sed (your examples were format-consistent, so I used that) we pass it in and ask for the UNIX Epoch timestamp of that date string using date +%s -d $(...), to represent the file with a math-handy number.
Subtract that from the current date in the same format, you get the approximate age of the file in seconds. Divide that by the number of seconds in a day and you get days old. The file date will default to midnight, but the math will drop fractions, so it sorts out.
here's the file list I made, working from your examples
$: cat listOfFileNames
fileone.log.2018-03-23
fileone.log.2018-09-23
file_two_2018-03-23.log
file_two_2018-08-23.log
filethree.log.2018-03-23
filethree.log.2018-10-02
file_four_file_four_2018-03-23.log
file_four_file_four_2019-03-23.log
I added a file for each that would be within the 90 days as of this posting - including one that is "post-dated", which can easily happen with this sort of thing.
Here's the output.
$: ./chkDates
fileone.log.2018-03-23 is old
fileone.log.2018-09-23 is not
file_two_2018-03-23.log is old
file_two_2018-08-23.log is not
filethree.log.2018-03-23 is old
filethree.log.2018-10-02 is not
file_four_file_four_2018-03-23.log is old
file_four_file_four_2019-03-23.log is not
That what you had in mind?
An alternate pure-bash way to get just the date string
(You still need date to convert to the epoch seconds...)
instead of
filedate=$(
date +%s -d $(
echo $f | sed -E 's/.*([0-9]{4}-[0-9]{2}-[0-9]{2}).*/\1/'
) # this returns the date substring
) # this converts it to an epoch integer of seconds since 1/1/70
which doesn't seem to be working for you, try this:
tmp=${f%[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]*} # unwanted prefix
d=${f#$tmp} # prefix removed
tmp=${f#*[0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9]} # unwanted suffix
filedate=${d%$tmp} # suffix removed
filedate=$( date +%s --date=$filedate ) # epoch time
This is hard to read, but doesn't have to spawn as many subprocesses to get the work done. :)
If that doesn't work, then I'm suspicious of your version of date. Mine:
$: date --version
date (GNU coreutils) 8.26
UPDATE:
Simple Version:
Method for using the date inside of the file's name :
typeset stamp=$(date --date="90 day ago" +%s)
for file in /directory/*.log; do
fdate="$(echo "$file" | sed 's/[^0-9-]*//g')"
fstamp=$(date -d "${fdate} 00:00:00" +"%s")
if [ ${fstamp} -le ${stamp} ] ; then
echo "${file} : ${fdate} (${fstamp})"
fi
done
A More Complete Version:
This version will look at all files, if it fails to make a date value from the file it moves on.
typeset stamp=$(date --date="90 day ago" +%s)
for file in /tmp/* ; do
fdate="$(echo "$file" | sed 's/[^0-9-]*//g')"
fstamp=$(date -d "${fdate} 00:00:00" +"%s" 2> /dev/null)
[[ $? -ne 0 ]] && continue
if [ ${fstamp} -le ${stamp} ] ; then
echo "${file} : ${fdate} (${fstamp})"
fi
done
output:
/tmp/file_2016-05-23.log : 2016-05-23 (1463976000)
/tmp/file_2017-05-23.log : 2017-05-23 (1495512000)
/tmp/file_2018-05-23.log : 2018-05-23 (1527048000)
/tmp/file_2018-06-23.log : 2018-06-23 (1529726400)
/tmp/file_2018-07-23.log : 2018-07-23 (1532318400)
in this example the following were ignored :
/tmp/file_2018-08-23.log : 2018-08-23 (1534996800)
/tmp/file_2018-10-18.log : 2018-10-18 (1539835200)

Is it really slow to handle text file(more than 10K lines) with shell script?

I have a file with more than 10K lines of record.
Within each line, there are two date+time info. Below is an example:
"aaa bbb ccc 170915 200801 12;ddd e f; g; hh; 171020 122030 10; ii jj kk;"
I want to filter out the lines the days between these two dates is less than 30 days.
Below is my source code:
#!/bin/bash
filename="$1"
echo $filename
touch filterfile
totalline=`wc -l $filename | awk '{print $1}'`
i=0
j=0
echo $totalline lines
while read -r line
do
i=$[i+1]
if [ $i -gt $[j+9] ]; then
j=$i
echo $i
fi
shortline=`echo $line | sed 's/.*\([0-9]\{6\}\)[ ][0-9]\{6\}.*\([0-9]\{6\}\)[ ][0-9]\{6\}.*/\1 \2/'`
date1=`echo $shortline | awk '{print $1}'`
date2=`echo $shortline | awk '{print $2}'`
if [ $date1 -gt 700000 ]
then
continue
fi
d1=`date -d $date1 +%s`
d2=`date -d $date2 +%s`
diffday=$[(d2-d1)/(24*3600)]
#diffdays=`date -d $date2 +%s` - `date -d $date1 +%s`)/(24*3600)
if [ $diffday -lt 30 ]
then
echo $line >> filterfile
fi
done < "$filename"
I am running it in cywin. It took about 10 second to handle 10 lines. I use echo $i to show the progress.
Is it because i am using some wrong way in my script?
This answer does not answer your question but gives an alternative method to your shell script. The answer to your question is given by Sundeep's comment :
Why is using a shell loop to process text considered bad practice?
Furthermore, you should be aware that everytime you call sed, awk, echo, date, ... you are requesting the system to execute a binary which needs to be loaded into memory etc etc. So if you do this in a loop, it is very inefficient.
alternative solution
awk programs are commonly used to process log files containing timestamp information, indicating when a particular log record was written. gawk extended the awk standard with time-handling functions. The one you are interested in is :
mktime(datespec [, utc-flag ]) Turn datespec into a timestamp in the
same form as is returned by systime(). It is similar to the function
of the same name in ISO C. The argument, datespec, is a string of the
form "YYYY MM DD HH MM SS [DST]". The string consists of six or seven
numbers representing, respectively, the full year including century,
the month from 1 to 12, the day of the month from 1 to 31, the hour of
the day from 0 to 23, the minute from 0 to 59, the second from 0 to
60, and an optional daylight-savings flag.
The values of these numbers need not be within the ranges specified;
for example, an hour of -1 means 1 hour before midnight. The
origin-zero Gregorian calendar is assumed, with year 0 preceding year
1 and year -1 preceding year 0. If utc-flag is present and is either
nonzero or non-null, the time is assumed to be in the UTC time zone;
otherwise, the time is assumed to be in the local time zone. If the
DST daylight-savings flag is positive, the time is assumed to be
daylight savings time; if zero, the time is assumed to be standard
time; and if negative (the default), mktime() attempts to determine
whether daylight savings time is in effect for the specified time.
If datespec does not contain enough elements or if the resulting time
is out of range, mktime() returns -1.
As your date format is of the form yymmdd HHMMSS we need to write a parser function convertTime for this. Be aware in this function we will pass times of the form yymmddHHMMSS. Furthermore, using a space delimited fields, your times are located in field $4$5 and $11$12. As mktime converts the time to seconds since 1970-01-01 onwards, all we need to do is to check if the delta time is smaller than 30*24*3600 seconds.

awk 'function convertTime(t) {
s="20"substr(t,1,2)" "substr(t,3,2)" "substr(t,5,2)" "
s= s substr(t,7,2)" "substr(t,9,2)" "substr(t,11,2)"
return mktime(s)
}
{ t1=convertTime($4$5); t2=convertTime($11$12)}
(t2-t1 < 30*3600*24) { print }' <file>
If you are not interested in the real delta time (your sed line removes the actual time of the day), than you can adopt it to :
awk 'function convertTime(t) {
s="20"substr(t,1,2)" "substr(t,3,2)" "substr(t,5,2)" "
s= s "00 00 00"
return mktime(s)
}
{ t1=convertTime($4); t2=convertTime($11)}
(t2-t1 < 30*3600*24) { print }' <file>
If the dates are not in the fields, you can use match to find them :
awk 'function convertTime(t) {
s="20"substr(t,1,2)" "substr(t,3,2)" "substr(t,5,2)" "
s= s substr(t,7,2)" "substr(t,9,2)" "substr(t,11,2)"
return mktime(s)
}
{ match($0,/[0-9]{6} [0-9]{6}/);
t1=convertTime(substr($0,RSTART,RLENGTH));
a=substr($0,RSTART+RLENGTH)
match(a,/[0-9]{6} [0-9]{6}/)
t2=convertTime(substr(a,RSTART,RLENGTH))}
(t2-t1 < 30*3600*24) { print }' <file>
With some modifications, often without speed in mind, I can reduce the processing time by 50% - which is a lot:
#!/bin/bash
filename="$1"
echo "$filename"
# touch filterfile
totalline=$(wc -l < "$filename")
i=0
j=0
echo "$totalline" lines
while read -r line
do
i=$((i+1))
if (( i > ((j+9)) )); then
j=$i
echo $i
fi
shortline=($(echo "$line" | sed 's/.*\([0-9]\{6\}\)[ ][0-9]\{6\}.*\([0-9]\{6\}\)[ ][0-9]\{6\}.*/\1 \2/'))
date1=${shortline[0]}
date2=${shortline[1]}
if (( date1 > 700000 ))
then
continue
fi
d1=$(date -d "$date1" +%s)
d2=$(date -d "$date2" +%s)
diffday=$(((d2-d1)/(24*3600)))
# diffdays=$(date -d $date2 +%s) - $(date -d $date1 +%s))/(24*3600)
if (( diffday < 30 ))
then
echo "$line" >> filterfile
fi
done < "$filename"
Some remarks:
# touch filterfile
Well - the later CMD >> filterfile overwrites this file and creates one, if it doesn't exist.
totalline=$(wc -l < "$filename")
You don't need awk, here. The filename output is surpressed if wc doesn't see the filename.
Capturing the output in an array:
shortline=($(echo "$line" | sed 's/.*\([0-9]\{6\}\)[ ][0-9]\{6\}.*\([0-9]\{6\}\)[ ][0-9]\{6\}.*/\1 \2/'))
date1=${shortline[0]}
date2=${shortline[1]}
allows us array access and saves another call to awk.
On my machine, your code took about 42s for 2880 lines (on your machine 2880 s?) and about 19s for the same file with my code.
So I suspect, if you aren't running it on an i486-machine, that cygwin might be a slowdown. It's a linux environment for windows, isn't it? Well, I'm on a core Linux system. Maybe you try the gnu-utils for Windows - the last time I looked for them, they were advertised as gnu-utils x32 or something, maybe there is an a64-version available by now.
And the next thing I would have a look at, is the date calculation - that might be a slowdown too.
2880 lines isn't that much, so I don't suspect that my SDD drive plays a huge role in the game.

Create a set of files based off a date range

How would you go about doing a few lines of code in Bash to accomplish the following. I'm trying to build up my skills in Bash and learn how to handle more small tasks directory from the command line.
Steps:
Specify a start date and an end date. Load all the dates in between including the start and end date into a "list"
Loop over the list creating a file like this each time.
(requires date formatting)
2017-11-10.w
2017-11-11.w
2017-11-12.w
You could convert the input dates to Unix timestamps, then add the number of seconds per day and touch a file named after the result until you are past the end date:
#!/bin/bash
startstamp=$(date -d "$1" +'%s')
endstamp=$(date -d "$2" +'%s')
secs_per_day=$(( 24 * 3600 ))
for (( thedate = startstamp; thedate <= endstamp; thedate += secs_per_day )); do
touch "$(date -d "#$thedate" '+%F.w')"
done
The %s formatting string (a GNU extension) prints the number of seconds since the Unix epoch, and # in the argument to the -d option indicates that the date is in that format. %F is short for %Y-%m-%d, which translates to YYYY-MM-DD.
Example usage:
$ ./dates 2017-11-10 2017-11-15
$ ls -1
2017-11-10.w
2017-11-11.w
2017-11-12.w
2017-11-13.w
2017-11-14.w
2017-11-15.w
dates

Custom date verification - BASH - using just grep or awk etc

I saw couple of posts (some depends upon date -d $xyz to verify) but I'm trying to create an until loop where the user should be re-prompted to enter the value of a date format until it matches the custom date format.
My date format (what I need for Splunk) is m/d/yyyy:h:m:s or mm/dd/yyyy:hh:mm:ss
which means, if m (month number) is a single digit lets say 1 for January, then both 1 or 01 values are possible for date format but 0 or 00 is NOT a valid value. Value range is 01-to->12 or 1-to->12 but not greater than 12.
Similarly, the same rule applies to d (day number), it can be 01-to->10-to->31 or 1-to->31 but not 00 or more than 31 and all other yyyy (year), h (hour), m (minute), s (second) part.
What could be a minimal code (obfuscated is fine) to do this verification in BASH? It seems like date -d ??? doesn't provides this custom kind of verification for date/times!
OK, I can write one verifyDateFormatfunc() to do this, but I know there are people who have already written a one-liner / minimal snippet to verify this for sure. grep -f .. (where bunch of regex are listed line by line for all possible combinations, again the main code will look very minimal if I follow this? as the patterns sitting in -f file for grep will be transparent to a user) -or creating a map funcation (based on delimiters) for value ranges?
Possible values:
1/03/2017:23:0:15
02/4/2017:0:1:2
09/05/2017:10:10:0
10/6/2017:12:14:16
Here's an unholy extended regular expression (POSIX ERE):
^([1-9]|1[0-2]|0[1-9])/([1-9]|0[1-9]|[12][0-9]|3[01])/[0-9]{4}:([0-9]|0[0-9]|1[0-9]|2[0-3]):([0-9]|[0-5][0-9]):([0-9]|[0-5][0-9])$
that will test for the date/time patterns you specified (m/d/yyyy:h:m:s and mm/dd/yyyy:hh:mm:ss), with:
month: 1-12, 01-12
day: 1-31, 01-31
year: 0000-9999
hour: 0-23, 00-23
minute: 0-59, 00-59
second: 0-59, 00-59
You can use in an awk program that will exit with success (exit code 0) if the (first) line is a valid date/time (wrapped in a shell function that tests the first argument, for convenience):
#!/bin/bash
is_datetime_valid() {
awk '{exit $0!~"^([1-9]|1[0-2]|0[1-9])/([1-9]|0[1-9]|[12][0-9]|3[01])/[0-9]{4}:([0-9]|0[0-9]|1[0-9]|2[0-3]):([0-9]|[0-5][0-9]):([0-9]|[0-5][0-9])$"}' <<<"$1"
}
Or, if you prefer a pure bash solution (with ERE support in bash v3.0+):
#!/bin/bash
is_datetime_valid() {
local pat='^([1-9]|1[0-2]|0[1-9])/([1-9]|0[1-9]|[12][0-9]|3[01])/[0-9]{4}:([0-9]|0[0-9]|1[0-9]|2[0-3]):([0-9]|[0-5][0-9]):([0-9]|[0-5][0-9])$'
[[ $1 =~ $pat ]]
}
You can use it like:
if is_datetime_valid "1/03/2017:23:0:15"; then
# yup, it's valid
else
# ney, it's invalid
fi
Tested on a few examples:
#!/bin/bash
samples=(
"1/03/2017:23:0:15" "02/4/2017:0:1:2" "09/05/2017:10:10:0" "10/6/2017:12:14:16"
"00/03/2017:23:0:15" "1/33/2017:23:0:15"
)
for dt in "${samples[#]}"; do
if is_datetime_valid "$dt"; then
echo "$dt is valid"
else
echo "$dt is invalid"
fi
done
Gives:
1/03/2017:23:0:15 is valid
02/4/2017:0:1:2 is valid
09/05/2017:10:10:0 is valid
10/6/2017:12:14:16 is valid
00/03/2017:23:0:15 is invalid
1/33/2017:23:0:15 is invalid
I do not know whether using BSD date is an option for you, but it has what you are looking for.
There the date checker function can look like this
is_datetime_valid() {
date -j -f "%m/%d/%Y:%T" $1 1> /dev/null 2>&1
return $?
}

how do you through loop a certain date range

I have files in a directory that are date based but not obviously date-stamped.
File_yyyymmdd_record.log
These are lying around in a directory for a few years worth of time.
Now if these were simply numbers all I needed to do was get the difference and incremenet a counter to push the value
var=substring( File_yyyymmdd_record.log ) /* get the yyyymmdd part */
var2=substring( File2_yyyymmdd_record.log ) /* get the yyyymmdd part */
delta=var2-var1
set i=delta and loop through to get the values for all these recordID's ( record ID is the yyyymmdd part )
The problem is if I have 2 different months and also years in the directory say 20131210 and 20140110
the difference not going to gimme all the recordID's in that directory , since, when it spills over to the next month the plain numeric calculation is not applicable- it should be a date based calculation.
what I want to do is use 2 input parameters to the shell
shell.sh recordID1 recordID2
and based on these it will find all records and store them some place and loop through each record as an input like this
find <dir> -iname recordID* ...<some awk and sed here> |
while read recordID ;
do <stuff >
done
Anyway this can be achieved esp in 2 contexts-
First the date calculation part and the other is to store these recordID's so I can cycle through them. Maybe echo them to a tmp file is what comes off the bat.
For the date calculation part - I tried this and it works . But not sure if it will falter some time / situation
echo $((($(date -u -d 2010-04-29 +%s) - $(date -u -d 2010-03-28 +%s)) / 86400))
So given recordID1 as 20100328 I have 32 days recordID's to look for in that directory.
You have to advance dates for 32 days from recordID1 and store them some place.
How best can all this be done.
I got your points, you need find out log files with file name between 20131210 and 20140110 .
(no need convert to epoch time)
#! /usr/bin/bash
sep=20131210
eep=20140110
find /DIR -type f -name "*.log" |while read file
do
d=${file:5:8}
if [ "$d" -ge "$sep" ] && [ "$d" -le "$eep" ]; then
do <stuff >
fi
done
Something like this should do:
s=20130102 # start date
e=20130202 # end date
sep=$(date +"%s" -d"$s") # conv to epoch
eep=$(date +"%s" -d"$e")
for f in *.log; do
d=$(date +"%s" -d$(sed -n 's/^[^_]*_\([^_]*\)_[^_]*.log/\1/p' <<< "$f"))
if [ "$d" -ge "$sep" ] && [ "$d" -le "$eep" ]; then
echo $f
fi
done

Resources