Select directories with date in their names from a date range - bash

I'm creating a list of directories with the requested date range in their name.
Directories are all labeled other_2019-07-18T00-00-00. The T is messing me up!
Copied this loop from somewhere.
#!/bin/bash
curdate=$(date +%Y-%m-%d%H-%M-%S)
#
for o in other_*; do
tmp=${o##other_}
tmp=$(echo "$tmp" | sed 's/T//') # clean up prefixes
fdate=$(date -d "${tmp}")
(( curdate < fdate )) && echo "$o"
done
I expect the final echo to include the path of all dir that match.

Unlike AWK, Bash comparison operator < works only on numerical values.
Please try instead:
#!/bin/bash
curdate=$(date +%Y%m%d%H%M%S)
for o in other_*; do
tmp=${o##other_}
fdate=$(echo "$tmp" | sed 's/[-T]//g') # numeralization
(( curdate < fdate )) && echo "$o"
done
As an alternative, you can compare epoch times:
#!/bin/bash
curdate=$(date +%s)
for o in other_*; do
tmp=${o##other_}
tmp=$(echo "$tmp" | sed 's/T/ /' | sed 's/\([0-9][0-9]\)-\([0-9][0-9]\)-\([0-9][0-9]\)$/\1:\2:\3/')
fdate=$(date -d "$tmp" +%s)
(( curdate < fdate )) && echo "$o"
done
Hope this helps.

Instead of dropping T...
date -d 2019-03-23T00-06-28
date: invalid date '2019-03-23T00-06-28'
ok, but:
date -d 2019-03-23T00:06:28
Sat Mar 23 00:06:28 UTC 2019
So we have to replace last two dashes by ::
As your question is tagged bash:
file="somepath/other_2019-07-18T00-00-00.extension"
time=${file#*other_} # suppress from left until 'other_'
time=${time%.*} # suppress extension
time=${time//-/:} # replace all dash by a `:`
time=${time/:/-} # replace 1nd `:` by a dash
time=${time/:/-} # replace 1nd `:` by a dash (again)
date -d $time
Thu Jul 18 00:00:00 UTC 2019
This could by written:
printf -v now "%(%s)T" -1 # bashism for current date to variable $now
for file in somepath/other_*.ext ;do
time=${file#*other_} time=${time%.*} time=${time//-/:}
time=${time/:/-} time=${time/:/-}
read fdate < <(date +%s -d $time)
((fdate > now)) && { echo $file: $((fdate - now)) ; }
done
Reducing forks (to date) improve quickness:
for matching your sample, you could replace for file in somepath/other_*.ext ;do by for file in other_*; do. This must work quite same.
fifo=/tmp/fifo-date-$$
mkfifo $fifo
exec 5> >(exec stdbuf -o0 date -f - +%s >$fifo 2>&1)
echo now 1>&5
exec 6< $fifo
read -t 1 -u 6 now
rm $fifo
for file in otherdates/*.ext ; do
time=${file#*other_} time=${time%.*} time=${time//-/:}
time=${time/:/-} time=${time/:/-}
echo $time 1>&5 && read -t 1 -u 6 fdate
((fdate > now)) && {
echo $file: $((fdate - now))
}
done
exec 6>&-
exec 5>&-
In this, I run date +%s in background, with -f argument, date will interpret each incoming line, then answer UNIX_TIME. So $now is first read from date process, by:
echo now >&5 && # now, the string
read -u 6 now # read will populate `$now` variable
Nota, once fifo opened by both input and output, they could be deleted. It will remain for process until process close them.

there is no space between day and hour, which causes date not to be able to read the date. Try:
sed 's/T/ /'

Related

Shell script - is there a faster way to write date/time per second between start and end time?

I have this script (which works fine) that will write all the date/time per second, from a start date/time till an end date/time to a file
while read line; do
FIRST_TIMESTAMP="20230109-05:00:01" #this is normally a variable that changes with each $line
LAST_TIMESTAMP="20230112-07:00:00" #this is normally a variable that changes with each $line
date=$FIRST_TIMESTAMP
while [[ $date < $LAST_TIMESTAMP || $date == $LAST_TIMESTAMP ]]; do
date2=$(echo $date |sed 's/ /-/g' |sed "s/^/'/g" |sed "s/$/', /g")
echo "$date2" >> "OUTPUTFOLDER/output_LABELS_$line"
date=$(date -d "$date +1 sec" +"%Y%m%d %H:%M:%S")
done
done < external_file
However this sometimes needs to run 10 times, and the start date/time and end date/time sometimes lies days apart.
Which makes the script take a long time to write all that data.
Now I am wondering if there is a faster way to do this.
Avoid using a separate date call for each date. In the next example I added a safety parameter maxloop, avoiding loosing resources when the dates are wrong.
#!/bin/bash
awkdates() {
maxloop=1000000
awk \
-v startdate="${first_timestamp:0:4} ${first_timestamp:4:2} ${first_timestamp:6:2} ${first_timestamp:9:2} ${first_timestamp:12:2} ${first_timestamp:15:2}" \
-v enddate="${last_timestamp:0:4} ${last_timestamp:4:2} ${last_timestamp:6:2} ${last_timestamp:9:2} ${last_timestamp:12:2} ${last_timestamp:15:2}" \
-v maxloop="${maxloop}" \
'BEGIN {
T1=mktime(startdate);
T2=mktime(enddate);
linenr=1;
while (T1 <= T2) {
printf("%s\n", strftime("%Y%m%d %H:%M:%S",T1));
T1+=1;
if (linenr++ > maxloop) break;
}
}'
}
mkdir -p OUTPUTFOLDER
while IFS= read -r line; do
first_timestamp="20230109-05:00:01" #this is normally a variable that changes with each $line
last_timestamp="20230112-07:00:00" #this is normally a variable that changes with each $line
awkdates >> "OUTPUTFOLDER/output_LABELS_$line"
done < <(printf "%s\n" "line1" "line2")
Using epoch time (+%s and #) with GNU date and GNU seq to
produce datetimes in ISO 8601 date format:
begin=$(date -ud '2023-01-12T00:00:00' +%s)
end=$(date -ud '2023-01-12T00:00:12' +%s)
seq -f "#%.0f" "$begin" 1 "$end" |
date -uf - -Isec
2023-01-12T00:00:00+00:00
2023-01-12T00:00:01+00:00
2023-01-12T00:00:02+00:00
2023-01-12T00:00:03+00:00
2023-01-12T00:00:04+00:00
2023-01-12T00:00:05+00:00
2023-01-12T00:00:06+00:00
2023-01-12T00:00:07+00:00
2023-01-12T00:00:08+00:00
2023-01-12T00:00:09+00:00
2023-01-12T00:00:10+00:00
2023-01-12T00:00:11+00:00
2023-01-12T00:00:12+00:00
if you're using macOS/BSD's date utility instead of the gnu one, the equivalent command to parse would be :
(bsd)date -uj -f '%FT%T' '2023-01-12T23:34:45' +%s
1673566485
...and the reverse process is using -r flag instead of -d, sans "#" prefix :
(bsd)date -uj -r '1673566485' -Iseconds
2023-01-12T23:34:45+00:00
(gnu)date -u -d '#1673566485' -Iseconds
2023-01-12T23:34:45+00:00

Mac OSX Shell script parse ISO 8601 date and add one second?

I am trying to figure out how to parse a file with ISO 8601-formatted time stamps, add one second and then output them to a file.
All the examples I have found don't really tell me how to do it with ISO 8601 date/time strings.
Example:
read a csv of times like: "2017-02-15T18:47:59" (some are correct, others are not)
and spit out in a new file "2017-02-15T18:48:00"
mainly just trying to correct a bunch of dates that have 59 seconds at the end to round up to the 1 second mark.
This is my current progress:
#!/bin/bash
while IFS='' read -r line || [[ -n "$line" ]]; do
# startd=$(date -j -f '%Y%m%d' "$line" +'%Y%m%d');
# echo "$startd";
startd=$(date -j -u -f "%a %b %d %T %Z %Y" $line)
#startd=$(date -j -f '%Y%m%d' "$line" +'%Y%m%d');
echo "$startd";
done < "$1"
Any help would be appreciated
jm666's helpful perl answer will be much faster than your shell loop-based approach.
That said, if you want to make your bash code work on macOS, with its BSD date implementation, here's a solution:
# Define the input date format, which is also used for output.
fmt='%Y-%m-%dT%H:%M:%S'
# Note: -j in all date calls below is needed to suppress setting the
# system date.
while IFS= read -r line || [[ -n "$line" ]]; do
# Parse the line at hand using input format specifier (-f) $fmt,
# and output-format specifier (+) '%s', which outputs a Unix epoch
# timestamp (in seconds).
ts=$(date -j -f "$fmt" "$line" +%s)
# See if the seconds-component (%S) is 59...
if [[ $(date -j -f %s "$ts" +%S) == '59' ]]; then
# ... and, if so, add 1 second (-v +1S).
line=$(date -j -f %s -v +1S "$ts" +"$fmt")
fi
# Output the possibly adjusted timestamp.
echo "$line"
done < "$1"
Note that input dates such as 2017-02-15T18:47:59 are interpreted as local time, because they contain no time-zone information.
This could do the job
perl -MTime::Piece -nlE '$f=q{%Y-%m-%dT%H:%M:%S};$t=Time::Piece->strptime($_,$f)+1;say $t->strftime($f)' < dates.txt
if the dates.txt contains
2017-02-15T18:47:59
2016-02-29T23:59:59
2017-02-28T23:59:59
2015-12-31T23:59:59
2010-10-10T10:10:10
the above produces
2017-02-15T18:48:00
2016-03-01T00:00:00
2017-03-01T00:00:00
2016-01-01T00:00:00
2010-10-10T10:10:11

Implementing a datalogger in bash

Hi I'm a newby in Bash scripting.
I need to log a data stream from a specific IP address and generate a logfile for each day as "file-$date.log" (i.e at 00:00:00 UT close the previous day file and create the correspondig to the new one)
I need to show data stream on screen while it is logged in a file
I try this solution but not works well because never closesthe initial file
apparently the condition check never executes while the first command of the pipe it is something different to an constant string like echo "something".
#!/bin/bash
log_data(){
while IFS= read -r line ; do printf '%s %s\n' "$(date -u '+%j %Y-%m-%d %H:%M:%S')" "$line"; done
}
register_data() {
while : ;
do
> stream.txt
DATE=$(date -u "+%j %Y-%m-%d %H:%M")
HOUR=$(date -u "+%H:%M:%S")
file="file-$DATE.log"
while [[ "${HOUR}" != 00:00:00 ]];
do
tail -f stream.txt | tee "${file}"
sleep 1
HOUR=$(date -u "+%H:%M:%S")
done
> stream.txt
done
}
nc -vn $IP $IP_port | log_data >> stream.txt &
register_data
I'll will be glad if someone can give me some clues to solve this problem.

Parsing date and time format - Bash

I have date and time format like this(yearmonthday):
20141105 11:30:00
I need assignment year, month, day, hour and minute values to variable.
I can do it year, day and hour like this:
year=$(awk '{print $1}' log.log | sed 's/^\(....\).*/\1/')
day=$(awk '{print $1}' log.log | sed 's/^.*\(..\).*/\1/')
hour=$(awk '{print $2}' log.log | sed 's/^\(..\).*/\1/')
How can I do this for month and minute?
--
And I need that every line of my log file:
20141105 11:30:00 /bla/text.1
20141105 11:35:00 /bla/text.2
20141105 11:40:00 /bla/text.3
....
I'm trying read line by line this log file and do this:
mkdir -p "/bla/backup/$year/$month/$day/$hour/$minute"
mv $file "/bla/backup/$year/$month/$day/$hour/$minute"
Here is my not working code:
#!/bin/bash
LOG=/var/log/LOG
while read line
do
year=${line:0:4}
month=${line:4:2}
day=${line:6:2}
hour=${line:9:2}
minute=${line:12:2}
file=$(awk '{print $3}')
if [ -f "$file" ]; then
printf -v path "%s/%s/%s/%s/%s" $year $month $day $hour $minute
mkdir -p "/bla/backup/$path"
mv $file "/bla/backup/$path"
fi
done < $LOG
You don't need to call out to awk to date at all, use bash's substring operations
d="20141105 11:30:00"
yr=${d:0:4}
mo=${d:4:2}
dy=${d:6:2}
hr=${d:9:2}
mi=${d:12:2}
printf -v dir "/bla/%s/%s/%s/%s/%s/\n" $yr $mo $dy $hr $mi
echo "$dir"
/bla/2014/11/05/11/30/
Or directly, without all the variables.
printf -v dir "/bla/%s/%s/%s/%s/%s/\n" ${d:0:4} ${d:4:2} ${d:6:2} ${d:9:2} ${d:12:2}
Given your log file:
while read -r date time file; do
d="$date $time"
printf -v dir "/bla/%s/%s/%s/%s/%s/\n" ${d:0:4} ${d:4:2} ${d:6:2} ${d:9:2} ${d:12:2}
mkdir -p "$dir"
mv "$file" "$dir"
done < filename
or, making a big assumption that there are no whitespace or globbing characters in your filenames:
sed -r 's#(....)(..)(..) (..):(..):.. (.*)#mv \6 /blah/\1/\2/\3/\4/\5#' | sh
date command also do this work
#!/bin/bash
year=$(date +'%Y' -d'20141105 11:30:00')
day=$(date +'%d' -d'20141105 11:30:00')
month=$(date +'%m' -d'20141105 11:30:00')
minutes=$(date +'%M' -d'20141105 11:30:00')
echo "$year---$day---$month---$minutes"
You can use only one awk
month=$(awk '{print substr($1,5,2)}' log.log)
year=$(awk '{print substr($1,0,4)}' log.log)
minute=$(awk '{print substr($2,4,2)}' log.log)
etc
I guess you are processing the log file, which each line starts with the date string. You may have already written a loop to handle each line, in your loop, you could do:
d="$(awk '{print $1,$2}' <<<"$line")"
year=$(date -d"$d" +%Y)
month=$(date -d"$d" +%m)
day=$(date -d"$d" +%d)
min=$(date -d"$d" +%M)
Don't repeat yourself.
d='20141105 11:30:00'
IFS=' ' read -r year month day min < <(date -d"$d" '+%Y %d %m %M')
echo "year: $year"
echo "month: $month"
echo "day: $day"
echo "min: $min"
The trick is to ask date to output the fields you want, separated by a character (here a space), to put this character in IFS and ask read to do the splitting for you. Like so, you're only executing date once and only spawn one subshell.
If the date comes from the first line of the file log.log, here's how you can assign it to the variable d:
IFS= read -r d < log.log
eval "$(
echo '20141105 11:30:00' \
| sed 'G;s/\(....\)\(..\)\(..\) \(..\):\(..\):\(..\) *\(.\)/Year=\1\7Month=\2\7Day=\3\7Hour=\4\7Min=\5\7Sec=\6/'
)"
pass via a assignation string to evaluate. You could easily adapt to also check the content by replacing dot per more specific pattern like [0-5][0-9] for min and sec, ...
posix version so --posix on GNU sed
I wrote a function that I usually cut and paste into my script files
function getdate()
{
local a
a=(`date "+%Y %m %d %H %M %S" | sed -e 's/ / /'`)
year=${a[0]}
month=${a[1]}
day=${a[2]}
hour=${a[3]}
minute=${a[4]}
sec=${a[5]}
}
in the script file, on a line of it's own
getdate
echo "year=$year,month=$month,day=$day,hour=$hour,minute=$minute,second=$sec"
Of course, you can modify what I provided or use answer [6] above.
The function takes no arguments.

How to find the first occurence of date which is greater than or eqaul to particular date in text file using shell script

past_date='2013-11-14'
initial_time=$(grep -o -m1 "$past_date [0-9][0-9]:[0-9][0-9]:[0-9][0-9]" logfile.txt)
/* Here I am trying to find the first occurence of date which is greater than or eqaul to '2013-11-14', Above code I have tried ,It is giving only that particular line of file, If that date is not found It has to give next date which is greater than 2013-11-14 date */
Using awk
past_date='20131114'
awk '{d=$1;gsub(/-/,"",d);if (d>=p) {print;exit}}' p=$past_date logfile
2013-11-15 15:45:40 Starting agent install process
If you use bash, then you might want to try something like:
past_date='2013-11-14'
initial_time=$(grep -oP '\d{4}-\d\d-\d\d \d\d:\d\d:\d\d' < logfile.txt | \
while read LINE ; do if [ "$LINE" '>' "$past_date" ]; then echo $LINE; break; fi ; done)
while read line
do
initial_time=`echo $line | sed -e 's/\([0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9]\).*/\1/'`
file_content_date=`date -d "$initial_time" +%Y%m%d`
comparison_past_date=`date -d "$past_date" +%Y%m%d`
if [ $comparison_past_date -le $file_content_date ]; then
comparison_start_date=`date -d "$file_content_date" +%Y%m%d`
break
fi
done < logfile.txt
fi

Resources