Parsing date and time format - Bash - bash

I have date and time format like this(yearmonthday):
20141105 11:30:00
I need assignment year, month, day, hour and minute values to variable.
I can do it year, day and hour like this:
year=$(awk '{print $1}' log.log | sed 's/^\(....\).*/\1/')
day=$(awk '{print $1}' log.log | sed 's/^.*\(..\).*/\1/')
hour=$(awk '{print $2}' log.log | sed 's/^\(..\).*/\1/')
How can I do this for month and minute?
--
And I need that every line of my log file:
20141105 11:30:00 /bla/text.1
20141105 11:35:00 /bla/text.2
20141105 11:40:00 /bla/text.3
....
I'm trying read line by line this log file and do this:
mkdir -p "/bla/backup/$year/$month/$day/$hour/$minute"
mv $file "/bla/backup/$year/$month/$day/$hour/$minute"
Here is my not working code:
#!/bin/bash
LOG=/var/log/LOG
while read line
do
year=${line:0:4}
month=${line:4:2}
day=${line:6:2}
hour=${line:9:2}
minute=${line:12:2}
file=$(awk '{print $3}')
if [ -f "$file" ]; then
printf -v path "%s/%s/%s/%s/%s" $year $month $day $hour $minute
mkdir -p "/bla/backup/$path"
mv $file "/bla/backup/$path"
fi
done < $LOG

You don't need to call out to awk to date at all, use bash's substring operations
d="20141105 11:30:00"
yr=${d:0:4}
mo=${d:4:2}
dy=${d:6:2}
hr=${d:9:2}
mi=${d:12:2}
printf -v dir "/bla/%s/%s/%s/%s/%s/\n" $yr $mo $dy $hr $mi
echo "$dir"
/bla/2014/11/05/11/30/
Or directly, without all the variables.
printf -v dir "/bla/%s/%s/%s/%s/%s/\n" ${d:0:4} ${d:4:2} ${d:6:2} ${d:9:2} ${d:12:2}
Given your log file:
while read -r date time file; do
d="$date $time"
printf -v dir "/bla/%s/%s/%s/%s/%s/\n" ${d:0:4} ${d:4:2} ${d:6:2} ${d:9:2} ${d:12:2}
mkdir -p "$dir"
mv "$file" "$dir"
done < filename
or, making a big assumption that there are no whitespace or globbing characters in your filenames:
sed -r 's#(....)(..)(..) (..):(..):.. (.*)#mv \6 /blah/\1/\2/\3/\4/\5#' | sh

date command also do this work
#!/bin/bash
year=$(date +'%Y' -d'20141105 11:30:00')
day=$(date +'%d' -d'20141105 11:30:00')
month=$(date +'%m' -d'20141105 11:30:00')
minutes=$(date +'%M' -d'20141105 11:30:00')
echo "$year---$day---$month---$minutes"

You can use only one awk
month=$(awk '{print substr($1,5,2)}' log.log)
year=$(awk '{print substr($1,0,4)}' log.log)
minute=$(awk '{print substr($2,4,2)}' log.log)
etc

I guess you are processing the log file, which each line starts with the date string. You may have already written a loop to handle each line, in your loop, you could do:
d="$(awk '{print $1,$2}' <<<"$line")"
year=$(date -d"$d" +%Y)
month=$(date -d"$d" +%m)
day=$(date -d"$d" +%d)
min=$(date -d"$d" +%M)

Don't repeat yourself.
d='20141105 11:30:00'
IFS=' ' read -r year month day min < <(date -d"$d" '+%Y %d %m %M')
echo "year: $year"
echo "month: $month"
echo "day: $day"
echo "min: $min"
The trick is to ask date to output the fields you want, separated by a character (here a space), to put this character in IFS and ask read to do the splitting for you. Like so, you're only executing date once and only spawn one subshell.
If the date comes from the first line of the file log.log, here's how you can assign it to the variable d:
IFS= read -r d < log.log

eval "$(
echo '20141105 11:30:00' \
| sed 'G;s/\(....\)\(..\)\(..\) \(..\):\(..\):\(..\) *\(.\)/Year=\1\7Month=\2\7Day=\3\7Hour=\4\7Min=\5\7Sec=\6/'
)"
pass via a assignation string to evaluate. You could easily adapt to also check the content by replacing dot per more specific pattern like [0-5][0-9] for min and sec, ...
posix version so --posix on GNU sed

I wrote a function that I usually cut and paste into my script files
function getdate()
{
local a
a=(`date "+%Y %m %d %H %M %S" | sed -e 's/ / /'`)
year=${a[0]}
month=${a[1]}
day=${a[2]}
hour=${a[3]}
minute=${a[4]}
sec=${a[5]}
}
in the script file, on a line of it's own
getdate
echo "year=$year,month=$month,day=$day,hour=$hour,minute=$minute,second=$sec"
Of course, you can modify what I provided or use answer [6] above.
The function takes no arguments.

Related

Shell script - is there a faster way to write date/time per second between start and end time?

I have this script (which works fine) that will write all the date/time per second, from a start date/time till an end date/time to a file
while read line; do
FIRST_TIMESTAMP="20230109-05:00:01" #this is normally a variable that changes with each $line
LAST_TIMESTAMP="20230112-07:00:00" #this is normally a variable that changes with each $line
date=$FIRST_TIMESTAMP
while [[ $date < $LAST_TIMESTAMP || $date == $LAST_TIMESTAMP ]]; do
date2=$(echo $date |sed 's/ /-/g' |sed "s/^/'/g" |sed "s/$/', /g")
echo "$date2" >> "OUTPUTFOLDER/output_LABELS_$line"
date=$(date -d "$date +1 sec" +"%Y%m%d %H:%M:%S")
done
done < external_file
However this sometimes needs to run 10 times, and the start date/time and end date/time sometimes lies days apart.
Which makes the script take a long time to write all that data.
Now I am wondering if there is a faster way to do this.
Avoid using a separate date call for each date. In the next example I added a safety parameter maxloop, avoiding loosing resources when the dates are wrong.
#!/bin/bash
awkdates() {
maxloop=1000000
awk \
-v startdate="${first_timestamp:0:4} ${first_timestamp:4:2} ${first_timestamp:6:2} ${first_timestamp:9:2} ${first_timestamp:12:2} ${first_timestamp:15:2}" \
-v enddate="${last_timestamp:0:4} ${last_timestamp:4:2} ${last_timestamp:6:2} ${last_timestamp:9:2} ${last_timestamp:12:2} ${last_timestamp:15:2}" \
-v maxloop="${maxloop}" \
'BEGIN {
T1=mktime(startdate);
T2=mktime(enddate);
linenr=1;
while (T1 <= T2) {
printf("%s\n", strftime("%Y%m%d %H:%M:%S",T1));
T1+=1;
if (linenr++ > maxloop) break;
}
}'
}
mkdir -p OUTPUTFOLDER
while IFS= read -r line; do
first_timestamp="20230109-05:00:01" #this is normally a variable that changes with each $line
last_timestamp="20230112-07:00:00" #this is normally a variable that changes with each $line
awkdates >> "OUTPUTFOLDER/output_LABELS_$line"
done < <(printf "%s\n" "line1" "line2")
Using epoch time (+%s and #) with GNU date and GNU seq to
produce datetimes in ISO 8601 date format:
begin=$(date -ud '2023-01-12T00:00:00' +%s)
end=$(date -ud '2023-01-12T00:00:12' +%s)
seq -f "#%.0f" "$begin" 1 "$end" |
date -uf - -Isec
2023-01-12T00:00:00+00:00
2023-01-12T00:00:01+00:00
2023-01-12T00:00:02+00:00
2023-01-12T00:00:03+00:00
2023-01-12T00:00:04+00:00
2023-01-12T00:00:05+00:00
2023-01-12T00:00:06+00:00
2023-01-12T00:00:07+00:00
2023-01-12T00:00:08+00:00
2023-01-12T00:00:09+00:00
2023-01-12T00:00:10+00:00
2023-01-12T00:00:11+00:00
2023-01-12T00:00:12+00:00
if you're using macOS/BSD's date utility instead of the gnu one, the equivalent command to parse would be :
(bsd)date -uj -f '%FT%T' '2023-01-12T23:34:45' +%s
1673566485
...and the reverse process is using -r flag instead of -d, sans "#" prefix :
(bsd)date -uj -r '1673566485' -Iseconds
2023-01-12T23:34:45+00:00
(gnu)date -u -d '#1673566485' -Iseconds
2023-01-12T23:34:45+00:00

How to write a shell script to perform below changes in the Parameter file

I am looking to write a shell script that finds and replaces the parameter values every time a workflow is run. I am a beginner and trying to learn more about this.
For example
-$$mp_Custom_Filter_D_Prem=AND BLSB.BLSB_CREATE_DTM <= '2019-02-28'
--$$mp_Custom_Filter_D_LEP=AND convert(date,(substring(SBSR.SBSR_SOURCE,9,2)+substring(SBSR.SBSR_SOURCE,5,2)+substring(SBSR.SBSR_SOURCE,7,2))) <= '2019-02-28'
I have these 2 parameters in the file and I want the script to change the dates to advance a month and set its last day.
So far the date 2019-02-28 I would like the script to find the dates and replace it to 2019-03-31.
I have tried the below script and its not the same result
PRM_FIL_DIR=$1 PRM_FIL_NME=$2 LOG_FIL_DIR=$3 LOG_FIL_NME=$4 echo Begining of the log File > $LOG_FIL_DIR/$LOG_FIL_NME From=`grep '$$mp_Custom_Filter_D_Prem=AND BLSB.BLSB_CREATE_DTM <' $PRM_FIL_DIR/$PRM_FIL_NME | awk -F= '{print $3}'|uniq| sed "s/'//g" | awk '$1=$1' ` echo $From is Date value found in $PRM_FIL_DIR/$PRM_FIL_NME parameter File >> $LOG_FIL_DIR/$LOG_FIL_NME To=`date -d "$From 2 month -1 day" +%Y-%m-%d` echo $To is the value to be replaced in $PRM_FIL_DIR/$PRM_FIL_NME parameter File >> $LOG_FIL_DIR/$LOG_FIL_NME From_FNL="'$From'" To_FNL="'$To'" sed -i "s/$From_FNL/$To_FNL/g" "$PRM_FIL_DIR/$PRM_FIL_NME" echo Date values replaced >> $LOG_FIL_DIR/$LOG_FIL_NME echo End of the log File >> $LOG_FIL_DIR/$LOG_FIL_NME PRM_FIL_DIR=$1 PRM_FIL_NME=$2 LOG_FIL_DIR=$3 LOG_FIL_NME=$4 echo Begining of the log File > $LOG_FIL_DIR/$LOG_FIL_NME From=`grep '$$mp_Custom_Filter_D_Prem=AND BLSB.BLSB_CREATE_DTM <' $PRM_FIL_DIR/$PRM_FIL_NME | awk -F= '{print $3}'|uniq| sed "s/'//g" | awk '$1=$1' ` echo $From is Date value found in $PRM_FIL_DIR/$PRM_FIL_NME parameter File >> $LOG_FIL_DIR/$LOG_FIL_NME To=`date -d "$From 2 month -1 day" +%Y-%m-%d` echo $To is the value to be replaced in $PRM_FIL_DIR/$PRM_FIL_NME parameter File >> $LOG_FIL_DIR/$LOG_FIL_NME From_FNL="'$From'" To_FNL="'$To'" sed -i "s/$From_FNL/$To_FNL/g" "$PRM_FIL_DIR/$PRM_FIL_NME" echo Date values replaced >> $LOG_FIL_DIR/$LOG_FIL_NME echo End of the log File >> $LOG_FIL_DIR/$LOG_FIL_NME
I want the output to be the last day of the month i.e 2019-03-31 and actual output is 2019-03-29
Try this flow,
lastmonth=$( date -d "-$(date +%d) days" +%Y%m%d)
currentmonth=$( date -d "-$(date +%d) days month" +%Y%m%d)
sed "s/$lastmonth/$currentmonth/g" filename.txt

Error message "date: Argument list too long" bash

So, everything works fine in the code, except for one tiny little thing.
This part:
if [ "$LIMITHOURS" -gt "0" -a "$LIMITHOURS" -lt "24" ]; then
x=$(($LIMITHOURS*60*60))
fi
SDATE=$( echo "01/jan/2003:11:00:06 +0100"| sed 's/[/]/ /g' |sed 's/:/ /')
EDATE=$(date --date "$SDATE - $x seconds" +"%d%m%Y%H%M%S")
#echo "$SDATE"
#echo "$EDATE"
while read LINE; do
CDATE=$( awk '{print $4}'| sed 's/[[]//' | sed 's/[/]//g' |sed 's/://g' )
DATE=$(date --date "$CDATE" +"%d%m%Y%H%M%S")
#echo "$CDATE"
done < "$FILENAME"
When I try to run the script, I get the error message "date: Argument list too long
" and I know that the problem is in the while loop, with:
DATE=$(date --date "$CDATE" +"%d%m%Y%H%M%S")
Anyone who know any solution for this? I want the date format in ddmmYYYYHHMMSS, eg. 23102002120022
You can find rest of the script here: http://pastebin.com/PMk2QDre
This code:
while read LINE; do
CDATE=$( awk '{print $4}'| sed 's/[[]//' | sed 's/[/]//g' |sed 's/://g' )
DATE=$(date --date "$CDATE" +"%d%m%Y%H%M%S")
#echo "$CDATE"
done < "$FILENAME"
will read one line from $FILENAME into the variable LINE, but then the first call to awk is reading the rest of the lines. The resulting CDATE value is probably too large to fit in a single command line, never mind it containing too many dates. You probably wanted
echo "$LINE" | awk '{print $4}' | ...
A simpler way to strip the undesirable characters from LINE, however, is
CDATE=${LINE//[\/[:]}

How to find the first occurence of date which is greater than or eqaul to particular date in text file using shell script

past_date='2013-11-14'
initial_time=$(grep -o -m1 "$past_date [0-9][0-9]:[0-9][0-9]:[0-9][0-9]" logfile.txt)
/* Here I am trying to find the first occurence of date which is greater than or eqaul to '2013-11-14', Above code I have tried ,It is giving only that particular line of file, If that date is not found It has to give next date which is greater than 2013-11-14 date */
Using awk
past_date='20131114'
awk '{d=$1;gsub(/-/,"",d);if (d>=p) {print;exit}}' p=$past_date logfile
2013-11-15 15:45:40 Starting agent install process
If you use bash, then you might want to try something like:
past_date='2013-11-14'
initial_time=$(grep -oP '\d{4}-\d\d-\d\d \d\d:\d\d:\d\d' < logfile.txt | \
while read LINE ; do if [ "$LINE" '>' "$past_date" ]; then echo $LINE; break; fi ; done)
while read line
do
initial_time=`echo $line | sed -e 's/\([0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9] [0-9][0-9]:[0-9][0-9]:[0-9][0-9]\).*/\1/'`
file_content_date=`date -d "$initial_time" +%Y%m%d`
comparison_past_date=`date -d "$past_date" +%Y%m%d`
if [ $comparison_past_date -le $file_content_date ]; then
comparison_start_date=`date -d "$file_content_date" +%Y%m%d`
break
fi
done < logfile.txt
fi

Nested for loop issue will not move to next file

find_date=$(stat -c %y $files | awk '{print $1}')
#Grabbing each file from the array
for file in "${files[#]}"; do
# We get the date part
file_date=''
#Reading the date and breaking by the - mark
IFS="-" read -ra parts <<< "$file"
unset file_date
find_date=$(stat -c %y $files | awk '{print $1}')
echo "File Date: " $find_date
for t in "${find_date[#]}"; do
#Putting the date into the array
if [[ $t == [0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9] ]]; then
file_date=$t
break
fi
done
echo "T: " $t
So everything works but the for loop where it should move to the next file. When i run my script i notice all the errors for the STAT command not working because after it does the first file it is still trying to STAT that file and not the next one in the list
reset the file_date so it does not 'remember' the result of the previous loop, ie:
unset file_date
for t in "${find_date[#]}"; do
#Putting the date into the array
if [[ $t == [0-9][0-9][0-9][0-9]-[0-9][0-9]-[0-9][0-9] ]]; then
file_date=$t
break
fi
done
This line:
find_date=$(stat -c %y $files | awk '{print $1}')
files is the array, so $files expands to the first element of that array each time through the loop. You want to use the file variable that you use to iterate over the array.
for file in "${files[#]}"; do
...
find_date=$(stat -c %y "$file" | awk '{print $1}')
# ^^^^^^^
...
done

Resources