I am looking to write a shell script that finds and replaces the parameter values every time a workflow is run. I am a beginner and trying to learn more about this.
For example
-$$mp_Custom_Filter_D_Prem=AND BLSB.BLSB_CREATE_DTM <= '2019-02-28'
--$$mp_Custom_Filter_D_LEP=AND convert(date,(substring(SBSR.SBSR_SOURCE,9,2)+substring(SBSR.SBSR_SOURCE,5,2)+substring(SBSR.SBSR_SOURCE,7,2))) <= '2019-02-28'
I have these 2 parameters in the file and I want the script to change the dates to advance a month and set its last day.
So far the date 2019-02-28 I would like the script to find the dates and replace it to 2019-03-31.
I have tried the below script and its not the same result
PRM_FIL_DIR=$1 PRM_FIL_NME=$2 LOG_FIL_DIR=$3 LOG_FIL_NME=$4 echo Begining of the log File > $LOG_FIL_DIR/$LOG_FIL_NME From=`grep '$$mp_Custom_Filter_D_Prem=AND BLSB.BLSB_CREATE_DTM <' $PRM_FIL_DIR/$PRM_FIL_NME | awk -F= '{print $3}'|uniq| sed "s/'//g" | awk '$1=$1' ` echo $From is Date value found in $PRM_FIL_DIR/$PRM_FIL_NME parameter File >> $LOG_FIL_DIR/$LOG_FIL_NME To=`date -d "$From 2 month -1 day" +%Y-%m-%d` echo $To is the value to be replaced in $PRM_FIL_DIR/$PRM_FIL_NME parameter File >> $LOG_FIL_DIR/$LOG_FIL_NME From_FNL="'$From'" To_FNL="'$To'" sed -i "s/$From_FNL/$To_FNL/g" "$PRM_FIL_DIR/$PRM_FIL_NME" echo Date values replaced >> $LOG_FIL_DIR/$LOG_FIL_NME echo End of the log File >> $LOG_FIL_DIR/$LOG_FIL_NME PRM_FIL_DIR=$1 PRM_FIL_NME=$2 LOG_FIL_DIR=$3 LOG_FIL_NME=$4 echo Begining of the log File > $LOG_FIL_DIR/$LOG_FIL_NME From=`grep '$$mp_Custom_Filter_D_Prem=AND BLSB.BLSB_CREATE_DTM <' $PRM_FIL_DIR/$PRM_FIL_NME | awk -F= '{print $3}'|uniq| sed "s/'//g" | awk '$1=$1' ` echo $From is Date value found in $PRM_FIL_DIR/$PRM_FIL_NME parameter File >> $LOG_FIL_DIR/$LOG_FIL_NME To=`date -d "$From 2 month -1 day" +%Y-%m-%d` echo $To is the value to be replaced in $PRM_FIL_DIR/$PRM_FIL_NME parameter File >> $LOG_FIL_DIR/$LOG_FIL_NME From_FNL="'$From'" To_FNL="'$To'" sed -i "s/$From_FNL/$To_FNL/g" "$PRM_FIL_DIR/$PRM_FIL_NME" echo Date values replaced >> $LOG_FIL_DIR/$LOG_FIL_NME echo End of the log File >> $LOG_FIL_DIR/$LOG_FIL_NME
I want the output to be the last day of the month i.e 2019-03-31 and actual output is 2019-03-29
Try this flow,
lastmonth=$( date -d "-$(date +%d) days" +%Y%m%d)
currentmonth=$( date -d "-$(date +%d) days month" +%Y%m%d)
sed "s/$lastmonth/$currentmonth/g" filename.txt
Related
I have this script (which works fine) that will write all the date/time per second, from a start date/time till an end date/time to a file
while read line; do
FIRST_TIMESTAMP="20230109-05:00:01" #this is normally a variable that changes with each $line
LAST_TIMESTAMP="20230112-07:00:00" #this is normally a variable that changes with each $line
date=$FIRST_TIMESTAMP
while [[ $date < $LAST_TIMESTAMP || $date == $LAST_TIMESTAMP ]]; do
date2=$(echo $date |sed 's/ /-/g' |sed "s/^/'/g" |sed "s/$/', /g")
echo "$date2" >> "OUTPUTFOLDER/output_LABELS_$line"
date=$(date -d "$date +1 sec" +"%Y%m%d %H:%M:%S")
done
done < external_file
However this sometimes needs to run 10 times, and the start date/time and end date/time sometimes lies days apart.
Which makes the script take a long time to write all that data.
Now I am wondering if there is a faster way to do this.
Avoid using a separate date call for each date. In the next example I added a safety parameter maxloop, avoiding loosing resources when the dates are wrong.
#!/bin/bash
awkdates() {
maxloop=1000000
awk \
-v startdate="${first_timestamp:0:4} ${first_timestamp:4:2} ${first_timestamp:6:2} ${first_timestamp:9:2} ${first_timestamp:12:2} ${first_timestamp:15:2}" \
-v enddate="${last_timestamp:0:4} ${last_timestamp:4:2} ${last_timestamp:6:2} ${last_timestamp:9:2} ${last_timestamp:12:2} ${last_timestamp:15:2}" \
-v maxloop="${maxloop}" \
'BEGIN {
T1=mktime(startdate);
T2=mktime(enddate);
linenr=1;
while (T1 <= T2) {
printf("%s\n", strftime("%Y%m%d %H:%M:%S",T1));
T1+=1;
if (linenr++ > maxloop) break;
}
}'
}
mkdir -p OUTPUTFOLDER
while IFS= read -r line; do
first_timestamp="20230109-05:00:01" #this is normally a variable that changes with each $line
last_timestamp="20230112-07:00:00" #this is normally a variable that changes with each $line
awkdates >> "OUTPUTFOLDER/output_LABELS_$line"
done < <(printf "%s\n" "line1" "line2")
Using epoch time (+%s and #) with GNU date and GNU seq to
produce datetimes in ISO 8601 date format:
begin=$(date -ud '2023-01-12T00:00:00' +%s)
end=$(date -ud '2023-01-12T00:00:12' +%s)
seq -f "#%.0f" "$begin" 1 "$end" |
date -uf - -Isec
2023-01-12T00:00:00+00:00
2023-01-12T00:00:01+00:00
2023-01-12T00:00:02+00:00
2023-01-12T00:00:03+00:00
2023-01-12T00:00:04+00:00
2023-01-12T00:00:05+00:00
2023-01-12T00:00:06+00:00
2023-01-12T00:00:07+00:00
2023-01-12T00:00:08+00:00
2023-01-12T00:00:09+00:00
2023-01-12T00:00:10+00:00
2023-01-12T00:00:11+00:00
2023-01-12T00:00:12+00:00
if you're using macOS/BSD's date utility instead of the gnu one, the equivalent command to parse would be :
(bsd)date -uj -f '%FT%T' '2023-01-12T23:34:45' +%s
1673566485
...and the reverse process is using -r flag instead of -d, sans "#" prefix :
(bsd)date -uj -r '1673566485' -Iseconds
2023-01-12T23:34:45+00:00
(gnu)date -u -d '#1673566485' -Iseconds
2023-01-12T23:34:45+00:00
from my wp cli I receive a date format like this :
YYYY-mm-ddHH-mm-ss for example : 2020-02-2514:24:25
I would like to convert it to timestamp but the format date is incorrect.
I would like to split the date and time with space between them but I have no idea to do that currently.
with regex may be but I just seen how to replace space and I am a nooby with bash regex.
thank you for your help
Just use cut
root#a036fb1c94fa:~# DATE=$(echo "2020-02-2514:24:25" | cut -b-10)
root#a036fb1c94fa:~# TIME=$(echo "2020-02-2514:24:25" | cut -b11-)
root#a036fb1c94fa:~# TIMESTAMP=$(date -d "$DATE $TIME" +"%s")
root#a036fb1c94fa:~# echo $TIME
14:24:25
root#a036fb1c94fa:~# echo $DATE
2020-02-25
root#a036fb1c94fa:~# echo $TIMESTAMP
1582637065
Explanation:
echo "2020-02-2514:24:25" | cut -b-10
echo the string and cut it before the 10th byte
echo "2020-02-2514:24:25" | cut -b11-
echo the string and cut it from the 11th byte until the end
echo date -d "$DATE $TIME" +"%s"
give the right format to unix date command, with +"%s" to get its timestamp
You don't need any regex. You can select substrings in bash.
If the variable wpdate contains the string returned by wp cli, the corresponding timestamp can be put into the variable timestamp as follows:
timestamp=$(date '+%s' --date "${wpdate:0:10} ${wpdate:10:8}")
Explanation: ${wpdate:10:8} means the substring of wpdate starting at position 10 and containing 8 chars.
I have file containing multiple lines in format "[dd.mm.yyyy.] text value". I need to convert this to "Unix epoch| text value". I tried to use awk to do this but I can't seem to find the correct command
For example, if the file is:
[30.08.2013 13:54:49.126] Foo
[30.08.2013 13:56:49.126] Bar
[30.08.2013 13:59:49.126] Foo bar
I use the following (probably too complex awk command):
cat sample.txt | cut -c 2- |awk -F'[. :]' ' { $cmd="date --date " "\""$3$2$1" "$4":"$5":"$6"\""" +%s" ; $cmd |& getline epoch; close($cmd); printf epoch"|"; print $0 ;}';
The problem is that I get the time in epoch correctly but I can't access the rest of the line. The $0 (and other $ variables) contain the date command. So the output is
1377863689|date --date "20130830 13:54:49" +%s
1377863809|date --date "20130830 13:56:49" +%s
1377863989|date --date "20130830 13:59:49" +%s
What I wish to get is
1377863689|Foo
1377863809|Bar
1377863989|Foo bar
Is there a (preferably simple) way of accomplishing this? Should I use some other tool?
Assuming you have gawk (fair assumption since you are using GNU date) you can do this all internally to gawk:
$ awk 'match($0, /\[(.*)\] (.*)/, a) &&
match(a[1], /([0-9]{2})\.([0-9]{2})\.([0-9]{4}) ([0-9:]+)(\.[0-9]+)/,b) {
gsub(/:/," ",b[4])
s=b[3] " " b[2] " " b[1] " " b[4]
print mktime(s) "|" a[2]
}' file
1377896089|Foo
1377896209|Bar
1377896389|Foo bar
Or, a Bash solution:
while IFS= read -r line; do
if [[ "$line" =~ \[([[:digit:]]{2})\.([[:digit:]]{2})\.([[:digit:]]{4})\ +([[:digit:]:]+)\.([[:digit:]]+)\]\ +(.*) ]]
then
printf "%s|%s\n" $(gdate +"%s" --date="${BASH_REMATCH[3]}${BASH_REMATCH[2]}${BASH_REMATCH[1]} ${BASH_REMATCH[4]}") "${BASH_REMATCH[6]}"
fi
done <file
I propose to simplify it to
IFS=' |.|[';
while read -r _ day month year hour _ name; do
date=$(date --date "$year$month$day $hour" +%s);
echo "$date|$name";
done < sample.txt
Or, if you prefer to continue with awk
awk -F'[\\[\\]. ]' '{
split($0,a,"] ")
("date --date \"" $4$3$2" "$5"\" +%s") |& getline date
printf "%s|%s\n",date,a[2]
}' sample.txt
I have date and time format like this(yearmonthday):
20141105 11:30:00
I need assignment year, month, day, hour and minute values to variable.
I can do it year, day and hour like this:
year=$(awk '{print $1}' log.log | sed 's/^\(....\).*/\1/')
day=$(awk '{print $1}' log.log | sed 's/^.*\(..\).*/\1/')
hour=$(awk '{print $2}' log.log | sed 's/^\(..\).*/\1/')
How can I do this for month and minute?
--
And I need that every line of my log file:
20141105 11:30:00 /bla/text.1
20141105 11:35:00 /bla/text.2
20141105 11:40:00 /bla/text.3
....
I'm trying read line by line this log file and do this:
mkdir -p "/bla/backup/$year/$month/$day/$hour/$minute"
mv $file "/bla/backup/$year/$month/$day/$hour/$minute"
Here is my not working code:
#!/bin/bash
LOG=/var/log/LOG
while read line
do
year=${line:0:4}
month=${line:4:2}
day=${line:6:2}
hour=${line:9:2}
minute=${line:12:2}
file=$(awk '{print $3}')
if [ -f "$file" ]; then
printf -v path "%s/%s/%s/%s/%s" $year $month $day $hour $minute
mkdir -p "/bla/backup/$path"
mv $file "/bla/backup/$path"
fi
done < $LOG
You don't need to call out to awk to date at all, use bash's substring operations
d="20141105 11:30:00"
yr=${d:0:4}
mo=${d:4:2}
dy=${d:6:2}
hr=${d:9:2}
mi=${d:12:2}
printf -v dir "/bla/%s/%s/%s/%s/%s/\n" $yr $mo $dy $hr $mi
echo "$dir"
/bla/2014/11/05/11/30/
Or directly, without all the variables.
printf -v dir "/bla/%s/%s/%s/%s/%s/\n" ${d:0:4} ${d:4:2} ${d:6:2} ${d:9:2} ${d:12:2}
Given your log file:
while read -r date time file; do
d="$date $time"
printf -v dir "/bla/%s/%s/%s/%s/%s/\n" ${d:0:4} ${d:4:2} ${d:6:2} ${d:9:2} ${d:12:2}
mkdir -p "$dir"
mv "$file" "$dir"
done < filename
or, making a big assumption that there are no whitespace or globbing characters in your filenames:
sed -r 's#(....)(..)(..) (..):(..):.. (.*)#mv \6 /blah/\1/\2/\3/\4/\5#' | sh
date command also do this work
#!/bin/bash
year=$(date +'%Y' -d'20141105 11:30:00')
day=$(date +'%d' -d'20141105 11:30:00')
month=$(date +'%m' -d'20141105 11:30:00')
minutes=$(date +'%M' -d'20141105 11:30:00')
echo "$year---$day---$month---$minutes"
You can use only one awk
month=$(awk '{print substr($1,5,2)}' log.log)
year=$(awk '{print substr($1,0,4)}' log.log)
minute=$(awk '{print substr($2,4,2)}' log.log)
etc
I guess you are processing the log file, which each line starts with the date string. You may have already written a loop to handle each line, in your loop, you could do:
d="$(awk '{print $1,$2}' <<<"$line")"
year=$(date -d"$d" +%Y)
month=$(date -d"$d" +%m)
day=$(date -d"$d" +%d)
min=$(date -d"$d" +%M)
Don't repeat yourself.
d='20141105 11:30:00'
IFS=' ' read -r year month day min < <(date -d"$d" '+%Y %d %m %M')
echo "year: $year"
echo "month: $month"
echo "day: $day"
echo "min: $min"
The trick is to ask date to output the fields you want, separated by a character (here a space), to put this character in IFS and ask read to do the splitting for you. Like so, you're only executing date once and only spawn one subshell.
If the date comes from the first line of the file log.log, here's how you can assign it to the variable d:
IFS= read -r d < log.log
eval "$(
echo '20141105 11:30:00' \
| sed 'G;s/\(....\)\(..\)\(..\) \(..\):\(..\):\(..\) *\(.\)/Year=\1\7Month=\2\7Day=\3\7Hour=\4\7Min=\5\7Sec=\6/'
)"
pass via a assignation string to evaluate. You could easily adapt to also check the content by replacing dot per more specific pattern like [0-5][0-9] for min and sec, ...
posix version so --posix on GNU sed
I wrote a function that I usually cut and paste into my script files
function getdate()
{
local a
a=(`date "+%Y %m %d %H %M %S" | sed -e 's/ / /'`)
year=${a[0]}
month=${a[1]}
day=${a[2]}
hour=${a[3]}
minute=${a[4]}
sec=${a[5]}
}
in the script file, on a line of it's own
getdate
echo "year=$year,month=$month,day=$day,hour=$hour,minute=$minute,second=$sec"
Of course, you can modify what I provided or use answer [6] above.
The function takes no arguments.
I am not sure how this will be done in unix can anybody help/suggest to convert
2012-05-03T25:00:00 to 2012-05-04T01:00:00 in Unix command/script
In my file I have more than one occurrence with different times those needs to converted to next day
2012-05-03T25:00:00 to 2012-05-04T01:00:00
2012-05-03T26:50:00 to 2012-05-04T02:50:00
2012-05-03T31:59:59 to 2012-05-04T07:59:59
etc
I tried it but somehow sed is not working
Date.txt
2009-09-12T05:18:#00#+10:00,D,
2009-09-12T05:24:00+10:00,2009-09-12T05:24:#00#+10:00,D,
2009-09-12T05:25:00+10:00,2009-09-12T05:25:#00#+10:00,D,
2009-09-12T05:27:00+10:00,2009-09-12T05:27:#00#+10:00,D,
2009-09-12T30:29:00+10:00,2009-09-12T05:29:#00#+10:00,D,
2009-09-12T29:31:00+10:00,2009-09-12T05:31:#00#+10:00,D,
2009-09-12T28:33:00+10:00,,D,
2009-09-12T27:00:#00#+10:00,U,
2009-09-12T26:01:00+10:00,2009-09-12T05:01:#00#+10:00,U,
2009-09-12T24:04:00+10:00,2009-09-12T05:04:#00#+10:00,U,
2009-09-12T24:59:59+10:00,2009-09-12T05:06:#00#+10:00,U,
2009-09-12T30:08:00+10:00,2009-09-12T05:08:#00#+10:00,U,
2009-09-12T31:59:59+10:00,2009-09-12T05:10:#00#+10:00,U,
2009-09-12T05:17:00+10:00,,U,
2009-09-12T25:25:#00#+10:00,D,
script.sh
awk -F"T" -v OFS=',' '{print $1}' date.txt > tmpdate
uniq -d tmpdate > tmpuniq
rm tmpdate
date1=`cat tmpuniq`
date2=`date --set="$date1" +%F`
date3=$(date --date="$date2" -d "+1 day" +"%F")
T1=$date2+"T24"
T2=$date3+"T00"
echo $T1
echo $T2
dos2unix date.txt
#sed -i 's/$T1/$T2/g' date.txt > test.txt
#sed -i 's/"$T1"/"$T2"/g' date.txt > test.txt
sed -i 's/'$T1'/'$T2'/g' date.txt
Any help much appreciated
If you have Tcl on your system, the clock command makes it pretty easy:
set t 2012-05-03T31:59:59
lassign [split $t T] date time
lassign [split $time :] hour min sec
set base [clock scan $date -format %Y-%m-%d]
set new [clock add $base $hour hours $min minutes $sec seconds]
puts [clock format $new -format %Y-%m-%dT%T] ;# -> 2012-05-04T07:59:59
Update for cygwin:
first, using the cygwin installer, install the version 8.5.x of the "tcl" package (located in the Interpreters category). Then, you can do this
normalize_time() {
printf '
lassign [split "%s" T] date time
lassign [split $time :] hour min sec
set base [clock scan $date -format %%Y-%%m-%%d]
set new [clock add $base $hour hours $min minutes $sec seconds]
puts [clock format $new -format %%Y-%%m-%%dT%%T]
' "$1" | tclsh
}
normalize_time 2012-05-03T31:59:59
You will basically need to parse the erroneous time specification, convert everything to seconds, add it back together, and convert it back to a proper date.
date -d "date" +%s produces seconds since January 1, 1970. date -d #1234567890 converts a date in this format back to a regular human-readable date.
#!/bin/sh
IFS=T:
while read date hh mm ss; do
basedate=$(date -d "$date" +%s)
date -d #$(echo "$basedate+($hh*60*60)+($mm*60)+$ss" | bc) +%FT%T
done <<EOF
2012-05-03T25:00:00
2012-05-03T26:50:00
2012-05-03T31:59:59
EOF
I didn't have bc on this computer so I replaced my original attempt with a small awk script, but it's not very readable.
awk 'END { print '"$basedate"'+('"$hh"'*60*60)+('"$mm"'*60)+'"$ss"'}' </dev/null
If you don't have a recent enough version of GNU date, you can find some alternative ways at http://www.antonolsen.com/2006/04/06/bash-convert-unix-timestamp-to-a-date/
finally I got a solution with a bug (if date is last day of the month then this will not work but I can live with that)
and did a clumsy job
date1=`cat date.txt | awk -F. '{print substr($1,1,10)}'|uniq |head -1`
echo $date1
date2=$date1"T24"
date21=$date1"T25"
date22=$date1"T26"
date23=$date1"T27"
date24=$date1"T28"
date25=$date1"T29"
date26=$date1"T30"
date27=$date1"T31"
date3=$(date --date="$date1" -d "+1 day" +"%F")
echo $date2
cat date.txt | grep "$date2"
tmpdate=`echo $date1 | sed -e 's/-//g'`
echo $tmpdate
date4=`echo $(date -d $(echo \`expr $tmpdate + 1\`F) +"%F")`
echo $date4
date41=$date4"T00"
date42=$date4"T01"
date43=$date4"T02"
date44=$date4"T03"
date45=$date4"T04"
date46=$date4"T05"
date47=$date4"T06"
date48=$date4"T07"
sed -i 's/'$date2'/'$date41'/g' date.txt
sed -i 's/'$date21'/'$date42'/g' date.txt
sed -i 's/'$date22'/'$date43'/g' date.txt
sed -i 's/'$date23'/'$date44'/g' date.txt
sed -i 's/'$date24'/'$date45'/g' date.txt
sed -i 's/'$date25'/'$date46'/g' date.txt
sed -i 's/'$date26'/'$date47'/g' date.txt
sed -i 's/'$date27'/'$date48'/g' date.txt
If anyone wants to improve this script please go ahead.
Once again thanks to all.
First, use cat 'yourfile' to open it. Then do a loop on each line. Then use grep and cut. That's really simple to do and the most efficient way for you to learn is to go through the man pages of cat, grep, and cut.
Hope it'll help.