We have a 15 year (or so) old script we are trying to figure out and document. We have found some errors in it but one specific log file gives us much headache. and I would love some help figuring it out.
First the function that are run with the question:
#=========================================================#
# Define function removeOldBackupFile. #
#=========================================================#
removeOldBackupFile()
{
#set -x
echo "Removing old backups if they exists." >> "${report}"
local RCLOC=0
spaceBefore=$(getAvailableSpace ${backupDirectory})
timesToWait=60 # Wait a maximum of 10 minutes before bailing
cat ${oldDbContainer} | while read fileName
do
echo "Old file exists. Removing ${fileName}." >> "${report}"
removeFileIfExist "${fileName}"
RC=$?
echo "Resultcode for removing old backup is: RC=$RC." >> "${report}"
RCLOC=$(($RC+$RCLOC))
spaceAfter=$(getAvailableSpace ${backupDirectory})
# Wait for the OS to register that the file is removed
cnt=0
while [ $spaceAfter -le $spaceBefore ]; do
cnt=$((cnt+1))
if [ $cnt -gt $timesToWait ]; then
echo "Waited too long for space in ${backupDirectory}" | tee -a "${report}"
RCLOC=$(($RCLOC+1))
return $RCLOC
fi
sleep 10
spaceAfter=$(getAvailableSpace ${backupDirectory})
done
done
return $RCLOC
}
The place where this function is ran looks as follows:
#=========================================================#
# Remove old backupfiles if any exist. #
#=========================================================#
removeOldBackupFile
RC=$?
RCSUM=$(($RC+$RCSUM))
We have identified that the if condition is a bit wrong and the while loops would not work as intended if there are multiple files.
But what bothers us is output from a log file:
...
+ cnt=61
+ '[' 61 -gt 60 ']'
+ echo 'Waited too long for space in /<redacted>/backup'
+ tee -a /tmp/maintenanceBackupMessage.70927
Waited too long for space in /<redacted>/backup
+ RCLOC=1
+ return 1
+ return 0
+ RC=0
+ RCSUM=0
...
As seen in the log output after the inner loop have ran 60 times and ending it returns 1 as expected.. BUT! it also have return 0 after!? Why is it also returning 0?
We are unable to figure out the double returns... Any help appriciated
The first return executes in the subshell started by the pipe cat ${oldDbContainer} | while .... The second return is from return $RCLOC at the end of the function. Get rid of the useless use of cat:
removeOldBackupFile()
{
#set -x
echo "Removing old backups if they exists." >> "${report}"
local RCLOC=0
spaceBefore=$(getAvailableSpace ${backupDirectory})
timesToWait=60 # Wait a maximum of 10 minutes before bailing
while read fileName
do
...
done < ${oldDbContainer}
return $RCLOC
}
We have backup foldernames like "db_YYYY-MM-DD".
Is there some kind of command/expression I could use to compare the folder's (name-)date to the current (system-)date and delete anything older than X days?
Background:
We already tried "find" and "-mtime +X" but modification time is not a perfect solution because it updates when you modify/delete files inside the folders afterwards..
It is a rare condition but makes the -mtime comparison "faulty", so I look for an alternative and the folder's name doesn't change ever during creation/backup/modification..
You can use following way to generate the folder name
#foldername="db_"`date "+%Y-%m-%d"`
#echo $foldername
db_2017-08-15
#mkdir $foldername
Now you can compare like string with given date
so 7 days older date will be
givenDate="db_"`date -v-7d "+%Y-%m-%d"`
if ($foldername > $givenDate)
True if foldername sorts after givenDate lexicographically.
The good thing about the YYYY-MM-DD date format is that the dictionary comparison order matches the time order. Therefore you can simply extract the backup date from your directory name and compare it as a string with the corresponding date in the same format:
backupdirname=db_2017-08-12
X=6
backupdate="${backupdirname: -10}"
dropdate="$(date -d "$X days ago" +%Y-%m-%d)"
if [[ "$backupdate" < "$dropdate" ]]
then
echo "Delete $backupdirname"
else
echo "Keep $backupdirname"
fi
ref_date=$(date -d '2017-01-02' +%s)
for file in ./db_*; do
file_time=$(date -d ${file#*db_} +%s)
if [[ $file_time > $ref_date ]]; then
echo "$file date is newer than ref date"
# ...
else
echo "$file date is older or euqal to ref date"
# ...
fi
done
I have a bash script that creates a file with a timestamp as the name. Once some time passes, it is supposed to pick up that file and do something with it. I want it to pick it up after two hours, but for some reason, it is picking it up after 57 minutes (and 6 seconds). Can anyone point me to an error in my logic or assumptions?
Here are the details:
I have a variable set to 2 hours (7200 seconds):
SERVICE_DURATION=${SERVICE_DURATION:-7200} # seconds
I am setting the filename equal to the Unix timestamp concatenated with nanoseconds:
active_name=`date +%s%N`
echo "${1}" >> ${ACTIVE_DIR}/${active_name}
I then loop forever until the time is right:
while true
do
for fa in ${ACTIVE_DIR}/*
do
if [ $(basename ${fa}) -le $(($(date +%s%N) - ${SERVICE_DURATION} * 1000000000)) ]
then
exec 6< "${fa}"
read old_port <&6
read old_host <&6
read old_config <&6
exec 6<&-
logger -p daemon.info "Recycling port ${old_port}."
start_remote_service "${old_port}"
stop_remote_service "${old_port}" "${old_host}" "${old_config}" "${fa}"
sleep 2
fi
done
sleep 30
done
I can't see what is wrong with this. The filename ($(basename ${fa})) shouldn't be less than the current time minus the specified duration in nanoseconds ($(($(date +%s%N) - ${SERVICE_DURATION} * 1000000000))) until the duration has passed.
In order to keep the script from constantly checking, there is a sleep 30 at the end of the loop, so it could be that the time is somewhere between 56:36 and 57:06.
Any help would be appreciated. Thanks.
I have been working on this on and off for the last two months, and despite how many times I look at it, I can't make it work.
This script checks daily log files for a user defined variable, so they don't have to look through every one manually. It worked great checking the current month, but if the user wants to check back 20 days, and today is the 12th of this month, I wanted to be able to then go back to the previous month (not look for the log file with a date of 20150399 and so on). I have checked the logic for my date/day computations, and they seem okay (if there is a better way to do that in BASH, I am open to suggestions). What happens when I try to debug is unexpected end of file. I am somewhat new to writing scripts that contain more than 20 or so lines, but I just can't come up with what I am missing.
I have tried various fixes, to no avail, but I think this is the last iteration.
Ideas?
#!/bin/bash
########################################################
# multi_log_chk.sh
# This script will take input from the user and report which
# CyberFusion MFT logs contain what the user is looking for.
# Hopefully this will save the user having to search through every
# stinking log file to find what they are looking for.
# 20150406 pxg007 started typing
# 20150413 pxg007 added && comparison for back out (line 28)
# added message for no entries found (line 32, 38, 48-52)
# Added some further description (line 16)
# 20150424 pxg007 Added logic to calculate previous month and if necessary, year. (Lines 16-24, 60-78 )
#
########################################################
currDate=`date +%d%B%C%y`
currDay=`date +%d`
currMnth=`date +%m`
currYear=`date +%C%y`
case $currMnth in #Let's establish number of days for previous month
05 | 07 | 10 | 12 ) lastMnthD=30;;
01 |02 | 04 | 06 | 09 | 08 | 11 ) lastMnthD=31;;
03 ) lastMnthD=28;; ##and screw leap year
esac
if [ $currMnth -eq 01 ]; then ##accounting for January
lastMnth=12
else
lastMnth=$((currMnth-1))
fi
if [ $lastMnth -eq 12 ]; then ## accounting for Dec of previous year
lastMnthYr=$((currYear-1))
else
lastMnthYr=$currYear
fi
echo "This script will find entries for your query in whatever available MFT logs you request."
echo " "
echo "For instance - how many log files have transfer entries with \"DOG\" in them?"
echo " "
echo "I also will also give an estimate of how many transfers per log file contain your query, give or take a couple."
echo " "
echo "This search is case sensitive, so \"DOG\" is *** NOT *** the same as \"dog\""
echo " "
read -p "What text you are looking for? Punctuation is okay, but no spaces please. " looking ### what we want to find
echo " "
echo "Today's date is: $currDate."
echo " "
read -p "How many days back do you want to search(up to 25)? " daysBack ### How far back we are going to look
if [ "$daysBack" == 0 ] && [ "$daysBack" >> 25 ]; then
echo "I said up to 25 days. We ain't got more than that!"
exit 1
fi
echo " "
echo "I am going to search through the last $daysBack days of log files for:\"$looking\" "
echo " "
read -p "Does this look right? Press N to quit, or any other key to continue: " affirm
if [ "$affirm" = N ] && [ "$affirm" = n ]; then ###Yes, anything other than "N" or "n" is a go
echo "Quitter!"
exit 1
else
nada=0 ### Used to test for finding anything
backDate=$((currDay-daysBack)) ### current month iterator (assuming query covers only current month)
if (("$daysBack" => "$currDay")); then ## If there are more logs requested than days in the month...
lastMnthCnt=$((daysBack-currDay)) ### how many days to check last month
lastMnthStrt=$((lastMnthD-lastMnthCnt)) ## last month start and iterator
backDate=$(currDay-(daysBack-lastMnthCnt)) # Setting the iterator if we have to go back a month
while (("$lastMnthStrt" <= "$lastMnthD" )); do
foundIt=$(grep "$looking" /CyberFusion/log/Log.txt."$lastMnthYr$lastMnth$lastMnthStrt" | parsecflog | wc -l )
howMany=$((foundIt/40+1)) ### Add one in case there are less than 40 lines in the record.
if (("$foundIt" > 0))
then
nada=$((nada+1))
echo "Log.txt.$lastMnthYr$lastMnth$lastMnthStrt contains $looking in approximately $howMany transfer records."
lastMnthStrt=$((lastMnthStrt+1))
echo " "
else
lastMnthStrt=$((lastMnthStrt+1))
fi
fi
backDate=$((currDay-daysBack)) ### current month iterator (assuming query covers only current month)
while (("$backDate" <= "$currDay")); do
foundIt=$(grep "$looking" /CyberFusion/log/Log.txt."$backDate" | parsecflog | wc -l )
howMany=$((foundIt/40+1)) ### Add one in case there are less than 40 lines in the record.
if (("$foundIt" > 0))
then
nada=$((nada+1))
echo "Log.txt.$backDate contains $looking in approximately $howMany transfer records."
backDate=$((backDate+1))
echo " "
else
backDate=$((backDate+1))
fi
if [ "$nada" \< 1 ]
then
echo " "
echo "I found no entries for $looking in any log file."
fi
You are missing the keyword 'done' on lines 81 and 96 and also a final 'fi' keyword on the last line.
Also as others suggested you can do
date -d "20 days ago" +"%d%B%C%y"
to easily get dates in the past
Is there a way to embed the last command's elapsed wall time in a Bash prompt? I'm hoping for something that would look like this:
[last: 0s][/my/dir]$ sleep 10
[last: 10s][/my/dir]$
Background
I often run long data-crunching jobs and it's useful to know how long they've taken so I can estimate how long it will take for future jobs. For very regular tasks, I go ahead and record this information rigorously using appropriate logging techniques. For less-formal tasks, I'll just prepend the command with time.
It would be nice to automatically time every single interactive command and have the timing information printed in a few characters rather than 3 lines.
This is minimal stand-alone code to achieve what you want:
function timer_start {
timer=${timer:-$SECONDS}
}
function timer_stop {
timer_show=$(($SECONDS - $timer))
unset timer
}
trap 'timer_start' DEBUG
PROMPT_COMMAND=timer_stop
PS1='[last: ${timer_show}s][\w]$ '
Using your replies and some other threads, I wrote this prompt which I want to share with you. I took a screenshot in wich you can see :
White : Last return code
Green and tick mark means success (return code was 0)
Red and cross mark means error (return code was >0)
(Green or Red) : Last command execution time in parenthesis
(Green or Red) : Current date time (\t)
(Green if not root, Red if root) : the logged username
(Green) : the server name
(Blue) : the pwd directory and the usual $
Here is the code to put in your ~/.bashrc file :
function timer_now {
date +%s%N
}
function timer_start {
timer_start=${timer_start:-$(timer_now)}
}
function timer_stop {
local delta_us=$((($(timer_now) - $timer_start) / 1000))
local us=$((delta_us % 1000))
local ms=$(((delta_us / 1000) % 1000))
local s=$(((delta_us / 1000000) % 60))
local m=$(((delta_us / 60000000) % 60))
local h=$((delta_us / 3600000000))
# Goal: always show around 3 digits of accuracy
if ((h > 0)); then timer_show=${h}h${m}m
elif ((m > 0)); then timer_show=${m}m${s}s
elif ((s >= 10)); then timer_show=${s}.$((ms / 100))s
elif ((s > 0)); then timer_show=${s}.$(printf %03d $ms)s
elif ((ms >= 100)); then timer_show=${ms}ms
elif ((ms > 0)); then timer_show=${ms}.$((us / 100))ms
else timer_show=${us}us
fi
unset timer_start
}
set_prompt () {
Last_Command=$? # Must come first!
Blue='\[\e[01;34m\]'
White='\[\e[01;37m\]'
Red='\[\e[01;31m\]'
Green='\[\e[01;32m\]'
Reset='\[\e[00m\]'
FancyX='\342\234\227'
Checkmark='\342\234\223'
# Add a bright white exit status for the last command
PS1="$White\$? "
# If it was successful, print a green check mark. Otherwise, print
# a red X.
if [[ $Last_Command == 0 ]]; then
PS1+="$Green$Checkmark "
else
PS1+="$Red$FancyX "
fi
# Add the ellapsed time and current date
timer_stop
PS1+="($timer_show) \t "
# If root, just print the host in red. Otherwise, print the current user
# and host in green.
if [[ $EUID == 0 ]]; then
PS1+="$Red\\u$Green#\\h "
else
PS1+="$Green\\u#\\h "
fi
# Print the working directory and prompt marker in blue, and reset
# the text color to the default.
PS1+="$Blue\\w \\\$$Reset "
}
trap 'timer_start' DEBUG
PROMPT_COMMAND='set_prompt'
Another very minimal approach is:
trap 'SECONDS=0' DEBUG
export PS1='your_normal_prompt_here ($SECONDS) # '
This shows the number of seconds since the last simple command was started. The counter is not reset if you simply hit Enter without entering a command -- which can be handy when you just want to see how long the terminal has been up since you last did anything in it. It works fine for me in Red Hat and Ubuntu. It did NOT work for me under Cygwin, but I'm not sure if that's a bug or just a limitation of trying to run Bash under Windows.
One possible drawback to this approach is that you keep resetting SECONDS, but if you truly need to preserve SECONDS as the number of seconds since initial shell invocation, you can create your own variable for the PS1 counter instead of using SECONDS directly. Another possible drawback is that a large seconds value such as "999999" might be be better displayed as days+hours+minutes+seconds, but it's easy to add a simple filter such as:
seconds2days() { # convert integer seconds to Ddays,HH:MM:SS
printf "%ddays,%02d:%02d:%02d" $(((($1/60)/60)/24)) \
$(((($1/60)/60)%24)) $((($1/60)%60)) $(($1%60)) |
sed 's/^1days/1day/;s/^0days,\(00:\)*//;s/^0//' ; }
trap 'SECONDS=0' DEBUG
PS1='other_prompt_stuff_here ($(seconds2days $SECONDS)) # '
This translates "999999" into "11days,13:46:39". The sed at the end changes "1days" to "1day", and trims off empty leading values such as "0days,00:". Adjust to taste.
You could utilize this zsh-borrowed hook for bash: http://www.twistedmatrix.com/users/glyph/preexec.bash.txt
Timing done with this hook (Mac OS X): Use Growl to monitor long-running shell commands
If you hadn't set up any of the other answers before you kicked off your long-running job and you just want to know how long the job took, you can do the simple
$ HISTTIMEFORMAT="%s " history 2
and it will reply with something like
654 1278611022 gvn up
655 1278611714 HISTTIMEFORMAT="%s " history 2
and you can then just visually subtract the two timestamps (anybody know how to capture the output of the shell builtin history command?)
I took the answer from Ville Laurikari and improved it using the time command to show sub-second accuracy:
function timer_now {
date +%s%N
}
function timer_start {
timer_start=${timer_start:-$(timer_now)}
}
function timer_stop {
local delta_us=$((($(timer_now) - $timer_start) / 1000))
local us=$((delta_us % 1000))
local ms=$(((delta_us / 1000) % 1000))
local s=$(((delta_us / 1000000) % 60))
local m=$(((delta_us / 60000000) % 60))
local h=$((delta_us / 3600000000))
# Goal: always show around 3 digits of accuracy
if ((h > 0)); then timer_show=${h}h${m}m
elif ((m > 0)); then timer_show=${m}m${s}s
elif ((s >= 10)); then timer_show=${s}.$((ms / 100))s
elif ((s > 0)); then timer_show=${s}.$(printf %03d $ms)s
elif ((ms >= 100)); then timer_show=${ms}ms
elif ((ms > 0)); then timer_show=${ms}.$((us / 100))ms
else timer_show=${us}us
fi
unset timer_start
}
trap 'timer_start' DEBUG
PROMPT_COMMAND=timer_stop
PS1='[last: ${timer_show}][\w]$ '
Of course this requires a process to be started, so it's less efficient, but still fast enough that you wouldn't notice.
I found that trap ... DEBUG was running every time $PROMPT_COMMAND was called, resetting the timer, and therefore always returning 0.
However, I found that history records times, and I tapped into these to get my answer:
HISTTIMEFORMAT='%s '
PROMPT_COMMAND="
START=\$(history 1 | cut -f5 -d' ');
NOW=\$(date +%s);
ELAPSED=\$[NOW-START];
$PROMPT_COMMAND"
PS1="\$ELAPSED $PS1"
It's not perfect though:
If history doesn't register the command (e.g. repeated or ignored commands), the start time will be wrong.
Multi-line commands don't get the date extracted properly from history.
Here's my take on Thomas'
uses date +%s%3N to get milliseconds as base unit,
simplified following code (less zeros)
function t_now {
date +%s%3N
}
function t_start {
t_start=${t_start:-$(t_now)}
}
function t_stop {
local d_ms=$(($(t_now) - $t_start))
local d_s=$((d_ms / 1000))
local ms=$((d_ms % 1000))
local s=$((d_s % 60))
local m=$(((d_s / 60) % 60))
local h=$((d_s / 3600))
if ((h > 0)); then t_show=${h}h${m}m
elif ((m > 0)); then t_show=${m}m${s}s
elif ((s >= 10)); then t_show=${s}.$((ms / 100))s
elif ((s > 0)); then t_show=${s}.$((ms / 10))s
else t_show=${ms}ms
fi
unset t_start
}
set_prompt () {
t_stop
}
trap 't_start' DEBUG
PROMPT_COMMAND='set_prompt'
Then add $t_show to your PS1
Another approach for bash 4.x and above would be to use coproc with PS0 and PS1 like below:
cmd_timer()
{
echo $(( SECONDS - $(head -n1 <&"${CMD_TIMER[0]}") ))
}
coproc CMD_TIMER ( while read; do echo $SECONDS; done )
echo '' >&"${CMD_TIMER[1]}" # For value to be ready on first PS1 expansion
export PS0="\$(echo '' >&${CMD_TIMER[1]})"
export PS1="[ \$(cmd_timer) ] \$"
This is a .bashrc ready snippet.
It is especially useful for everyone that uses undistract-me which overwrites trap DEBUG for its own purposes.
If somone just wants to see the time of execution,
add this line to bash_profile
trap 'printf "t=%s\n" $(date +%T.%3N)' DEBUG
Translated version for zsh.
Append to your ~/.zshrc file
function preexec() {
timer=$(date +%s%3N)
}
function precmd() {
if [ $timer ]; then
local now=$(date +%s%3N)
local d_ms=$(($now-$timer))
local d_s=$((d_ms / 1000))
local ms=$((d_ms % 1000))
local s=$((d_s % 60))
local m=$(((d_s / 60) % 60))
local h=$((d_s / 3600))
if ((h > 0)); then elapsed=${h}h${m}m
elif ((m > 0)); then elapsed=${m}m${s}s
elif ((s >= 10)); then elapsed=${s}.$((ms / 100))s
elif ((s > 0)); then elapsed=${s}.$((ms / 10))s
else elapsed=${ms}ms
fi
export RPROMPT="%F{cyan}${elapsed} %{$reset_color%}"
unset timer
fi
}
A version with split hours, minutes and seconds inspired by the zsh spaceship prompt, based on Ville's answer and this time conversion function by perreal.
I also added a threshold variable so that the timer only displays for long running commands.
time_threshold=5;
function convert_secs {
((h=${1}/3600))
((m=(${1}%3600)/60))
((s=${1}%60))
if [ $h -gt 0 ]; then printf "${h}h "; fi
if [ $h -gt 0 ] || [ $m -gt 0 ]; then printf "${m}m "; fi
if [ $s -gt 0 ]; then printf "${s}s "; fi
}
function timer_start {
timer=${timer:-$SECONDS}
}
function timer_stop {
timer_time=$(($SECONDS - $timer))
if [ ! -z $timer_time ] && [ $timer_time -ge ${time_threshold} ]; then
timer_show="took $(convert_secs $timer_time)"
else
timer_show=""
fi
unset timer
}
trap 'timer_start' DEBUG
PROMPT_COMMAND=timer_stop
PS1='\n\w ${timer_show}\n\\$ '
For the coloured output in my screenshot:
bold=$(tput bold)
reset=$(tput sgr0)
yellow=$(tput setaf 3)
cyan=$(tput setaf 6)
PS1='\n${bold}${cyan}\w ${yellow}${timer_show}${reset}\n\\$ '
Will putting a \t in PS1 work for you?
It does not give the elapsed time but it should be easy enough to subtract the times when necessary.
$ export PS1='[\t] [\w]\$ '
[14:22:30] [/bin]$ sleep 10
[14:22:42] [/bin]$
Following the OP's comment that he is already using \t.
If you can use tcsh instead of bash, you can set the time variable.
/bin 1 > set time = 0
/bin 2 > sleep 10
0.015u 0.046s 0:10.09 0.4% 0+0k 0+0io 2570pf+0w
/bin 3 >
You can change the format of the printing to be less ugly (se the tcsh man page).
/bin 4 > set time = ( 0 "last: %E" )
/bin 5 > sleep 10
last: 0:10.09
/bin 6 >
I do not know of a similar facility in bash
this is my version
use date to format time, only calc days
set terminal title
use \$ in PS1 for user $ + root #
show return code / exit code
use date -u to disable DST
use hidden names like _foo
_x_dt_min=1 # minimum running time to show delta T
function _x_before {
_x_t1=${_x_t1:-$(date -u '+%s.%N')} # float seconds
}
function _x_after {
_x_rc=$? # return code
_x_dt=$(echo $(date -u '+%s.%N') $_x_t1 | awk '{printf "%f", $1 - $2}')
unset _x_t1
#_x_dt=$(echo $_x_dt | awk '{printf "%f", $1 + 86400 * 1001}') # test
# only show dT for long-running commands
# ${f%.*} = int(floor(f))
(( ${_x_dt%.*} >= $_x_dt_min )) && {
_x_dt_d=$((${_x_dt%.*} / 86400))
_x_dt_s='' # init delta T string
(( $_x_dt_d > 0 )) && \
_x_dt_s="${_x_dt_s}${_x_dt_d} days + "
# format time
# %4N = four digits of nsec
_x_dt_s="${_x_dt_s}$(date -u -d0+${_x_dt}sec '+%T.%4N')"
PS1='rc = ${_x_rc}\ndT = ${_x_dt_s}\n\$ '
} || {
PS1='rc = ${_x_rc}\n\$ '
}
# set terminal title to terminal number
printf "\033]0;%s\007" $(tty | sed 's|^/dev/\(pts/\)\?||')
}
trap '_x_before' DEBUG
PROMPT_COMMAND='_x_after'
PS1='\$ '
sample output:
$ sleep 0.5
rc = 0
$ sleep 1
rc = 0
dT = 00:00:01.0040
$ sleep 1001d
rc = 0
dT = 1001 days + 00:00:00.0713
$ false
rc = 1
$