Slurm wait option: show time waiting - time

Question

When you apply the --wait flag in a slurm script, is it possible to display how long it has been waiting in real time?

Answer 1

When sbatch is used with the --wait option, the command does not exit until the submitted job terminates.
There is no additional option available to show the pending time.
However, you can open another session and execute the following command to display the pending time (in seconds) if the job is still in pending state:
squeue --Format=PendingTime -j <jobid> --noheader
One time display
If you simply wish to know the elapsed time before the job was scheduled, you can add the following line in your batch script:
echo "waited: $(squeue --Format=PendingTime -j $SLURM_JOB_ID --noheader | tr -d ' ')s"
Note: the tr command is used here to delete the trailing spaces added by squeue
Real time counter
If you would like to display the elapsed time in real time you can remove the --wait option and use a sbatch-wrapper such as:
#!/bin/sh
# Time before issuing another squeue command
# XXX: Ensure this is large enough to avoid flooding the Slurm controller
WAIT=20
# Convert seconds to days:hours:minutes:seconds format
seconds_to_days()
{
printf '%dd:%dh:%dm:%ds\n' $(($1/86400)) $(($1%86400/3600)) $(($1%3600/60)) $(($1%60))
}
# Convert days-hours:minutes:seconds time format to seconds
squeue_time_to_seconds()
{
local time=$(echo $1 | tr -d ' ') # Removing spaces
# Print input and return if the time format is not recongized
echo $time | grep -q ':' ||
{
printf "$time"
return
}
# Check if time contains hours, otherwise add 0 hour
[ $(echo $time | awk -F: '{print NF-1}') -eq 2 ] || time="0:$time"
# Check if time contains days, otherwise add 0 day
echo $time | grep -q '-' || time="0-$time"
# Parse and convert to seconds
echo $time | tr '-' ':' |
awk -F: '{ print ($1 * 86400) + ($2 * 3600) + ($3 * 60) + $4 }'
}
# Poll job counter with squeue
squeue_polling()
{
local counter=$1
local counter_description=$2
local jobid=$3
local prev_time="-${WAIT}"
while true; do
elapsed_time=$(squeue --Format=$counter -j $jobid --noheader || exit $?)
elapsed_time=$(squeue_time_to_seconds "$elapsed_time")
# Return in case no counter is found
if [ -z "$elapsed_time" ]; then
echo; return
fi
# Update one more time the counter if it is not progressing anymore
if [ "$elapsed_time" -lt "$((prev_time + WAIT ))" ]; then
printf "\33[2K\r$counter_description: $(seconds_to_days $prev_time)\n"
return
fi
# Update the counter without calling squeue to release the pressure on
# the Slurm controller
for i in $(seq 1 $WAIT); do
printf "\33[2K\r$counter_description: $(seconds_to_days $(($elapsed_time + i)))"
sleep 1
done
prev_time=$elapsed_time
done
}
# Execute sbatch and display the output
OUTPUT=$(sbatch $#)
echo $OUTPUT
# Exit on error
if [ $? -ne 0 ]; then
exit $?
fi
# Parse the job ID
JOBID=$(echo $OUTPUT | sed -rn 's/Submitted batch job ([0-9]+)/\1/p')
# Display pending time until the job is scheduled
squeue_polling 'PendingTime' 'Pending time' $JOBID
# Display the time used by the allocation until the job is over
squeue_polling 'TimeUsed' 'Allocation time' $JOBID
It will act as if you submitted the job with the --wait flag (i.e. will return when the job completed). The pending time will be updated in real time
./sbatch-wait <options> <batch script>
Submitted batch job 42
Pending time: 0d:0h:1m:0s
Allocation time: 0d:0h:1m:23s

Answer 2

An easy way is to (ab)use the pv command like this:
sbatch --wait ... | pv -t
It will look like this:
$ sbatch --wait --wrap "sleep 30" | pv -t
Submitted batch job 123456
0:00:42
and the stopwatch will stop when the job is completed