With SLURM, I run the command
squeue -u enter_username
and I get an table output with the following heading
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
I'm trying to capture the duration of time the job has been running for. I couldn't find an environmental variable provided by SLURM to capture this time, so I think I'm left parsing the output of squeue. This is not as easy as I thought it would be, because the wall clock does not have a fixed format. In other words, it doesn't always show dd-HH:MM:SS. If there are no days, then the output is just HH:MM:SS, and if there are no hours the output is MM:SS, etc.
I'm doing this with bash and I need to capture the day (dd) and hour (HH) and assign each of them to a variable. I'm having a hard time doing this when the format is dynamic.
To capture the time entry, I'm simply doing the following (within a SLURM bash script)
time_str=$(squeue -u enter_username | grep "$SLURM_JOB_ID" | cut -d "R" -f2- | gawk '{print $1}')
As I said before, time_str does not have a fixed format. Hoping someone with experience can help.
From reading the man page of the squeue command, it seems that you can simplify the problem by having squeue only output the information you need:
squeue -h -j ${jobid} -O timeused
Then your task is simply to parse that output, which can be done as follows:
#!/bin/bash
line="-$(squeue -h -j ${jobid} -O timeused)" # Leading '-' aids parsing.
parts=( 0 0 0 0 )
index=3
while [ ${#line} -gt 0 ]; do
parts[${index}]=${line##*[-:]}
line=${line%[-:]*}
((index--))
done
Now the array ${parts[*]} contains exactly 4 elements, 0 to 3, representing days, hours, minutes and seconds respectively.
The solution presented by #Andrew Vickers above works as expected. However, I took this one step further to enforce a fixed 2 digit format
index=0
while [ ${index} -lt 4 ]; do
if [ ${#parts[${index}]} -lt 2 ]; then
parts[${index}]="0${parts[${index}]}"
fi
((index++))
done
The conditional check could be incorporated into his answer, but his loop would need to be adjusted to format all variables.
Related
Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.
Closed 6 years ago.
Edit the question to include desired behavior, a specific problem or error, and the shortest code necessary to reproduce the problem. This will help others answer the question.
This question does not appear to be about a specific programming problem, a software algorithm, or software tools primarily used by programmers. If you believe the question would be on-topic on another Stack Exchange site, you can leave a comment to explain where the question may be able to be answered.
Improve this question
I'd like to monitor the average rate at which lines are being added to a log file in a bash shell.
I can currently monitor how many lines are in the file each second via the command
watch -n 1 'wc -l log.txt'
However, this gives me the total count of lines when I would prefer a rate instead. In other words, I would like a command to every so often output the number of lines that have been added to the file since the command was executed divided by the number of seconds the command has been running.
For a rough count of lines per second, try:
tail -f log.txt | { count=0; old=$(date +%s); while read line; do ((count++)); s=$(date +%s); if [ "$s" -ne "$old" ]; then echo "$count lines per second"; count=0; old=$s; fi; done; }
(Bash required.)
Or, as spread out over multiple lines:
tail -f log.txt | {
count=0
old=$(date +%s)
while read line
do
((count++))
s=$(date +%s)
if [ "$s" -ne "$old" ]
then
echo "$count lines per second"
count=0
old=$s
fi
done
}
This uses date to record the time in seconds. Meanwhile, it counts the number of lines produced by tail -f log.txt. Every time another second passes, the count of lines seen during that second is printed.
Demonstration
One one terminal, run the command:
while sleep 0.1; do echo $((count++)); done >>log.txt
This command writes one line to the file log.txt every roughly tenth of a second.
In another terminal, run:
$ tail -f log.txt | { count=0; old=$(date +%s); while read line; do ((count++)); s=$(date +%s); if [ "$s" -ne "$old" ]; then echo "$count lines per second"; count=0; old=$s; fi; done; }
15 lines per second
10 lines per second
10 lines per second
10 lines per second
9 lines per second
10 lines per second
Due to buffering, the first count is off. Subsequent counts are fairly accurate.
Simple script you can deploy:
Filename="log.txt"
ln_1=`wc -l $Filename | awk '{print $1}'`
while true
do
ln_2=${ln_1}
sleep 1
ln_1=`wc -l $Filename | awk '{print $1}'`
echo $(( ln_1-ln_2 )) lines increased
done
The tail command supports watching for appended lines via --follow option, which accepts a file descriptor, or file name. With this option, tail periodically checks for file changes. The interval of the checks depends on whether kernel supports inotify. Inotify-based implementations detect the changes promptly (I would say, almost instantly). If, however, the kernel doesn't support inotify, tail resorts to periodic checks. In the latter case, tail sleeps for one second by default. The sleep interval can be changed with --sleep-interval option.
I wouldn't rely on the sleep interval in calculations, however:
When ‘tail’ uses inotify, this polling-related option is usually ignored.
Especially because Bash has a built-in seconds counter, the SECONDS variable (see info bash SECONDS):
This variable expands to the number of seconds since the shell was started. Assignment to this variable resets the count to the value assigned, and the expanded value becomes the value assigned plus the number of seconds since the assignment.
Thus, you can initialize SECONDS to 1, run a loop reading the output of tail, and calculate the speed as number_of_lines / $SECONDS. But this will produce average for the entire execution time. Average for the last N seconds is much more practical. It is also easy to implement, as Bash allows to reset the seconds counter.
Example
The following example implements the idea. The also features watch-like output in interactive mode.
# The number of seconds for which we calculate the average speed
timespan=8
# The number of lines
lines=0
# We'll assume that the shell is running in interactive mode,
# if the standard output descriptor (1) is attached to the terminal.
# See http://www.tldp.org/LDP/abs/html/intandnonint.html
if [ -t 1 ]; then
is_interactive=1
format='%d lines/sec'
else
is_interactive=
format='%d lines/sec\n'
fi
# Reset the built-in seconds counter.
# Also, prevent division by zero in the calculation below.
SECONDS=1
# Save cursor position, if interactive
test -n "$is_interactive" && tput sc
while read line; do
if [[ $(( $SECONDS % $timespan )) -eq 0 ]]; then
SECONDS=1
lines=0
fi
if test -n "$is_interactive"; then
# Restore cursor position, then delete line
tput rc; tput el1
fi
printf "$format" $(( ++lines / SECONDS ))
done < <(tail -n0 -F log.txt)
P.S.
There are many other ways to get an offset in seconds. For example, you can fetch the current Unix time using the built-in printf function:
# -1 represents the current time
# %s is strftime's format string for the number of seconds since the Epoch
timestamp=$(builtin printf '%(%s)T' -1)
Another way is to invoke the date command: date +%s.
But I believe that reading from the SECONDS variable is faster and cleaner.
If I have 3 different scripts to run at various times each time a file is written to, how would a bash script be written to run a specific script only at a specific time. This is not as simple as a cron job (though cron could probably swap out the .sh file based on time), I am looking for the time variables.
For instance:
If 9am-11:30 am run scriptA.sh if file.txt is changed.
If 11:30am-5:45pm run scriptB.sh if file is changed.
If 5:46pm-8:59am run scriptC.sh if file is changed.
I asked a similar question but I don't think I was clear enough about the variables I am seeking or how to define them..
The traditional tool for comparing time stamps to determine whether work needs to be performed or not is make. Its default behavior is to calculate a dependency chain for the specified target(s) and determine whether any of the dependent files have changed; if not, the target does not need to be remade. This is a great tool for avoiding recompilation, but it easily extends to other tasks.
In concrete terms, you'd create a flag file (say, .made) and specify it as dependent on your file. Now, if file has changed, .made needs to be recreated, and so make will run the commands you specify to do so. In this scenario, we would run a simple piece of shell script, then touch .made to communicate the latest (successful) run time to future runs.
What remains is for the recipe to run different commands at different times. My approach to that would be a simple case statement. Notice that make interprets dollar signs, so we need to double those dollar signs which should be passed through to the shell.
.made: file
case $$(date +%H:%M) in \
09:* | 10:* | 11:[0-2]? ) \
scriptA.sh ;; \
11:[3-5]? | 1[2-6]:* | 17:[0-3]? | 17:4[0-5]) \
scriptB.sh;; \
17:4[6-9] | 17:5? | 1[89]:* | 2?:* | 0[0-8]:* ) \
scriptC.sh;; \
esac
touch $# # $# expands to the current target
The entire case statement needs to be passed as a single logical line to the shell, so we end up with those pesky backslashes to escape the newlines.
Also notice that make is picky about indentation; each (logical) line in the recipe should be preceded by a literal tab character.
The default behavior of make is to run the first target in the file; this Makefile only contains one target, so make is equivalent to make .made.
Also notice that make cares about exit codes; if scriptA, scriptB, or scriptC could exit with a non-zero exit status, that is regarded by make as a fatal error, and the .made file will not be updated. (You can easily guard against this, though.)
I see there are two issues. 1, how to determine if the current hour is within a particular range. 2, how to determine if a file has been modified, recently.
Here's how I would approach that:
#!/bin/bash
now=$( date +%s )
current_hour=$( date +%H )
file_mod=$( ls -l --time-style=+%s file.txt | awk '{print $(NF-1)}' )
file_age=$(( $now - $file_mod ))
if [[ $current_hour -gt 9 ]] && \
[[ $current_hour -lt 11 ]] && \
[[ $file_age -lt 3600 ]]
then
./scriptA.sh
fi
Here I'm using the bash operators for numeric comparison: -gt, -lt
For greater granularity with time, you would need to compute the amount of time since midnight. Perhaps something like:
hour=$( date +%H )
min=$( date +%M )
minutes=$( echo "( 60 * $hour ) + $min" | bc -l )
eleven_thirty=$( echo "( 60 * 11 ) + 30" | bc -l )
Well, since Bash variables can store string or integer only,
"date" variables are just a string manipulations, like example below:
hours=`date +%H`
minutes=`date +%M`
sum=$(($hours + $minutes))
digit=`echo $sum | sed -e 's/\(^.*\)\(.$\)/\2/'` # counts total digits
#!/bin/bash
z=1
b=$(date)
while [[ $z -eq 1 ]]
do
a=$(date)
if [ "$a" == "$b" ]
then
b=$(date -d "+7 days")
rsync -v -e ssh user#ip_address:~/sample.tgz /home/kartik2
sleep 1d
fi
done
I want to rsync a file every week !! But if I start this script on every boot the file will be rsynced every time the system starts !! How to alter the code to satisfy week basis rsync ? ( PS- I don't want to do this through cronjob - school assignment)
You are talking about having this run for weeks, right? So, we have to take into account that the system will be rebooted and it needs to be run unattended. In short, you need some means of ensuring the script is run at least once every week even when no one is around. The options look like this
Option 1 (worst)
You set a reminder for yourself and you log in every week and run the script. While you may be reliable as a person, this doesn't allow you to go on vacation. Besides, it goes against our principle of "when no one is around".
Option 2 (okay)
You can background the process (./once-a-week.sh &) but this will not reliable over time. Among other things, if the system restarts then your script will not be operating and you won't know.
Option 3 (better)
For this to be reliable over weeks one option is to daemonize the script. For a more detailed discussion on the matter, see: Best way to make a shell script daemon?
You would need to make sure the daemon is started after reboot or system failure. For more discussion on that matter, see: Make daemon start up with Linux
Option 4 (best)
You said no cron but it really is the best option. In particular, it would consume no system resources for the 6 days, 23 hours and 59 minutes when it does not need to running. Additionally, it is naturally resilient to reboots and the like. So, I feel compelled to say that creating a crontab entry like the following would be my top vote: #weekly /full/path/to/script
If you do choose option 2 or 3 above, you will need to make modifications to your script to contain a variable of the week number (date +%V) in which the script last successfully completed its run. The problem is, just having that in memory means that it will not be sustained past reboot.
To make any of the above more resilient, it might be best to create a directory where you can store a file to serve as a semaphore (e.g. week21.txt) or a file to store the state of the last run. Something like once-a-week.state to which you would write a value when run:
date +%V > once-a-week.state # write the week number to a file
Then to read the value, you would:
file="/path/to/once-a-week.state" # the file where the week number is stored
read -d $'\x04' name < "$file"
echo "$name"
You would then need to check to see if the week number matched this present week number and handle the needed action based on match or not.
#!/bin/bash
z=1
b=$(cat f1.txt)
while [[ $z -eq 1 ]]
do
a=$(date +"%d-%m-%y")
if [ "$a" == "$b" ] || [ "$b" == "" ] || [$a -ge $b ]
then
b=$(date +"%d-%m-%y" -d "+7 days")
echo $b > f1.txt
rsync -v -e ssh HOST#ip:~/sample.tgz /home/user
if [ $? -eq 0 ]
then
sleep 1d
fi
fi
done
This code seems to works well and good !! Any changes to it let me know
My script (in bash) aims to do this job:
gets start and stop time from a file, file_A. The time range usually is 3-24 hours.
Based on this time window of [start_time, stop_time] got from file_A,
I need to find specific files among totally 10k log files(and will increase along with the experimental running), each of which recorded around 30 minutes. That's to say, I have to find 6-50 log files among 10k ones.
After confirming the correct log files, I need to print out interesting data.
Step 1) and 3) are OK, I did it already.
Right now, I'm stuck in step 2), Especially in two places:
(a). How to select appropriate files efficiently by their names since the log files named as time. Every log file named as log_201305280650 which means 2013 / May 28 / 06 :50. That's to say, according to the time get from file_A, I need to confirm the corresponding log files by their names which is a hint of time.
(b). Once the files are selected, read the items(like temperature, pressure etc) from this file whose time is inside of time window. Because each file records 30 minutes, which means some of the entry in this file can't satisfy time window.
For instance,
From step 1), My time window is set to [201305280638, 201305290308].
from step 2), I know the log file(log_201305280650) contains the start time of 201305280638. So I need to read all of temperature and pressure for the entries below 201305280638.
the log files name is log_201305280650 (= 2013 / May 28 / 06 :50)
Time temperature pressure ...
201305280628 100, 120 ...
201305280629 100, 120 ...
... ... ...
201305280638 101, 121 ...
201305280639 99, 122 ...
... ... ...
201305280649 101, 119 ...
201305280650 102, 118 ...
My fake script is following.
get time_start from /path/file_A
get time_stop from /path/file_A
for file in /path_to_log_files/*
do
case "$file" in
*)
If [[log file name within time window of (time_start, time_stop)]]; then
loop over this file to get the entry whose time is just within (time_start, time_stop)
read out temperature and pressure etc.
fi
esac
done
Quite a job using bash. Perl or python would have been easier, they both have date/time modules.
I spent a while doing the usual date slicing and it was horrible, so I cheated and used file timestamps instead. Bash has some limited timestamp checking, and this uses that. OK, it does some file IO, but these are empty files and what the hell!
lower=201305280638
upper=201305290308
filename=log_201305280638
filedate=${filename:4}
if (( filedate == upper )) || (( filedate == lower ))
then
echo "$filename within range"
else
# range files
touch -t $lower lower.$$
touch -t $upper upper.$$
# benchmark file
touch -t $filedate file.$$
if [[ file.$$ -nt $upper ]]
then
echo "$filename is too young"
elif [[ file.$$ -ot $lower ]]
then
echo "$filename is too old"
else
echo "$filename is just right"
fi
rm lower.$$ upper.$$ file.$$
fi
-nt is "newer-than"
-ot is "older-than"
Hence the check for equality at the start. You can use a similar check for the timestamps within the file (your second issue). But honestly, can't you use perl or python?
Maybe something along the lines of this would work for you? I am using $start and $end for the start and end times from file_A. I
eval cat log_{$start..$end} 2> /dev/null | sort -k1 | sed -n "/$start/,/$end/p"
This assumes that your log files are in the format
time temperature pressure ...
with no headers or other such text
It may be easier to use awk and the +"%s" option of the date command in stead of literal date and time. This option converts date/time to seconds from epoch (01-01-1970). The resulting number is easy to work with. After all, it's just a number. As an example I made a small bash script. First, a simulation:
#!/bin/bash
#simulation: date and time
start_dt="2013-09-22 00:00:00"
end_dt="2013-09-22 00:00:00"
start_secs=$(date -d "start_dt" +"%s")
end_secs=$(date -d "end_dt" +"%s")
#simulation: set up table (time in secs, temperature, pressure per minute)
> logfile
for ((i=$start_secs;i<$end_secs;i=i+60)); do
echo $i $[90+$[RANDOM %20]] $[80+$[RANDOM %30]] >> logfile
done
Here's the actual script to get the user range and to print it out:
echo "Enter start of range:"
read -p "Date (YYYY-MM-DD): "sdate
read -p "Time (HH:MM:SS) : "stime
echo "Enter end of range:"
read -p "Date (YYYY-MM-DD): "edate
read -p "Time (HH:MM:SS) : "etime
#convert to secs
rstart=$(date -d "$sdate $stime" +"%s")
rend=$(date -d "$edate $etime" +"%s")
#print it to screen
awk -v rstart=$rstart -v rend=$rend '{if($1 >= rstart && $1 <= rend)print $0}' logfile
The awk command is very suited for this. It is fast and can handle large files. I hope this gives you ideas.
So, I want to write a bash script that are a sequence of steps and ill identify it as "task#". However, each step is only completed and can run as long as the user wants.
Do task1
if keypressed stop task1 and move on #this is the part I need help with. There can be up to 10 of these move on steps.
Do task2
...
kina like top; it keeps doing stuff until you hit q to quite, however, i want to move on to the next thing
you can use read builtin command with option -t and -n
while :
do
# TASK 1
date
read -t 1 -n 1 key
if [[ $key = q ]]
then
break
fi
done
# TASK 2
date +%s
kev's great solution works well even in Bash 3.x., but it introduces a 1-second delay (-t 1) in every loop iteration.
In Bash 3.x, the lowest supported value for -t (timeout) is 1 (second), unfortunately.
Bash 4.x supports 0 and fractional values, however:
A solution that supports an arbitrary key such as q requires a nonzero -t value, but you can specify a value very close to 0 to minimize the delay:
#!/bin/bash
# !! BASH 4.x+ ONLY
while :; do
# Loop command
date
# Check for 'q' keypress *waiting very briefly* and exit the loop, if found.
read -t 0.01 -r -s -N 1 && [[ $REPLY == 'q' ]] && break
done
# Post-loop command
date +%s
Caveat: The above uses 0.01 as the almost-no-timeout value, but, depending on your host platform, terminal program and possibly CPU speed/configuration, a larger value may be required / a smaller value may be supported. If the value is too small, you'll see intermittent error setting terminal attributes: Interrupted system call errors - if anyone knows why, do tell us.
Tip of the hat to jarno for his help with the following:
Using -t 0, works as follows, according to help read (emphasis added):
If TIMEOUT is 0, read returns
immediately, without trying to read any data, returning
success only if input is available on the specified
file descriptor.
As of Bash v4.4.12 and 5.0.11, unfortunately, -t 0 seems to ignore -n / -N, so that only an ENTER keypress (or a sequence of keypresses ending in ENTER) causes read to indicate that data is available.
If anyone knows whether this is a bug or whether there's a good reason for this behavior, do let us know.
Therefore, only with ENTER as the quit key is a -t 0 solution currently possible:
#!/bin/bash
# !! BASH 4.x+ ONLY
while :; do
# Loop command
date
# Check for ENTER keypress and, after clearing the input buffer
# with a dummy `read`, exit the loop.
read -t 0 -r -N 1 && { read -r; break; }
done
# Post-loop command
date +%s