I want to do a task every 5 mins. I want to control when i can start and when i can end.
One way is to use sleep in a while true loop, another way is to use cronjob. Which one is preferred performance-wise?
Thanks!
cron is almost always the best solution.
If you try to do it yourself with a simple script running in a while loop:
while true; do
task
sleep 300
done
you eventually find that nothing is happening because your task failed due to a transient error. Or the system rebooted. Or some such. Making your script robust enough to deal with all these eventualities is hard work, and unnecessary. That's what cron is for, after all.
Also, if the task takes some non-trivial amount of time, the above simple-minded while loop will slowly shift out of sync with the clock. That could be fixed:
while true; do
task
sleep $((300 - $(date +%s) % 300))
done
Again, it's hardly worth it since cron will do that for you, too. However, cron will not save you from starting the task before the previous invocation finished, if the previous invocation got stuck somehow. So it's not a completely free ride, but it still provides you with some additional robustness.
A simple approach to solving the stuck-task problem is to use the flock utility. For example, you could cron a script containing the following:
(
flock -n 8 || {
logger -p user.warning Task is taking too long
# You might want to kill the stuck task here. See pkill
exit 1
}
# Do the task here
) 8> /tmp/my_task.lck
Use a cron job. Cron is made for this type of use case. It frees you of having to to code the while loop yourself.
However, cron may be unsuitable if the run time of the script is unpredictable and exceeds the timer schedule.
Performance-wise It is hard to tell unless you share what the script does and how often it does it. But generally speaking, neither option should have a negative impact on performance.
Related
Quite simply I have a shell script with some long-running operations that I run in the background and then wait for in a loop (so I can check if they're taking too long, report progress etc.).
However one case that I'd also like to check for is when the system has been put to sleep manually (I've already taken steps to ensure it shouldn't auto-sleep while my script is running).
Currently I do this in a fairly horrible way, namely my script runs sleep in a loop for a few seconds at a time, checking each time if the task is still running. To detect sleep I check if the time elapsed was longer than expected, like so:
start=$(date +%s)
while sleep 5; do
if [ $(($(date +%s) - $start)) -gt 6 ]; then
echo 'System may have been asleep'
start=$(date +%s)
elif kill -0 $PID; then
echo 'Task is still running'
start=$(date +%s)
else
echo 'Task is complete'
break
fi
done
The above is very much simplified so please forgive any mistakes, it's just to give the basic idea; for example, on platforms where the wait command supports timeouts I already use that in place of sleep.
Now, while this mostly works, it's not especially pretty and it's not really detecting sleep, but guessing whether the system might have slept; for example, it can't differentiate cases where a system hangs long enough to confound the check, making the check time longer will help with this, but it's still guesswork.
On macOS I can more reliably check for sleep using pmset -g uuid which returns a new UUID if the system went to sleep. What I would like to know is, are there any alternatives for other platforms?
In essence all I need is a way to find out if the system has been asleep since the last time I checked, though if there's a way to receive a signal or such instead then that may be even better.
While I'm looking to hear of the best options available on various platforms, I crucially need a shell agnostic option as well that I can use as a reliable fallback, as I'd still like the shell script to be as portable as possible.
Im writing a bash script that essentially fires off a python script that takes roughly 10 hours to complete, followed by an R script that checks the outputs of the python script for anything I need to be concerned about. Here is what I have:
ProdRun="python scripts/run_prod.py"
echo "Commencing Production Run"
$ProdRun #Runs python script
wait
DupCompare="R CMD BATCH --no-save ../dupCompareTD.R" #Runs R script
$DupCompare
Now my issues is that often the python script can generate a whole heap of different processes on our linux server depending on its input, with lots of different PIDs AND we have heaps of workers using the same server firing off scripts. As far as I can tell from reading, the 'wait' command must wait for all processes to finish or for a specific PID to finish, but when i cannot tell what or how many PIDs will be assigned/processes run, how exactly do I use it?
EDIT: Thank you to all that helped, here is what caused my dilemma for anyone google searching this. I broke up the ProdRun python script into its individual script that it was itself calling, but still had the issue, I think found that one of these scripts was also calling another smaller script that had a "&" at the end of it that was ignoring any commands to wait on it inside the python script itself. Simply removing this and inserting a line of "os.system()" allowed all the code to run sequentially.
It sounds like you are trying to implement a job scheduler with possibly some complex dependencies between different tasks. I recommend to use a job scheduler instead. It allows you to specify to run those jobs whilst also benefitting from features like monitoring, handling exceptional cases, errors, ...
Examples are: the open source rundeck https://github.com/rundeck/rundeck or the commercial one http://www.bmcsoftware.uk/it-solutions/control-m.html
Make your Python program wait on the children it spawns. That's the proper way to fix this scenario. Then you don't have to wait for Python after it finishes (sic).
(Also, don't put your commands in variables.)
I read this interesting question, that basically says that I should always avoid reaching PID of processes that aren't child processes. It's well explained and makes perfect sense.
BUT, while OP was trying to do something that cron isn't meant to, I'm in a very different situation :
I want to run a process say every 5 minutes, but once in a hundred times it takes a little more than 5 minutes to run (and I can't have two instances running at once).
I don't want to kill or manipulate other processes, I just want to end my process without doing anything if another instance of the process is running.
Is it ok to fetch PID of "not-child processes" in that case ? If so, how would I do it ?
I've tried doing if pgrep "myscript"; then ... or stuff like that, but the process finds its own PID. I need to detect if it finds more than one.
(Initially before being redirected I read this question, but the solution given doesn't work: it can give pid of the process using it)
EDIT: I should have mentioned it before, but if the script is already in use I still need to write something in a log file, at least : date>>script.log; echo "Script already in use">>script.log", I may be wrong but I think flock doesn't allow to do that.
Use lckdo or flock to avoid duplicated running.
DESCRIPTION
lckdo runs a program with a lock held, in order to prevent multiple
processes from running in parallel. Use just like nice or nohup.
Now that util-linux contains a similar command named flock, lckdo is
deprecated, and will be removed from some future version of moreutils.
Of course you can implement this primitive lockfile feature by yourself.
if [ ! -f /tmp/my.lock ];then
touch /tmp/my.lock
run prog
rm -f /tmp/my.lock
fi
I'm running a process that will take, optimistically, several hours, and in the worst case, probably a couple of days.
I've tried a couple of times to run it and it just never seems to complete (I should add, I didn't write the program, it's just a big dataset). I know my syntax for the command is correct as I use it all the time for smaller data and it works properly (I'll spare you the details as it is obscure for SO and I don't think that relevant to the question).
Consequently, I'd like to leave the program unattended running as a fork with &.
Now, I'm not totally sure whether the process is just grinding to a halt or is running but taking much longer than expected.
Is there any way to check the progress of the process other than ps and top + 1 (to check CPU use).
My only other thought was to get the process to output a logfile and periodically check to see if the logfile has grown in size/content.
As a sidebar, is it necessary to also use nohup with a forked command?
I would use screen for this purpose. see the man for more reference
Brief summary how to use:
screen -S some_session_name - starts a new screen session named session_name
Ctrl + a + d - detach session
screen -r some_session_name returns you to your session
The idea
Say I have a few scripts. For example:
script1
script2
script3
I want each script to:
Do something
Run next script
Wait
Cleanup
The wait is simply to wait for the next script to complete.
The problem
A recursive solution is rather straightforward. The problem is that each script then needs to check if there is a next script. This is ok but a minor mistake in a script and it becomes a debugging hell, especially if there are many scripts.
For this reason I was thinking to do it in a linear way. Having a main script (script1) keeping control of everything. The main issue is the wait part.
How do I make script1 to pause script2 until script3 has completed so that it cleans up?
The easiest would be to simply split each worker script in two parts: the real work and the cleanup. Then your master script can run each of the scripts in sequence, followed by each of the cleanup scripts.
Another way to go about this would be to use a "build system" like SCons, which may work well if you can define the inputs and outputs of each script as filenames and let SCons schedule the work and support the "clean" command. This will be a bit of a steep learning curve, but for serious systems where debugging may be needed often, it may be more beneficial.