I setup a cronjob to call myscript.sh every 5 min which then calls a php file between 30 sec and 3 in time and I don't get it why the average Interval is 05:09.
I want to call cron2_.php every 4-8 min but no chance to achieve that.
Tank you.
Cron Job: */5 * * * * myscript.sh
Shell script:
#!/bin/sh
# Grab a random value between 60-180 or ( between 30sec and 3 minutes )
value=$RANDOM
while [ $value -gt 180 ] || [ $value -lt 30 ] ;
do
value=$RANDOM
done
# Sleep for that time.
sleep $value
# Exectue Cron.
echo "Exectued on:$(date)" >> public_html/log_file.txt
exec php -f public_html/cron2_.php
Here is the exectuion time for 2 hours:
Average Interval -> 05:09
Execution Time Interval Min:Sec
13:02:52 00:00
13:07:06 04:14
13:11:35 04:29
13:17:34 05:59
13:21:55 04:21
13:26:54 04:59
13:32:00 05:06
13:35:50 03:50
13:42:44 06:54
13:47:03 04:19
13:51:26 04:23
13:56:48 05:22
14:01:53 05:05
14:07:42 05:49
14:12:15 04:33
14:16:22 04:07
14:23:01 06:39
14:27:17 04:16
14:32:21 05:04
14:35:57 03:36
14:42:14 06:17
14:45:44 03:30
14:52:52 07:08
14:56:50 03:58
15:02:57 06:07
15:06:43 03:46
15:12:26 05:43
15:16:29 04:03
15:22:00 05:31
15:25:35 03:35
15:31:51 06:16
15:37:51 06:00
15:42:56 05:05
15:47:32 04:36
15:50:36 03:04
15:55:45 05:09
16:02:15 06:30
16:06:10 03:55
16:11:11 05:01
16:15:56 04:45
16:21:58 06:02
16:25:56 03:58
16:31:09 05:13
16:37:06 05:57
16:42:30 05:24
16:45:36 03:06
You want your script to run every 4 to 8 minutes. Let's say then that we want, on average, one execution every 6 minutes. In that case, set the crontab line to:
*/6 * * * * myscript.sh
Next, in your script, put a random delay of zero to two minutes:
sleep $(($RANDOM % 120))
Consider two extreme cases. First, suppose that one job waits the maximum 2 minutes and the next waits the minimum of 0 minutes. The time between their executions is 4 minutes. For the second case, consider the opposite: the first job waits the minimum of 0 minutes and the second waits the maximum of 2 minutes. In this case, the time between their executions is 8 minutes. Thus, this approach achieves a wait of 4 to 8 minutes with an average wait of 6 minutes.
Related
I was trying to run multiple srun jobs within a single sbatch script on a cluster. The sbatch script is as follows:
#!/bin/bash
#SBATCH -N 1
#SBATCH -n 1
#SBATCH -c 64
#SBATCH --time=200:00:00
#SBATCH -p amd_256
for i in {0..6} ;
do
cd ${i}
( srun -c 8 ./MD 150 20 300 20 20 0 0 > log.out 2>&1 & )
sleep 20
cd ..
done
cd 7/
srun -c 8 ./MD 100 20 300 20 20 0 0 > log.out 2>&1
cd ..
wait
In this script I submitted multiple srun jobs. One problem with this script is that 0-6th job will be killed after the 7th job is finished. Here is the error message I got for the 0-6th job:
srun: Job step aborted: Waiting up to 62 seconds for job step to finish.
slurmstepd: error: *** STEP 3801214.0 ON j2308 CANCELLED AT 2021-12-22T11:02:22 ***
srun: error: j2308: task 0: Terminated
Any idea on how to fix this?
The line
( srun -c 8 ./MD 150 20 300 20 20 0 0 > log.out 2>&1 & )
creates a subshells and puts them into the background inside the subshell. So the wait-call in the last line doesn't know about those background processes, as they are part of a different shell/process. And since the batch script is now finished, the job will be terminated.
Try this:
( srun -c 8 ./MD 150 20 300 20 20 0 0 > log.out 2>&1 ) &
As an example: Try
( sleep 60 & )
wait
and
( sleep 60 ) &
wait
to see the difference.
I have log files that are broken down into between 1 and 4 "Tasks". In each "Task" there are sections for "WU Name" and "estimated CPU time remaining". Ultimately, I want to the bash script output to look like this 3 Task example;
Task 1 Mini_Protein_binds_COVID-19_boinc_ 0d:7h:44m:28s
Task 2 shapeshift_pair6_msd4X_4_f_e0_161_ 0d:4h:14m:22s
Task 3 rep730_0078_symC_reordered_0002_pr 1d:1h:38m:41s
So far; I can count the Tasks in the log. I can isolate x number of characters I want from the "WU Name". I can convert the "estimated CPU time remaining" in seconds to days:hours:minutes:seconds. And I can output all of that into 'pretty' columns. Problem is that I can only process 1 Task using;
# Initialize counter
counter=1
# Count how many iterations
cnt_wu=`grep -c "WU name:" /mnt/work/sec-conv/bnc-sample3.txt`
# Iterate the loop for cnt-wu times
while [ $counter -le ${cnt_wu} ]
do
core_cnt=$counter
wu=`cat /mnt/work/sec-conv/bnc-sample3.txt | grep -Po 'WU name: \K.*' | cut -c1-34`
sec=`cat /mnt/work/sec-conv/bnc-sample3.txt | grep -Po 'estimated CPU time remaining: \K.*' | cut -f1 -d"."`
dhms=`printf '%dd:%dh:%dm:%ds\n' $(($sec/86400)) $(($sec%86400/3600)) $(($sec%3600/60)) \ $(($sec%60))`
echo "Task ${core_cnt}" $'\t' $wu $'\t' $dhms | column -ts $'\t'
counter=$((counter + 1))
done
Note: /mnt/work/sec-conv/bnc-sample3.txt is a static one Task sample only used for this scripts dev.
What I can't figure out is the next step which is to be able to process x number of multiple Tasks. I can't figure out how to leverage the while/counter combination properly, and can't figure out how to increment through the occurrences of Tasks.
Adding bnc-sample.txt (contains 3 Tasks)
1) -----------
name: Rosetta#home
master URL: https://boinc.bakerlab.org/rosetta/
user_name: XXXXXXX
team_name:
resource share: 100.000000
user_total_credit: 10266.993660
user_expavg_credit: 512.420495
host_total_credit: 10266.993660
host_expavg_credit: 512.603669
nrpc_failures: 0
master_fetch_failures: 0
master fetch pending: no
scheduler RPC pending: no
trickle upload pending: no
attached via Account Manager: no
ended: no
suspended via GUI: no
don't request more work: no
disk usage: 0.000000
last RPC: Wed Jun 10 15:55:29 2020
project files downloaded: 0.000000
GUI URL:
name: Message boards
description: Correspond with other users on the Rosetta#home message boards
URL: https://boinc.bakerlab.org/rosetta/forum_index.php
GUI URL:
name: Your account
description: View your account information
URL: https://boinc.bakerlab.org/rosetta/home.php
GUI URL:
name: Your tasks
description: View the last week or so of computational work
URL: https://boinc.bakerlab.org/rosetta/results.php?userid=XXXXXXX
jobs succeeded: 117
jobs failed: 0
elapsed time: 2892439.609931
cross-project ID: 3538b98e5f16a958a6bdd2XXXXXXXXX
======== Tasks ========
1) -----------
name: shapeshift_pair6_msd4X_4_f_e0_161_X_0001_0001_fragments_abinitio_SAVE_ALL_OUT_946179_730_0
WU name: shapeshift_pair6_msd4X_4_f_e0_161_X_0001_0001_fragments_abinitio_SAVE_ALL_OUT_946179_730
project URL: https://boinc.bakerlab.org/rosetta/
received: Mon Jun 8 09:58:08 2020
report deadline: Thu Jun 11 09:58:08 2020
ready to report: no
state: downloaded
scheduler state: scheduled
active_task_state: EXECUTING
app version num: 420
resources: 1 CPU
estimated CPU time remaining: 26882.771040
slot: 1
PID: 28434
CPU time at last checkpoint: 3925.896000
current CPU time: 4314.761000
fraction done: 0.066570
swap size: 431 MB
working set size: 310 MB
2) -----------
name: rep730_0078_symC_reordered_0002_propagated_0001_0001_0001_A_v9_fold_SAVE_ALL_OUT_946618_54_0
WU name: rep730_0078_symC_reordered_0002_propagated_0001_0001_0001_A_v9_fold_SAVE_ALL_OUT_946618_54
project URL: https://boinc.bakerlab.org/rosetta/
received: Mon Jun 8 09:58:08 2020
report deadline: Thu Jun 11 09:58:08 2020
ready to report: no
state: downloaded
scheduler state: scheduled
active_task_state: EXECUTING
app version num: 420
resources: 1 CPU
estimated CPU time remaining: 26412.937920
slot: 2
PID: 28804
CPU time at last checkpoint: 3829.626000
current CPU time: 3879.975000
fraction done: 0.082884
swap size: 628 MB
working set size: 513 MB
3) -----------
name: Mini_Protein_binds_COVID-19_boinc_site3_2_SAVE_ALL_OUT_IGNORE_THE_REST_0aw6cb3u_944116_2_0
WU name: Mini_Protein_binds_COVID-19_boinc_site3_2_SAVE_ALL_OUT_IGNORE_THE_REST_0aw6cb3u_944116_2
project URL: https://boinc.bakerlab.org/rosetta/
received: Mon Jun 8 09:58:47 2020
report deadline: Thu Jun 11 09:58:46 2020
ready to report: no
state: downloaded
scheduler state: scheduled
active_task_state: EXECUTING
app version num: 420
resources: 1 CPU
estimated CPU time remaining: 27868.559616
slot: 0
PID: 30988
CPU time at last checkpoint: 1265.356000
current CPU time: 1327.603000
fraction done: 0.032342
swap size: 792 MB
working set size: 668 MB
Again, I appreciate any guidance!
I am attempting to use perl to rename files based on the folder they are in and the time created. Files GOPR1521.MP4 and GOPR7754.MP4 were created on two different cameras at the same time and date, and I want to be able their names to indicate that. For example .../GoProTravisL/GOPR1521.mp4 created at 12:32:38 should become 123238L_GOPR1520.mp4, and GOPR7754.MP4 becomes 123239R_GOPR7754.MP4. Right now the only problem is the time stamps. I would think its a problem with being wrong timezone or hour offset, but the minutes are off too. Is there something in perl I am missing when getting time stamps? Below is the perl code, what it outputs for times for each file, and what Finder on OS X says the creation times are.
Code:
#!/usr/bin/perl
use Time::Piece;
use File::stat;
use File::Find;
use File::Basename;
use File::Spec;
#files = <$ARGV[0]/>;
find({ wanted => \&process_file, no_chdir => 1 }, #files);
sub process_file {
my($filename, $dirs, $suffix) = fileparse($_,qr/\.[^.]*/);
if ((-f $_) && ($filename ne "" )) {
#print "\n\nThis is a file: $_";
#print "\nFile: $filename";
#print "\nDIR: $dirs";
my(#parsedirs) = File::Spec->splitdir($dirs);
my #strippeddirs;
foreach my $element ( #parsedirs ) {
push #strippeddirs, $element if defined $element and $element ne '';
}
$pardir = pop(#strippeddirs);
#print "\nParse DIR: ", $pardir;
#print "\nFile creation time: ";
$timestamp = localtime(stat($_)->ctime)->strftime("%H%M%S"); #gives time stamp
print $timestamp;
$newname = $timestamp . substr($pardir,-1) ."_". $filename . $suffix;
print "\nRename: $dirs$filename$suffix to $dirs$newname\n";
#rename ($dirs . $filename . $suffix,$dirs . $newname) || die ( "Error in renaming: " . $! );
} else {
print "\n\nThis is not file: $_\n";
}
}
Output of time stamps for each file:
/Volumes/Scratch/Raw/2016-03-21/GoProTravisL/
File: GOPR1520
File creation time: 05-55-21
File: GOPR1521
File creation time: 05-56-18
File: GOPR1522
File creation time: 05-57-44
File: GOPR1523
File creation time: 05-58-49
File: GP011520
File creation time: 05-59-53
/Volumes/Scratch/Raw/2016-03-21/GoProTravisR
File: GOPR7754
File creation time: 06-02-48
File: GOPR7755
File creation time: 06-04-19
File: GOPR7756
File creation time: 06-06-27
File: GOPR7757
File creation time: 00-06-16
File: GP017754
File creation time: 00-19-30
File: GP027754
File creation time: 00-22-20
Actual file times using ls:
MacTravis:2016-03-21 travis$ ls -lR /Volumes/Scratch/Raw/2016-03-21
total 0
drwxr-xr-x 8 travis admin 272 Apr 9 21:25 GoProTravisL
drwxr-xr-x 9 travis admin 306 Apr 9 21:25 GoProTravisR
/Volumes/Scratch/Raw/2016-03-21/GoProTravisL:
total 21347376
-rw------- 1 travis admin 4001240088 Mar 21 12:04 GOPR1520.MP4
-rw------- 1 travis admin 1447364149 Mar 21 12:31 GOPR1521.MP4
-rw------- 1 travis admin 2140532053 Mar 21 12:45 GOPR1522.MP4
-rw------- 1 travis admin 1649133454 Mar 21 13:00 GOPR1523.MP4
-rw------- 1 travis admin 1691562945 Mar 21 12:21 GP011520.MP4
/Volumes/Scratch/Raw/2016-03-21/GoProTravisR:
total 31941008
-rw------- 1 travis admin 4001129586 Mar 21 12:04 GOPR7754.MP4
-rw------- 1 travis admin 2166255754 Mar 21 12:31 GOPR7755.MP4
-rw------- 1 travis admin 3202301883 Mar 21 12:45 GOPR7756.MP4
-rw------- 1 travis admin 2466803806 Mar 21 12:08 GOPR7757.MP4
-rw------- 1 travis admin 4001257192 Mar 21 11:27 GP017754.MP4
-rw------- 1 travis admin 516025454 Mar 21 11:29 GP027754.MP4
ctime is the "time of last status change", which I believe is the time the inode was last modified. It is NOT the file's creation time[1]. ls lists the file modification time, so simply change from using ctime to using mtime.
Historically, the time at which a file was created wasn't tracked by file systems used on unix file systems. Some newer file systems track it, but I am unsure how to access it (nor is it needed here).
I have a bash script that creates a file with a timestamp as the name. Once some time passes, it is supposed to pick up that file and do something with it. I want it to pick it up after two hours, but for some reason, it is picking it up after 57 minutes (and 6 seconds). Can anyone point me to an error in my logic or assumptions?
Here are the details:
I have a variable set to 2 hours (7200 seconds):
SERVICE_DURATION=${SERVICE_DURATION:-7200} # seconds
I am setting the filename equal to the Unix timestamp concatenated with nanoseconds:
active_name=`date +%s%N`
echo "${1}" >> ${ACTIVE_DIR}/${active_name}
I then loop forever until the time is right:
while true
do
for fa in ${ACTIVE_DIR}/*
do
if [ $(basename ${fa}) -le $(($(date +%s%N) - ${SERVICE_DURATION} * 1000000000)) ]
then
exec 6< "${fa}"
read old_port <&6
read old_host <&6
read old_config <&6
exec 6<&-
logger -p daemon.info "Recycling port ${old_port}."
start_remote_service "${old_port}"
stop_remote_service "${old_port}" "${old_host}" "${old_config}" "${fa}"
sleep 2
fi
done
sleep 30
done
I can't see what is wrong with this. The filename ($(basename ${fa})) shouldn't be less than the current time minus the specified duration in nanoseconds ($(($(date +%s%N) - ${SERVICE_DURATION} * 1000000000))) until the duration has passed.
In order to keep the script from constantly checking, there is a sleep 30 at the end of the loop, so it could be that the time is somewhere between 56:36 and 57:06.
Any help would be appreciated. Thanks.
In my cron tab file, I have
* * * * * /Users/ajgauravdeep/test.sh
which looks like
1 #!/bin/sh
2
3 /bin/echo "Downloading builds"
4 #~luna/bin/mountebuild
5
6 #sleep 10
7
8 ##############---------Variables---------##############
9
10 fileWithBuildPath="/tmp/process.tmp.file.txt"
11 skihillDir="xyz"
12 lastBuild=`/bin/cat $fileWithBuildPath`
13 curBuild=`/usr/bin/readlink -n $skihillDir/x`
14
15 ##############---------Variables---------##############
16
17 /bin/echo lastbuild is $lastBuild
18
19 if [ "$curBuild" != "$lastBuild" ]; then
20 lastBuild=$curBuild
21 /bin/echo We have a new build :$curBuild
22 /bin/rm $fileWithBuildPath
23 /bin/echo "$lastBuild" > $fileWithBuildPath
24 fi
I don't see any output coming every minute on screen but when I have
* * * * * /Users/ajgauravdeep/test.sh > <some file>
I see that file is populated. Can anyone help?
Jobs run by cron are not connected to any terminal, much less your current terminal. You can't expect to a job with cron to write to a terminal.