Currently, I have a sequence job in DataStage.
Here is the flow:
StartLoop Activity --> UserVariables Activity --> Job Activity --> Execute Command --> Endloop Activity
The job will run every 30 minutes (8 AM - 8 PM) to get real data. The first loop iteration will load data from 8 PM the previous day to 8 AM the current day, and the others will load data that happens in the last 30 minutes.
The UserVariables Activity is to pass variables (SQL statement) to filter data getting in the Job Activity. The first iteration the UserVariables pass variable A (SQL statement 1) to the Job Activity, from the second iteration, it will pass variable B (SQL statement 2) to the Job Activity.
The Execute Command I currently set the 'Sleep 1800' command for the job to sleep 30 minutes to end the iteration of the loop. But I realized now that it is affected by the running time of each iteration. So with my knowing-nothing about shell script, I have searched for solutions and have this file to sleep until a specific time when minute like 30 or 00 (delay 0-1 minute but it's fine).
The shell script is below, I ran it fine on my system but no success on making it as part of the job.
#!/bin/bash
minute=$(date +%M)
num_1=30
num_2=60
if [ $minute -le 30 ];
then
wait=$((($num_1 - $minute)*$num_2))
sleep $wait
fi
if [ $minute -gt 30 ];
then
wait=$((($num_2 - $minute)*$num_2))
sleep $wait
fi
I am now facing 2 problems right now that I need your help with.
The job runs the first iteration fine with the variable A below:
select * from my_table where created_date between trunc(sysdate-1) + 20/24 and trunc(sysdate) + 8/24;
But from the second iteration it failed with the Job Activity with the variable B below:
select * from my_table where created_date between trunc(sysdate-1/48, 'hh') + 30*trunc(to_number(to_char(sysdate-1/48,'MI'))/30)/1440 and trunc(sysdate, 'hh') + 30*trunc(to_number(to_char(sysdate,'MI'))/30)/1440;
In the parallel job, the log said:
INPUT,0: The following SQL statement failed: select * from my_table where created_date between trunc(sysdate-1/48, hh) + 30*trunc(to_number(to_char(sysdate-1/48,MI))/30)/1440 and trunc(sysdate, hh) + 30*trunc(to_number(to_char(sysdate,MI))/30)/1440.
I realized that maybe it failed to run the parallel job because it removed the single quote in hh and MI.
Is it because when passing variables from UserVariables Activity to Job Activity the variable will remove all the quotes? And how can I fix this?
2. How can I make the shell script above as part of the job like Execute Command or some other stage. I have searched for solutions and I think it's about the ExecSH Before/ After Routine Activity. But after reading from IBM pages, I still don't know where to start with it.
Sorry for adding 2 questions at 1 post that makes it so long but it's very relative to each other so it will take lots of time to answer if I separate it into 2 posts and you guys need more information about it.
Thank you!
Try escaping the single quote characters (precede each with a backslash).
Execute the shell script from an Execute Command activity ahead of the Job activity.
I have this worker process in Heroku, which does some cleaning. It runs every two hours and listens to Heroku terminate signals. It works fine, but I'm seeing 100% dyno load all the time.
My question is: How to run this kind of worker process in Heroku without 100% dyno load? The loop causes the dyno load, but what to use instead of the infinite loop?
# Scheduler here
cleanup = Rufus::Scheduler.new
cleanup.cron '* */2 * * *' do
do_some_cleaning
end
# Signal trapping
Signal.trap("TERM") {
terminate = true
shut_down
exit 0
}
# Infinite loop
while terminate == false
end
It's because you're doing an infinite loop with no sleep cycles. This means you're basically telling the CPU that every single cycle you should be immediately executing a loop condition.
This will quickly use up your CPU.
Instead, try throwing a sleep statement into your infinite loop -- this will pause execution and bring your usage down to 0% =)
while terminate == false
sleep 1
end
I should have thought about it sooner. You can actually simply join rufus-scheduler's loop:
cleanup_scheduler = Rufus::Scheduler.new
cleanup_scheduler.cron '* */2 * * *' do
do_some_cleaning
end
Signal.trap('TERM') do
shut_down
exit 0
end
cleanup_scheduler.join
That joins rufus-scheduler scheduling thread and is pseudo equivalent to:
while !terminated
sleep 0.3
trigger_schedules_if_any
end
Basically, I have a series of commands I want to run every other sunday. I set a cron task to run the script every sunday, then this script only allows the script to run on even weeks, thus it only runs every other sunday. My question is, will this script still work going from year to year.
if [ $(($(date +'%U') % 2)) -eq 0 ]
then
some command
fi
You have what's known as the XY problem here.
You have a problem with this part of your shell script, and you want to solve the problem by fixing the script. In reality, fixing the root cause of the problem is easier.
Simply alter your cron job to run every other Sunday:
#----+-----+-----+-----+-----+-------------------------------------------------
#min |hour |day |month|day |command
# | |of mn| |of wk|
#----+-----+-----+-----+-----+-------------------------------------------------
03 04 * * 7 expr `date +%W` % 2 >/dev/null || fortnightly.sh
See How to instruct cron to execute a job every second week? for more info.
If you don't want to specify this with cron syntax, you can use the %s format instead of %U. This will give you the number of seconds since 1st Jan 1970 UTC. You can divide this to get a week number:
$(($(date +'%s') / 604800))
Then you can do your modulo test on that.
Note the number 604800 = 7 * 86400 = 7 * 24 * 60 * 60 ie the number of seconds in one week.
If you're running this every day, you'll want to know that it's actually a Sunday. So in this case, you would divide by 86400 to get a day number. Then, armed with the knowledge that day zero was a Thursday, you can check that the result (modulo 14) is either 3 or 10, depending on which Sunday you started at.
I want to set a cronjob entry that runs a script every 30 minutes from 9:00 to 18:00 but I do not want it to run at 18:30. The script should run for the first time at 9:00 and for the last time at 18:00. Is this possible?
Yes, it is possible; cron itself can't solve the task, but it is possible using additional shell command:
*/30 9-18 * * * root [ $(date +%H%M) = 1830 ] || your_command
your_command will be executed if and only if the current time is not equal to 18:30
You might need to have two entries:
0,30 9-17 * * * /script
0 18 * * * /script
Alternatively, you could modify your script to check if it's close to 18:30 or not, and exit early if so.
I am using Bash on RedHat. I need to schedule a cron job to run at at 9:00 AM on first Sunday of every month. How can I do this?
You can put something like this in the crontab file:
00 09 * * 7 [ $(date +\%d) -le 07 ] && /run/your/script
The date +%d gives you the number of the current day, and then you can check if the day is less than or equal to 7. If it is, run your command.
If you run this script only on Sundays, it should mean that it runs only on the first Sunday of the month.
Remember that in the crontab file, the formatting options for the date command should be escaped.
It's worth noting that what looks like the most obvious approach to this problem does not work.
You might think that you could just write a crontab entry that specifies the day-of-week as 0 (for Sunday) and the day-of-month as 1-7, like this...
# This does NOT work.
0 9 1-7 * 0 /path/to/your/script
... but, due to an eccentricity of how Cron handles crontab lines with both a day-of-week and day-of-month specified, this won't work, and will in fact run on the 1st, 2nd, 3rd, 4th, 5th, 6th, and 7th of the month (regardless of what day of the week they are) and on every Sunday of the month.
This is why you see the recommendation of using a [ ... ] check with date to set up a rule like this - either specifying the day-of-week in the crontab and using [ and date to check that the day-of-month is <=7 before running the script, as shown in the accepted answer, or specifying the day-of-month range in the crontab and using [ and date to check the day-of-week before running, like this:
# This DOES work.
0 9 1-7 * * [ $(date +\%u) = 7 ] && /path/to/your/script
Some best practices to keep in mind if you'd like to ensure that your crontab line will work regardless of what OS you're using it on:
Use =, not ==, for the comparison. It's more portable, since not all shells use an implementation of [ that supports the == operator.
Use the %u specifier to date to get the day-of-week as a number, not the %a operator, because %a gives different results depending upon the locale date is being run in.
Just use date, not /bin/date or /usr/bin/date, since the date utility has different locations on different systems.
You need to combine two approaches:
a) Use cron to run a job every Sunday at 9:00am.
00 09 * * 7 /usr/local/bin/once_a_week
b) At the beginning of once_a_week, compute the date and extract the day of the month via shell, Python, C/C++, ... and test that is within 1 to 7, inclusive. If so, execute the real script; if not, exit silently.
A hacky solution: have your cron job run every Sunday, but have your script check the date as it starts, and exit immediately if the day of the month is > 7...
This also works with names of the weekdays:
0 0 1-7 * * [ "$(date '+\%a')" == "Sun" ] && /usr/local/bin/urscript.sh
But,
[ "$(date '+\%a')" == "Sun" ] && echo SUNDAY
will FAIL on comandline due to special treatment of "%" in crontab (also valid for https://stackoverflow.com/a/3242169/2919695)
Run a cron task 1st monday, 3rd tuesday, last sunday, anything..
http://xr09.github.io/cron-last-sunday/
Just put the run-if-today script in the path and use it with cron.
30 6 * * 6 root run-if-today 1 Sat && /root/myfirstsaturdaybackup.sh
The run-if-today script will only return 0 (bash value for True) if it's the right date.
EDIT:
Now with simpler interface, just one parameter for week number.
# run every first saturday
30 6 * * 6 root run-if-today 1 && /root/myfirstsaturdaybackup.sh
# run every last sunday
30 6 * * 7 root run-if-today L && /root/lastsunday.sh
There is a hacky way to do this with a classic (Vixie, Debian) cron:
0 9 1-7 * */7
The day-of-week field starts with a star (*), and so cron considers it "unrestricted" and uses the AND logic between the day-of-month and the day-of-week fields.
*/7 means "every 7 days starting from weekday 0 (Sunday)". Effectively, this means "every Sunday".
Here's my article with more details: Schedule Cronjob for the First Monday of Every Month, the Funky Way
Note – it's a hack. If you use this expression, make sure to document it to avoid confusion later.
maybe use cron.hourly to call another script. That script will then check to see if it's the first sunday of the month and 9am, and if so, run your program. Sounds optimal enough to me :-).
If you don't want cron to run your job everyday or every Sunday you could write a wrapper that will run your code, determine the next first Sunday, and schedule itself to run on that date.
Then schedule that wrapper for the next first Sunday of the month. After that it will handle everything itself.
The code would be something like (emphasis on something...no error checking done):
#! /bin/bash
#We run your code first
/path/to/your/code
#now we find the next day we want to run
nskip=28 #the number of days we want to check into the future
curr_month=`date +"%m"`
new_month=`date --date='$nskip days' +"%m"`
if [[ curr_month = new_month ]]
then
((nskip+=7))
fi
date=`date --date='$nskip days' +"09:00AM %D` #you may need to change the format if you use another scheduler
#schedule the job using "at"
at -m $date < /path/to/wrapper/code
The logic is simple to find the next first Sunday. Since we start on the first Sunday of the current month, adding 28 will either put us on the last Sunday of the current month or the first Sunday of the next month. If it is the current month, we increment to the next Sunday (which will be in the first week of the next month).
And I used "at". I don't know if that is cheating. The main idea though is finding the next first Sunday. You can substitute whatever scheduler you want after that, since you know the date and time you want to run the job (a different scheduler may need a different syntax for the date, though).
try the following
0 15 10 ? * 1#1
http://www.quartz-scheduler.org/documentation/quartz-1.x/tutorials/crontrigger
00 09 1-7 * 0 /usr/local/bin/once_a_week
every sunday of first 7 days of the month