Please let me know the answers to these questions on Tivoli Workload Scheduler.
How could TWS warn the admins when the job didn’t kick off as scheduled?
How could TWS warn the admins if the job is running longer than expected?
How could TWS send email about the completion of the job once it is finished without errors?
Starting from TWS 8.4 you can define an event rule with two correlated events. In your case, these two events need to be defined of type Job Status Changed and with the property Status set to "Running". It needs also to define in this rule a timeout of X seconds/minutes/hours and associate the timeout to a SendEmail action. Then when the timeout expires and the status remains the unchanged (running) the email is sent automatically. Hope this helps
It would depend on what version of TWS you are running. But if you are talking about versions > 8.3 all these alerts and notifications can be set up in the Tivoli Dynamic Workload Console (TDWC).
These new versions of TWS introduce the new concept of 'even-driven' scheduling, where you can set events and triggers for each job, in which you could include an email being sent or forward the event to an event management system.
Related
I've been thinking about how to design a system that supports user created scheduled alerts. My problem is once the alerts are created and inserted into a database, I don't know what the best way to go about scheduling those alerts. Polling the database to see which alerts need to go out next doesn't seem entirely right to me.
What are some ways this could be handled on a scale where say a million users could create their own custom alerts like change baby diaper at 3pm everyday?
This problem is very suitable for cloud platforms. For example, you could use GCP Cloud Scheduler to invoke a cloud function when the alert is supposed to be sent out. The cloud function then calls some API to alert the user.
If cloud platforms are not an option, you could have your application spawn a new thread when an alert is created, and sleep that thread for a certain duration. When it wakes up, it sends the alert. Less elegant and less scalable than the first solution, but it would still work.
My question is a combination of this and this question on stackoverflow, however, answers there don't help me. I want to know that when in a Mesos cluster a task corresponding to a framework finishes how is the framework scheduler informed of this. The more details (like who initiates the communication, is there a lag?, what all information is included in the message etc..) you can provide better it will be for me. I was not able to find my answer even in the Mesos docs.
Frameworks are notified about tasks with Update event
Update Sent by the master whenever there is a status update that is generated by the executor, agent or master. Status updates should be used by executors to reliably communicate the status of the tasks that they manage. It is crucial that a terminal update (e.g., TASK_FINISHED, TASK_KILLED, TASK_FAILED) is sent by the executor as soon as the task terminates, in order for Mesos to release the resources allocated to the task. It is also the responsibility of the scheduler to explicitly acknowledge the receipt of status updates that are reliably retried. See ACKNOWLEDGE in the Calls section above for the semantics. Note that uuid and data are raw bytes encoded in Base64.
All communication (in V1 API) is imitated by a framework. Framework is calling subscribe method and keep connection open to revive updates. Basically when task is done communication looks like this: Task → Executor → Agent → Master → Framework
I want to inform the seller, that the buyer is coming soon (about 2 hours before pickup time) via mail.
I would normally do it the hard way with CRON and a database table. Checking hourly if I find an order with pickup time minus 2 hours, only then sending the mail out.
Now, I would like to know if you would recommend using Queueing Jobs for sending Mails out.
With
$when = now()->addDays(10); //I would dynamically set the date
Mail::to($order->seller())
->later($when, new BuyerIsComing($order));
I can delay the delivery of a queued email message.
But how safe would this be? Especially, if someone is ordering something but is picking it up in let us exaggerate two months?
Is the Laravel queueing system rigid enough to behave correctly after long delays (i.e. 2 months)?
Edit
I'm using Redis for Queueing
You actually have nothing to worry about. Sending mail usually increases the response time of your application, so it's a good thing you want to delay the sending.
Queues are the way to go and it's pretty easy to setup in Laravel. Laravel supports a couple of them out of the box. I would advise you start with database and then try beanstalk etc.
Lastly and somehow more importantly, use a process manager like Supervisor to monitor and maintain your queue workers...
Take a look at https://laravel.com/docs/5.7/queues for more insight.
Cheers.
If by safe, you mean reliable, then it would be little different than sending an email immediately. If there's ever a possibility that your server "hiccups" and doesn't send an email, that possibility would be the same now as 10 minutes from now. Once the job is in the queue, it is persisted until completion (unless you use a memory-based driver, like Redis, which could get reset if the server reboots).
If you are using a database queue driver or remote, the log of queued jobs will remain even if the server is unavailable for a short period of time. Your queue will be honored even if the exact time stamp for when you want to send the job has expired. For instance if you schedule to send an email at 1:00pm but your server is down at that exact moment, when it comes back online it will still see the job because it is stored as incomplete and the time for the job is in the past, which will trigger the execution of the job at the next time your queue worker checks the job list.
Of course, this assumes that you have your queue worker set up to always check jobs and automatically restart, even after a server failure, but that's a different discussion with lots of solutions...such as those shown here.
If you're using database driver with Laravel queues to process your email then you don't need to worry about anything.
Jobs are only removed from Jobs table if they are successfully completed otherwise their next attempt time is set which is few minutes in future and they are executed again (if your queue worker is online).
So its completely safe to use Laravel queues
I have following problem to solve:
Multiple Users can submit Jobs to a Queue via a Web Interface.
This Jobs are then stored in the Database via the database Queue Driver.
Now my problem is: I want the Queue to run all Jobs until inside a job I say something like $queue->pause() because for the next Job to run I need some confirmation from the User.
How would I do something like this?
run jobs
inside one job the job determines that it needs some confirmation from the user
halt the queue and keep that job which needs the confirmation in the queue
any user on the website can press a button which then would delete this confirmation job and start the queue again.
My current "solution" which didn't work was this:
create 2 different job types:
ImageProcessingJob
UserNotificaitonJob
The queue worked all ImageProcessingJobs until it hit a UserNotificationJob.
Inside the UserNotificationJob->handle() I called Artisan::call("queue:restart"); which stopped the Queue.
The problem with this solution is: The UserNotificationJob also got deleted. So if I would start the Queue again the Queue would immediately start with the remainig ImageProcessingJobs without waiting for the actual confirmation.
I'm also open to other architectural solutions without a Queue system.
One approach which avoids pausing the queue, is to have the UserNotificationJob wait on a SyncEvent (the SyncEvent is set when then confirmation comes back from the user). You can have this wait timing-out if you like, but then you need to repost the job to the queue. If you decide to timeout and repost, you can use job chaining to setup dependencies between jobs so that nothing can be run until the UserNotificationJob competes.
Another approach might be to simply avoid posting the remaining jobs until the confirmation is sent from the user.
I has some strange behaviour on production deployment for azure queue messages:
Some of the messages in queues appears with big delay - minutes, and sometimes 10 minutes.
Befere you ask about setting delayTimeout when we put message to queue - we do not set delayTimeout for that message, so message should appear almost immedeatly after it was placed in queue.
At that moments we do not have a big load. So my instances has no work load, and able to process message fast, but they just don't appear.
Our service process millions of messages per month, we able to identify that 10-50 messages processed with very big delay, by that we fail SLA in front of our customers.
Does anyone have any idea what can be reason?
How to overcome?
Did anyone faced similar issues?
Some general ideas for troubleshooting:
Are you certain that the message was queued up for processing - ie the queue.addmessage operation returned successfully and then you are waiting 10 minutes - meaning you can rule out any client side retry policies etc as being the cause of the problem.
Is there any chance that the time calculation could be subject to some kind of clock skew problems. eg - if one of the worker roles pulling messages has its close out of sync with the other worker roles you could see this.
Is it possible that in the situations where the message is appearing to be delayed that a worker role responsible for pulling the messages is actually failing or crashing. If the client calls GetMessage but does not respond with an appropriate acknowledgement within the time specified by the invisibilityTimeout setting then the message will become visible again as the Queue Service assumes the client did not process the message. You could tell if this was a contributing factor by looking at the dequeue count on these messages that are taking longer. More information can be found here: http://msdn.microsoft.com/en-us/library/dd179474.aspx.
Is it possible that the number of workers you have pulling items from the queue is insufficient at certain times of the day and the delays are simply caused by the queue being populated faster than you can pull messages from the queue.
Have you enabled logging for queues and then looked to see if you can find the specific operations (look at e2elatency and serverlatency).
http://blogs.msdn.com/b/windowsazurestorage/archive/tags/analytics+2d00+logging+_2600_amp_3b00_+metrics/. You should also enable client logging and try to determine if the client is having connectivity problems and the retry logic is possibly kicking in.
And finally if none of these appear to help can you please send me the server logs (and ideally the client side logs as well) along with your account information (no passwords) to JAHOGG at Microsoft dot com.
Jason
Azure Service bus has a property in the BrokeredMessage class called ScheduledEnqueueTimeUtc, it allows you to set a time for when the message is added to the queue (effectively creating a delay).
Are you sure that in your code your not setting this property, and this might be the cause for the delay?
You can find more info on this at this url: https://www.amido.com/azure-service-bus-how-to-delay-a-message-being-sent-to-the-queue/
If you are using WebJobs to process messages from the queue, it can be due to WebJobs configuration.
From an MSDN forum post by pranav rastogi:
Starting with 0.4.0-beta, the (WebJobs) SDK implements a random exponential back-off algorithm. As a result of this if there are no messages on the queue, the SDK will back off and start polling less frequently.
The following setting allows you to configure this behavior.
MaxPollingInterval for when a queue remains empty, the longest period of time to wait before checking for a message to. Default is 10min.
static void Main()
{
JobHostConfiguration config = new JobHostConfiguration();
config.Queues.MaxPollingInterval = TimeSpan.FromMinutes(1);
JobHost host = new JobHost(config);
host.RunAndBlock();
}