We use Spring Batch for some long running maintenance jobs. Very occasionally a job may get stuck on database/network hiccups. Is there a way email can be sent out upon those occasions, say, if any one step takes more than 2 hours to finish, a group of people will get an email alert?
To send the mail, you can use this class from Spring : org.springframework.mail.javamail.JavaMailSenderImpl
and to check your condition, you can just implement a loop inside a org.springframework.batch.core.StepListener.
This is if u want to receive a mail AFTER a step has finished and took more than 2 hours to finish.
To receive a mail while the step is still running, and has passed the 2 hours limit, it's harder and would require some multithreading development or some external job able to monitor your main job (through org.springframework.batch.core.explore.JobExplorer).
Thanks for the replies. We end up using some other monitoring tools systematically monitoring the batch job's performance and other systems' performances.
Related
I probably have found a similar question (and answer) but I wanted to know if any better alternative is available.
Link to similar question:
http://mqseries.net/phpBB/viewtopic.php?t=72601&sid=f62d9730d61ee2ee2a59986dd79defd1
I want to schedule a particular message flow every 5 seconds (or so). I'm using IIB 10 and it's not associated with MQ. So, Timer nodes are non functional.
I've read about scheduling it with cronjob but again it's getting dependent on the OS which is not my preference. Is there any alternative to the timeout notification node?
Can we use java.util.TimerTask or something similar to to it? Any helping hands please?
I don't know of any solution that does not require a cron job or other external scheduler.
Many organisations use a distributed scheduler like Ctrl-M for a wide range of tasks, and adding a couple of jobs to support the integration layer is not seen as a problem.
You can write you own timer flow using an infinite WHILE loop with SLEEP and PROPAGATE TO TERMINAL functions and sending HTTP requests or configure a "CallableFlow".
I'm creating a mechanism in my web server whereby a scheduled task will execute every 15 minutes and notify users if any activity has occurred within that time frame. It would work as follows:
Annotate a with #Scheduled and schedule to run every 15 minutes
When the task runs, scrape the database for any changes within 15 minutes of the current time
A couple problems I can see:
If I have to restart the server and it's down for longer than 15 minutes, I would need to look back longer than 15 minutes so that no activity is missed.
I m running a number of tomcat servers and only one of them needs to execute the task. Otherwise, duplicate emails will be sent to users.
Has anyone dealt with this before? I'm thinking that this should really be a task external to the web servers... that would solve the issue of duplicate emails being sent, but it wouldn't solve the server bounce issue.
Any ideas on how to solve would be greatly appreciated!
I would have done the following steps to perform the scheduling:
On Application startup query for tasks from database (only those which don't have a dirty flag set to false) and schedule it.
On each run of scheduled task put a dirty flag to suggest the task has run
Because I will be retrieving those tasks only which are marked as dirty, the issue of multiple emails should not occur even on server startup.
I am wondering if there is a way to monitor these automatically. Right now, in our production/QA/Dev environments - we have bunch of services running that are critical to the application. We also have automatic ETLs running on windows task scheduler at a set time of the day. Currently, I have to log into each server and see if all the services are running fine or not, or check event logs for any errors, or check task scheduler to see if ETLs ran well etc etc... I have to do all the manually... I am wondering if there is a tool out there that will do the monitoring for me and send emails only in case something needs attention (like ETLs fail to run, or service get stopped for whatever reason or errors in event log etc). Thanks for the help.
Paessler PRTG Network Monitor can do all that. we have very good experience with it.
http://www.paessler.com/prtg/features
Nagios is the best tool for monitoring. It checks for the server status as well the defined services in it and if any service goes down or system goes down, sends the mail to specified mail id.
Refer the : http://nagios.org/
Thanks for the above information. I looked at the above options but they have a price.. what I did is an inexpensive way to address my concerns..
For my windows task scheduler jobs that run every night - I installed this tool/service from codeplex that is working great.
http://motash.codeplex.com/documentation#CommentsAnchor
For Windows services - I am just setting the "Recovery" Tab in each service "property" with actions to do when it fails. (like restart, reboot, or run a program which could be an email that will notify)
I built a simple tool (https://cronitor.io) for monitoring periodic/scheduled tasks. The name is a play on "cron" from the unix world, but it is system/task agnostic. All you have to do is make an http request to a unique tracking URL whenever your job runs. If your job doesn't check-in according to the rules you define then it will send you an email/sms message.
It also allows you to track the duration of your jobs by making calls at the beginning and end of your task. This can be really useful for long running jobs since you can be alerted if they start taking too long to run. For example, I once had a backup task that was scheduled every hour. About six months after I set it up it started taking longer than an hour to run!
There is https://eyewitness.io - which is for monitoring server cron tasks, queues and websites. It makes sure each of your cron jobs run when they are supposed to, and alerts you if they failed to be run.
We Want to Create an online game like this.I think that ,this type of games have a scheduling software on web server. For Example : Player Click to create a resource And resource creation will be take a moment like 20 Minutes.(Every resource creation time will be different). This message will send to web server application but this message will not processed at same time for example must be processed after 20 Minutes. The web server application after getting the message must be put the order in the Queue.
We have Some big problems :
1- The Jobs must be complete by the web server application Even the player Exit the Game. I think that we must create something like Windows service on Web Server. Can we do it? or Is there a better way?
2- The Second problem depended on problem 1 .Because we will have many Jobs (every player can create 20,30 Jobs in every Loggin and we will have thousands of users) , So Our Scheduling System Must be Work On time . it's possible that , there is 100 , 1000 jobs in a same second , if application Can't Done Jobs in him Second will be use the next second Time and the next second jobs will shift to next second and etc. How We can do for this problem ?
Platform : .Net 3.5 On Windows 2003 Web server
Cheers
Asad Safari
Agile Coach , Scrum Master
On unix, use cron to schedule a script to run every minute - the script then handles all the jobs.
http://en.wikipedia.org/wiki/Cron
Well the specific answer would depend on which technology/language/platform you're using. In java, check the scheduling services provided by Quartz to schedule jobs; and check JMS/MDBs to implement the asynchronous event processing you mention in question 2.
I think my answer won't exactly fit your question but it will solve your problem. Both points at once.
First at all, why do you want to automatize the execution of this events? As far as I understand, you need to give the player access to his resource if and only if he created the resource an especific time ago.
Well, this is my solution:
Insert a field in your resource model class named usable_since or something like that and, when the player create his resource, just set it the value of this field to the current time plus the desired waiting time.
In this way you beat the two problems:
the resource will be usable even if the player gets offline and
you just won't need to set up so big job handler, even with a million users.
I hope it helps. I wish you the best luck in developing the game. Make it fun!
I had a somewhat similar question, as I'm also developing an application which relies on server-side scheduling: Job-Scheduling in Play! Framework. I use the Play Framework (Java) and it works like a charm; except I have no experience in how many jobs can be scheduled concurrently without bringing the server to its knees.
A web application I am developing needs to perform tasks that are too long to be executed during the http request/response cycle. Typically, the user will perform the request, the server will take this request and, among other things, run some scripts to generate data (for example, render images with povray).
Of course, these tasks can take a long time, so the server should not hang for the scripts to complete execution before sending the response to the client. I therefore need to perform the execution of the scripts async, and give the client a "the resource is here, but not ready" and probably tell it a ajax endpoint to poll, so it can retrieve and display the resource when ready.
Now, my question is not relative to the design (although I would very much enjoy any hints on this regard as well). My question is: does a system to solve this issue already exists, so I do not reinvent the square wheel ? If I had to, I would use a process queue manager to submit the task and put a HTTP endpoint to shoot out the status, something like "pending", "aborted", "completed" to the ajax client, but if something similar already exists specifically for this task, I would mostly enjoy it.
I am working in python+django.
Edit: Please note that the main issue here is not how the server and the client must negotiate and exchange information about the status of the task.
The issue is how the server handles the submission and enqueue of very long tasks. In other words, I need a better system than having my server submit scripts on LSF. Not that it would not work, but I think it's a bit too much...
Edit 2: I added a bounty to see if I can get some other answer. I checked pyprocessing, but I cannot perform submission of a job and reconnect to the queue at a later stage.
You should avoid re-inventing the wheel here.
Check out gearman. It has libraries in a lot of languages (including python) and is fairly popular. Not sure if anyone has any out of the box ways to easily connect up django to gearman and ajax calls, but it shouldn't be do complicated to do that part yourself.
The basic idea is that you run the gearman job server (or multiple job servers), have your web request queue up a job (like 'resize_photo') with some arguments (like '{photo_id: 1234}'). You queue this as a background task. You get a handle back. Your ajax request is then going to poll on that handle value until it's marked as complete.
Then you have a worker (or probably many) that is a separate python process connect up to this job server and registers itself for 'resize_photo' jobs, does the work and then marks it as complete.
I also found this blog post that does a pretty good job summarizing it's usage.
You can try two approachs:
To call webserver every n interval and inform a job id; server processes and return some information about current execution of that task
To implement a long running page, sending data every n interval; for client, that HTTP request will "always" be "loading" and it needs to collect new information every time a new data piece is received.
About second option, you can to learn more by reading about Comet; Using ASP.NET, you can do something similiar by implementing System.Web.IHttpAsyncHandler interface.
I don't know of a system that does it, but it would be fairly easy to implement one's own system:
create a database table with jobid, jobparameters, jobresult
jobresult is a string that will hold a pickle of the result
jobparameters is a pickled list of input arguments
when the server starts working on a job, it creates a new row in the table, and spwans a new process to handle that, passing that process the jobid
the task handler process updates the jobresult in the table when it has finished
a webpage (xmlrpc or whatever you are using) contains a method 'getResult(jobid)' that will check the table for a jobresult
if it finds a result, it returns the result, and deletes the row from the table
otherwise it returns an empty list, or None, or your preferred return value to signal that the job is not finished yet
There are a few edge-cases to take care of so an existing framework would clearly be better as you say.
At first You need some separate "worker" service, which will be started separately at powerup and communicated with http-request handlers via some local IPC like UNIX-socket(fast) or database(simple).
During handling request cgi ask from worker state or other data and replay to client.
You can signal that a resource is being "worked on" by replying with a 202 HTTP code: the Client side will have to retry later to get the completed resource. Depending on the case, you might have to issue a "request id" in order to match a request with a response.
Alternatively, you could have a look at existing COMET libraries which might fill your needs more "out of the box". I am not sure if there are any that match your current Django design though.
Probably not a great answer for the python/django solution you are working with, but we use Microsoft Message Queue for things just like this. It basically runs like this
Website updates a database row somewhere with a "Processing" status
Website sends a message to the MSMQ (this is a non blocking call so it returns control back to the website right away)
Windows service (could be any program really) is "watching" the MSMQ and gets the message
Windows service updates the database row with a "Finished" status.
That's the gist of it anyways. It's been quite reliable for us and really straight forward to scale and manage.
-al
Another good option for python and django is Celery.
And if you think that Celery is too heavy for your needs then you might want to look at simple distributed taskqueue.