I am wondering if there is a way to monitor these automatically. Right now, in our production/QA/Dev environments - we have bunch of services running that are critical to the application. We also have automatic ETLs running on windows task scheduler at a set time of the day. Currently, I have to log into each server and see if all the services are running fine or not, or check event logs for any errors, or check task scheduler to see if ETLs ran well etc etc... I have to do all the manually... I am wondering if there is a tool out there that will do the monitoring for me and send emails only in case something needs attention (like ETLs fail to run, or service get stopped for whatever reason or errors in event log etc). Thanks for the help.
Paessler PRTG Network Monitor can do all that. we have very good experience with it.
http://www.paessler.com/prtg/features
Nagios is the best tool for monitoring. It checks for the server status as well the defined services in it and if any service goes down or system goes down, sends the mail to specified mail id.
Refer the : http://nagios.org/
Thanks for the above information. I looked at the above options but they have a price.. what I did is an inexpensive way to address my concerns..
For my windows task scheduler jobs that run every night - I installed this tool/service from codeplex that is working great.
http://motash.codeplex.com/documentation#CommentsAnchor
For Windows services - I am just setting the "Recovery" Tab in each service "property" with actions to do when it fails. (like restart, reboot, or run a program which could be an email that will notify)
I built a simple tool (https://cronitor.io) for monitoring periodic/scheduled tasks. The name is a play on "cron" from the unix world, but it is system/task agnostic. All you have to do is make an http request to a unique tracking URL whenever your job runs. If your job doesn't check-in according to the rules you define then it will send you an email/sms message.
It also allows you to track the duration of your jobs by making calls at the beginning and end of your task. This can be really useful for long running jobs since you can be alerted if they start taking too long to run. For example, I once had a backup task that was scheduled every hour. About six months after I set it up it started taking longer than an hour to run!
There is https://eyewitness.io - which is for monitoring server cron tasks, queues and websites. It makes sure each of your cron jobs run when they are supposed to, and alerts you if they failed to be run.
Related
Is it possible to create a script that is always running on my VPS server? And what need i to do to run it the hole time? (I haven't yet a VPS server, but if this is possible i wants to buy one!
Yes you can, there are many methods to get your expected result.
Supervisord
Supervisord is a process control system that keeps any process running. It automatically start or restart your process whenever necessary.
When to use it: Use it when you need a process that run continuously, eg.:
A queue worker that reads a database continuously waiting for a job to run.
A node application that acts like a daemon
Cron
Cron allow you running processes regularly, in time intervals. You can for example run a process every 1 minute, or every 30 minutes, or any time interval you need.
When to use it: Use it when your process is not long running, it do a task and end, and you do not need it beign restarted automatically like on Supervisord, eg.:
A task that collects logs everyday and send it on a gzip by email
A backup routine.
Whatever you choose, there are many tutorials on the internet on how configuring both, so I'll not go into this details.
I have a Laravel application where the Application servers are behind a Load Balancer. On these Application servers, I have cron jobs running, some of which should only be run once (or run on one instance).
I did some research and found that people seem to favor a lock-system, where you keep all the cron jobs active on each application box, and when one goes to process a job, you create some sort of lock so the others know not to process the same job.
I was wondering if anyone had more details on this procedure in regards to AWS, or if there's a better solution for this problem?
You can build distributed locking mechanisms on AWS using DynamoDB with strongly consistent reads. You can also do something similar using Redis (ElastiCache).
Alternatively, you could use Lambda scheduled events to send a request to your load balancer on a cron schedule. Since only one back-end server would receive the request that server could execute the cron job.
These solutions tend to break when your autoscaling group experiences a scale-in event and the server processing the task gets deleted. I prefer to have a small server, like a t2.nano, that isn't part of the cluster and schedule cron jobs on that.
Check out this package for Laravel implementation of the lock system (DB implementation):
https://packagist.org/packages/jdavidbakr/multi-server-event
Also, this pull request solves this problem using the lock system (cache implementation):
https://github.com/laravel/framework/pull/10965
If you need to run stuff only once globally (so not once on every server) and 'lock' the thing that needs to be run, I highly recommend using AWS SQS because it offers exactly that: run a cron to fetch a ticket. If you get one, parse it. Otherwise, do nothing. So all crons are active on all machines, but tickets are 'in flight' when some machine requests a ticket and that specific ticket cannot be requested by another machine.
I would like to run a batch application (that is a short lived process that should not be restarted) on Pivotal CloudFoundry.
I am not sure how to do that. My current batch app is restarted repeatedly by Pivotal CF.
It seems there's a new CF primitive called a task - as opposed to a long-running process. Tasks are supposed to be available on CF 1.7 (see https://stackoverflow.com/a/35512113/536299).
I was neither able to find relevant information in the CF documentation nor to figure out which version of the Pivotal CF is currently being run...
Can someone please help?
I just got relevant information regarding short-lived/one-off processes on CF. It currently seems to be very difficult to run short-lived/one-off processes on CF.
This will change when CF v3's tasks becomes generally available.
Here is the information I was given:
Batch jobs are a little tricky on PWS and PCF because at the moment
the platform expects your application to continue running forever.
Even if the app exits successfully, the platform considers it to have
crashed and will restart it. There is support in v3 of the platform
for one-off tasks like batch jobs, so this will get easier in the
future. For now, what you need to do is to make the app run forever.
One option is to add a loop to the main method in the app, the loop
would essentially run the batch job, pause for some set amount of time
and repeat indefinitely.
So bottom-line is wait for CF v3's tasks.
See here for documentation about tasks: http://v3-apidocs.cloudfoundry.org/version/release-candidate/index.html#tasks
is there any way i can make my records in the database to be automatic. e.g i want a message to be sent to helpdesk if a requested service is not attended within 24 hours, without clicking anything.
technically it depends on the database you are using. if the database supports it, you could set up a scheduled job to scan the records and identify late services and email the helpdesk.
if the database doesn't support scheduled tasks then you could set up a client job on a timer to do the same thing.
This is what application software is for.
When the application saves to the database, the application also sends an email.
The traditional approach to this is to schedule a job (there are too many ways[1] to do that for me to go into details without knowing your server operating system, DBMS, and how much control you have to install or schedule programs on the server).
Your scheduled job would regularly check the database for records that have not been attended, and then take the appropriate action such as emailing the support team.
[1] Just so that this is not left completely unanswered; some DBMS (ex. SQL Server) have built in job scheduling facilities. You could run a Windows service on the server to do this. If not, you might consider running a Windows Service on one of your own servers to access the website (a great way to waste bandwidth).
Use a scheduler like this one, found on rufus site. You could program it to run, for instance, every hour, and make it do the job without human interaction.
I am a Java shop myself and I've been using quartz. It is quite good and usable if you can adjust to jruby.
I've never liked database or operating system based solutions, since you might not control them and often get asked to run on different environments.
Here's a very simple background job handler for Ruby:
codeforpeople.rubyforge.org/svn/bj/trunk/README
Easy to install and use. Fairly lightweight. It uses a SQL backend for managing concurrency. Runs on multiple machines simultaneously if you need it to.
Why not use the Windows scheduler?
I have several applications that have to run at certain times according to business rules not the typical every weekday at 1pm.
I also need a way for the applications to provide feedback of their progress so that I can have rules that notify me when the applications are running slow or aren't even running anymore.
What Windows API should I be looking into? (like, a time version of the FileWatcher apis)
What's the best way to have the application notify the scheduler of its progress (files, sockets, windows messages, ???)?
For Vista/Win2k8, there's the nice Task Scheduler 2.0 API: http://msdn.microsoft.com/en-us/library/aa384138(VS.85).aspx. Previous version have the Task Scheduler 1.0 API, but I've never used it.
AppControls has a CronJob component that you can use to create scheduled events. This saves your program from having to wake up every minute and check the schedule itself. Instead, just schedule the job and indicate a callback method.
I have used this component for scheduling jobs myself and have been very happy with the way that it works.
I think what you really want is a common framework for your applications that report to something (you or the system messages or tracing or perfmon, event log, whatever) and also to receive via some inter process protocol a way to receive messages and respond.
based on the reporting you can change the scheduling or make changes, etc.
So, there is some monitor app, and then each of your other apps does common reporting.
events I can think of:
- started
- stopped
- error
- normal log messages
- and of course specific things your apps do.
I think there are probably existing classes/framework that do this - you'll have to check around.
If it were me, I would make a service that could talk to all the other apps and perhaps was even an http server. It would be able to route messages to particular apps and start stop those processes and query them.
There are lots of ways to do what you want though. those were just off the top of my head.
Alternatively you might just be able to get these to be services and they handle messages sent to them. Their normal processing does nothing until they are "woken up" with some task command.
You have more questions in one. Normally you should split them. But let's overlook this and try to answer.
To schedule certain events (including running an application): Use TJvScheduledEvents from JVCL. IMHO JVCL is the best Delphi open source library around with extensive number of components, developers & support. TJvScheduledEvents is quite neat, uses threads for event scheduling and also you have in JVCL a detailed editor for your events (it needs a small hack to use it though).
To provide 'feedback' from your applications to a (remote) central point: A very very very good solution (if your requirements permit) is to log the progress of your applications in a table (let's call it LOG) on a Firebird server. In LOG you can have the following fields: COMPUTER, USERNAME, APPNAME, MSG, LOGDATE (etc. etc.). In the After Insert trigger of the LOG table you can fire an event (let's call it NEW_LOG). In your console app you can register the interest for this event and so, your application will be automatically updated with everything which happens in any of your applications, so you can do log analysis, graphs etc. Of course you can do it with IB, but IB costs.
...going on Windows API route you need headers (which probably aren't translated), you'll encounter our dearest Pointers/PChars etc. etc. Of course, building from scratch everything isn't worthwhile but when this is already done in a Delphi way, why don't use it?
Use service with a timer that is fired regulary (for example each minute). It reads the schedule and looks if some are due before the next iteration. If so, you can execute them.
You can add an interface that shows all running apps. For the feedback and query that using a desktop application.