I have enabled 'background process monitoring' in my newrelic account. It monitors all the delayed_job processes and provides information on the most time consuming task, memory and cpu usage.
Is there a way we can monitor the cron tasks as well with newrelic?
Thanks in advance!
http://blog.newrelic.com/2012/05/30/using-new-relic-to-monitor-ruby-background-tasks/
If you don't want your background tasks to affect your Apdex:
https://newrelic.com/docs/ruby/blocking-controller-instrumentation
Also the guys at New Relic are super helpful - don't be afraid to put in a support ticket!
Another good resource on monitoring custom backgrounds jobs with New Relic is this doc:
https://newrelic.com/docs/ruby/monitoring-ruby-background-processes-and-daemons#custom_background_jobs
Related
I setup a Mesos cluster which runs Apache Aurora framework, and i registered 100 cron jobs which run every min on a 5 slave machine pool. I found after scheduled 100 times, the cron jobs stacked in "PENDING" state. May i ask what kind of logs i can inspect and what is the possible problem ?
It could be a couple of things:
Do you still have sufficient resources in your cluster?
Are those resources offered to Aurora? Or maybe only to another framework?
Do you have any task constraints that prevent your tasks from being scheduled?
Possible information source:
What does the tooltip or the expanded status say on the UI? (as shown in the screenshot)
The Aurora scheduler has log files. However normally those are not needed for an end user to figure out why stuff is stuck in pending.
In case you are stuck here, it would probably be the best to drop by in the #aurora IRC channel on freenode.
I have started to look into Beanstalkd for queuing my Jobs for processing, I have following questions though that I request answers to:
Where can I enable Beanstalkd logging and where is the log file ?, so that in the event of issues from Beanstalkd , I can actually identify what the issue was.
What kind of performance impact is there if file sync for jobs is enabled and used?
How can I monitor the health of Beanstalkd and what are the vital health parameters of Beanstalkd that can be accessed via the running Beanstalkd application?
Please reply me with relevant answers/links.
thanks in advance
Ashish Sharma
Currently logging is not possible, there is a feature request here.
File sync for jobs must be enabled to have persistent mode. The impact is minimal, you don't notice it. You can still process 10k ops/second without any performance hit.
You can check out beanstalk admin panel it provides some numbers, that would help you at the beginning. But you have graph, nagios plugin available on the tools page.
You can check out other beanstalkd tools here.
I'm building a monitoring service similar to pingdom but monitoring different aspects of a system and using sidekiq to queue the tasks which is working well. What I need to do is to schedule sending out pings every minute, rather than using a cron based system which would require spinning up a new ruby instance every minute I have gone down the route of using sidetiq (notice the different spelling with a "t") which uses sidekiq's own queue to schedule future tasks. This feels like a neat solution, however I am concerned this may not be the most reliable way of scheduling tasks? If there are issues with the system (as there inevitable will be at some point) will this method of scheduling tasks be less reliable than using a cron based method and why?
Thanks
You are giving too short description of your system needs but I'll try to guess how it could be:
In the first place using sidekiq means that you'll also need an instance of redis and also means that you'll need a way to monitor the sidekiq process and restart it in case of failure and possibly redis server.
A method based on cron tasks will have fewer requirements therefore much less possibilities of failing.
cron has been around for a long time and it's battle tested and it's very very reliable, but has it's drawbacks too.
Said that, you can build a system with separate instances of redis in a master/slave configuration and you can also use Redis sentinel to implement a failover in case of the master failure, implement a monitoring/alerting system on this setup (you can use something super simple like this http://contribsys.com/inspeqtor/ from the sidekiq author) and you can also start several instances of sidekiq in different machines.
With all of that, you can have a quite reliable system for running sidekiq with sidetiq.
Hope it helps
I am wondering if there is a way to monitor these automatically. Right now, in our production/QA/Dev environments - we have bunch of services running that are critical to the application. We also have automatic ETLs running on windows task scheduler at a set time of the day. Currently, I have to log into each server and see if all the services are running fine or not, or check event logs for any errors, or check task scheduler to see if ETLs ran well etc etc... I have to do all the manually... I am wondering if there is a tool out there that will do the monitoring for me and send emails only in case something needs attention (like ETLs fail to run, or service get stopped for whatever reason or errors in event log etc). Thanks for the help.
Paessler PRTG Network Monitor can do all that. we have very good experience with it.
http://www.paessler.com/prtg/features
Nagios is the best tool for monitoring. It checks for the server status as well the defined services in it and if any service goes down or system goes down, sends the mail to specified mail id.
Refer the : http://nagios.org/
Thanks for the above information. I looked at the above options but they have a price.. what I did is an inexpensive way to address my concerns..
For my windows task scheduler jobs that run every night - I installed this tool/service from codeplex that is working great.
http://motash.codeplex.com/documentation#CommentsAnchor
For Windows services - I am just setting the "Recovery" Tab in each service "property" with actions to do when it fails. (like restart, reboot, or run a program which could be an email that will notify)
I built a simple tool (https://cronitor.io) for monitoring periodic/scheduled tasks. The name is a play on "cron" from the unix world, but it is system/task agnostic. All you have to do is make an http request to a unique tracking URL whenever your job runs. If your job doesn't check-in according to the rules you define then it will send you an email/sms message.
It also allows you to track the duration of your jobs by making calls at the beginning and end of your task. This can be really useful for long running jobs since you can be alerted if they start taking too long to run. For example, I once had a backup task that was scheduled every hour. About six months after I set it up it started taking longer than an hour to run!
There is https://eyewitness.io - which is for monitoring server cron tasks, queues and websites. It makes sure each of your cron jobs run when they are supposed to, and alerts you if they failed to be run.
Is there a way to set time table to a process in windows services?
As in Scheduled Tasks I want to start and stop a service at a certain time.
any suggestions?
Thanks
The way we have done this, is to start and stop the thread at the required time. We had to do it just because we were not allowed to create scheduled tasks. Scheduled tasks are best for this kind of situation. Windows service will consume some CPU cycles even though the thread has nothing to do.
To start and stop a service you need some external component acting as kind of a monitor.
Prasad Pimparkar