I have started to look into Beanstalkd for queuing my Jobs for processing, I have following questions though that I request answers to:
Where can I enable Beanstalkd logging and where is the log file ?, so that in the event of issues from Beanstalkd , I can actually identify what the issue was.
What kind of performance impact is there if file sync for jobs is enabled and used?
How can I monitor the health of Beanstalkd and what are the vital health parameters of Beanstalkd that can be accessed via the running Beanstalkd application?
Please reply me with relevant answers/links.
thanks in advance
Ashish Sharma
Currently logging is not possible, there is a feature request here.
File sync for jobs must be enabled to have persistent mode. The impact is minimal, you don't notice it. You can still process 10k ops/second without any performance hit.
You can check out beanstalk admin panel it provides some numbers, that would help you at the beginning. But you have graph, nagios plugin available on the tools page.
You can check out other beanstalkd tools here.
Related
I have a Laravel application where the Application servers are behind a Load Balancer. On these Application servers, I have cron jobs running, some of which should only be run once (or run on one instance).
I did some research and found that people seem to favor a lock-system, where you keep all the cron jobs active on each application box, and when one goes to process a job, you create some sort of lock so the others know not to process the same job.
I was wondering if anyone had more details on this procedure in regards to AWS, or if there's a better solution for this problem?
You can build distributed locking mechanisms on AWS using DynamoDB with strongly consistent reads. You can also do something similar using Redis (ElastiCache).
Alternatively, you could use Lambda scheduled events to send a request to your load balancer on a cron schedule. Since only one back-end server would receive the request that server could execute the cron job.
These solutions tend to break when your autoscaling group experiences a scale-in event and the server processing the task gets deleted. I prefer to have a small server, like a t2.nano, that isn't part of the cluster and schedule cron jobs on that.
Check out this package for Laravel implementation of the lock system (DB implementation):
https://packagist.org/packages/jdavidbakr/multi-server-event
Also, this pull request solves this problem using the lock system (cache implementation):
https://github.com/laravel/framework/pull/10965
If you need to run stuff only once globally (so not once on every server) and 'lock' the thing that needs to be run, I highly recommend using AWS SQS because it offers exactly that: run a cron to fetch a ticket. If you get one, parse it. Otherwise, do nothing. So all crons are active on all machines, but tickets are 'in flight' when some machine requests a ticket and that specific ticket cannot be requested by another machine.
I setup a Mesos cluster which runs Apache Aurora framework, and i registered 100 cron jobs which run every min on a 5 slave machine pool. I found after scheduled 100 times, the cron jobs stacked in "PENDING" state. May i ask what kind of logs i can inspect and what is the possible problem ?
It could be a couple of things:
Do you still have sufficient resources in your cluster?
Are those resources offered to Aurora? Or maybe only to another framework?
Do you have any task constraints that prevent your tasks from being scheduled?
Possible information source:
What does the tooltip or the expanded status say on the UI? (as shown in the screenshot)
The Aurora scheduler has log files. However normally those are not needed for an end user to figure out why stuff is stuck in pending.
In case you are stuck here, it would probably be the best to drop by in the #aurora IRC channel on freenode.
I am trying to use the Queue system in Laravel (4.1). Everything works as expected, with both Redis (with the native driver) and RabbitMQ.
The only "issue" I am experiencing is the poor performance. It looks like only 4 jobs per seconds can be processed (I push 1000 jobs in the queue to test it). Have you got any tip to improve the performance?
This is a old question but I thought I would post anyway. The problem is Laravel's default listener is not really a true queue consumer, it polls the queue at regular intervals unless it is already busy with a job. Using a true AMQP requires some additional libraries to be install from pecl. You can find that plugin here. I would also suggest using this composer package for you PHP AMQP library. You would then need to write your Laravel command.
Currently I'm writing a RabbitMQ handler for Laravel that will resolve this issue.
Old question, but for anyone coming here, the queue:work command has a daemon mode that runs just like queue:listen except that it doesn't have to restart/reload Laravel each time, making it much more performant. See the docs:
http://laravel.com/docs/4.2/queues
I have enabled 'background process monitoring' in my newrelic account. It monitors all the delayed_job processes and provides information on the most time consuming task, memory and cpu usage.
Is there a way we can monitor the cron tasks as well with newrelic?
Thanks in advance!
http://blog.newrelic.com/2012/05/30/using-new-relic-to-monitor-ruby-background-tasks/
If you don't want your background tasks to affect your Apdex:
https://newrelic.com/docs/ruby/blocking-controller-instrumentation
Also the guys at New Relic are super helpful - don't be afraid to put in a support ticket!
Another good resource on monitoring custom backgrounds jobs with New Relic is this doc:
https://newrelic.com/docs/ruby/monitoring-ruby-background-processes-and-daemons#custom_background_jobs
I have a Drupal 6 site that is frequently (about once a day) going down. The hosting provider is reporting that something in our site code is occupying all Apache threads but keeping them idle, making the server run out of threads to respond to new requests. A simple restart of Apache frees the threads and fixes the issue, though it reoccurs within a few hours or a day.
I have no idea how to troubleshoot this issue and have never come across PHP code doing this. Is there some kind of Apache settings change I can make to capture more information about what might be keeping a thread occupied but idle? What typical PHP routines can cause this behavior? I looked for code that connects to external resources, but didn't see any issues there.
Any hints for what to look at, capture more information, or PHP code that can cause this would be most useful.
With Drupal6 you could have the poormanscron module running sometimes, or even the classical cron (from crontab wget or whatever).
Then you could get one heavy cron operation putting your database under heavy stuff. Then if your database reponse time is becoming very slow every http request will become very slow (as for example sessions are in the database, and several hundreds queries are required for a drupal page). having all reqests slowing down may put all the avàilable php process in a 'occupied state'.
Restarting apache all current process are stoped. If you run the cron via wget and not via drush cron tasks are a nice thing to check (running cron via drush would make it run via php-cli' restarting apache would not kill the cron). You can try a module like elysia cron to get more details on cron tasks and maybe isolate the long ones (you have a report on tasks duration).
This effect (one request hurting bad the database, all requests slowing down, no more process available) could also be done by one bad piece of code coming from any of your installed modules. This would be harder to detect.
So I would ensure slow queries are tracked on MySQL (see my.cnf otinons), then analyse theses requests with tolls like mysqsla. The problem is that sometimes one query is so big that all query becames slow. Se use time of crash te detect the first ones. Use also tho MySQL option to track queries not using indexes.
Another way to get all apache process stalled on php operation with drupal is having a lock problem. Drupal is using is own lock implementation with MySQL. You could maybe add some watchdog (drupal internal debug messages) calls on theses files to try to detect locks problems.
Then you could also have sonme external http requests calls made by drupal. Calling external websites like facebook, google, some tiny url tools, or drupal.org module update things (which always try to find all modules, even the one you write). If the distant website is down or filtering your traffic you'll have problems (but the apache restart would not help you, so it may not be that).