Laravel's database queue runs a database query every second to see if there are jobs to be processed.
I know it's not a complex query, but we want to reduce the number of connections hitting our database and we don't have that many jobs right now to consume. We'd like to modify it to run every 15-30 seconds or even longer.
I don't see config option to do something like this in the documentation and haven't found questions that cover this type of use case.
I do see that rate limiting can be enabled when using Redis queues, but our project needs to use the database queue for the time being.
We're on Laravel 5.5 and PHP 7.0 right now and it will be a while before we upgrade to newer versions. I wanted to go with Laravel Horizon, but that requires an upgrade to PHP 7.1.
Any help is appreciated.
From the documentation of laravel
Worker Sleep Duration
When jobs are available on the queue, the worker will keep processing jobs with no delay
in between them. However, the sleep option determines how long (in seconds) the worker
will "sleep" if there are no new jobs available. While sleeping, the worker will not
process any new jobs - the jobs will be processed after the worker wakes up again.
php artisan queue:work --sleep=30
Related
I am using laravel 5 and queue driver beanstalkd already installed in application, could someone please suggest what i have to do for achieving parallel processing, i want to run jobs in parallel in a same or different queue. currently it oly process one job at a time which is very time-consuming.
You can have multiple workers each watching a number of tubes. Which jobs will be run first depends on any priority when they were put into the system, or simple first-come-first.
It's a very common pattern to start and keep workers (one, or many) running with a tool such as 'SupervisorD'.
Check the Laravel docs:
https://laravel.com/docs/5.5/queues#supervisor-configuration
Supervisor is that you need. Supervisor will control your workers, if they die, supervisor will restart them again. For parallel processing, check the attribute numprocs of Supervisor
From Laravel docs:
the numprocs directive will instruct Supervisor to run x queue:work processes and monitor all of them, automatically restarting them if they fail.
I've got multiple servers sharing a database - on each of them a cron job fires ever 5 min checking if a text message log entry doesn't exist, creates a text message log entry and sends out a text message. I thought that there would never be a situation where text messages are sent multiple times, as one server should be first.
Well - I was wrong and that scenario did happen:
A - check if log exists - it doesn't
B - check if log exists - it doesn't
A - create log
B - create log
A - send message
B - send message
I've changed this behaviour to introduce queue, which should mitigate the issue. While the crons will still fire, multiple jobs will be queued, and workers should pick up given jobs at different times, thus preventing of sending of message twice. Though it might as well end up being:
A - pick up job 1
B - pick up job 2
A - check if log exists - it doesn't
B - check if log exists - it doesn't
Etc or A and B might as well pickup the same job at exactly the same time.
The solution would be, I guess, to run one worker server. But then I've the situation that jobs from multiple servers are queued many times, and I can't check if they're already enqueued as we end up with first scenario.
I'm at loss on how to proceed here - while multiple server, one worker server setup will work, I don't want to end up with instances of the same job (coming from different servers) multiple times in the queue.
Maybe the solution to go for is to have one cron/queue/worker server, but I don't have experience with Laravel/multiserver environment to set it up.
The other problematic thing for me is - how to test this? I can't, I guess, test it locally unless there's a way I can spin VM instances that are synchronized with each other.
The easy answer:
The code that checks the database for the existing database entry could use a database transaction with a level high enough to make sure that everyone else that is trying to do the same thing at the same time will be blocked and wait for the job to finish/commit.
A really naive solution (assuming mysql) would be LOCK TABLES entries WRITE; followed by the logic, then UNLOCK TABLES when you're done.
This also means that no one can access the table while your job is doing the check. I hope the check is really quick, because you'll block all access to the table for a small time period every five minutes.
WRITE lock:
The session that holds the lock can read and write the table.
Only the session that holds the lock can access the table. No other session can access it until the lock is released.
Lock requests for the table by other sessions block while the WRITE lock is held.
Source: https://dev.mysql.com/doc/refman/5.7/en/lock-tables.html
That was a really boring answer, so I'll move on to the answer you're probably more interested in...
The server architecture answer:
Your wish to only have one job per time interval in your queue means that you should only have one machine dispatching the jobs. This is easiest done with one dedicated machine that only dispatches jobs from scheduled commands. (Laravel 5.5 introduced the ability to dispatch jobs directly from the scheduler; see Scheduling Queued Jobs)
You can then have an several worker machines processing the queue, and only one of them will pick up the job and execute it. Two worker machines will never execute the same job at the same time if everything works as usual*.
I would split up the web machines from the worker machines so that they can scale independently. I prefer having my web machines dedicated to web traffic, they are not processing jobs to make sure that any large amount of queued jobs will not affect my http response times.
So, I recommend the following machine types in your setup;
The scheduler - one single machine that runs the schedule and dispatches jobs.
Worker machines that handles your queue.
Web machines that handles visitors' traffic.
All machines will have identical source code for your Laravel application. They will also also have an identical configuration. The only think that is unique per machine type is ...
The scheduler has php artisan schedule:run in the crontab.
The workers have supervisor (or something similar) that runs php artisan queue:work.
The web servers have nginx + php-fpm and handles incoming web requests.
This setup will make sure that you will only get one job per 5 minute since there is only one machine that is pushing it. This setup will also make sure that the cpu load generated by the workers aren't affecting the web requests.
One issue with my answer is obvious; that single scheduler machine is a single point of failure. If it dies you will no longer have any of these scheduled jobs dispatched to the queue. That touches areas like server monitoring and health checks, which is out-of-scope of your question and are also highly dependant on your hosting provider.
Regarding that little asterisk; I can make up weird scenarios where a job is executed on several machines. This involves jobs that sleeps for longer than the timeout, while at the same time you've got an environment without support for terminating the job. This will cause the first worker to keep executing the job (since it cannot terminate it), and a second worker will consider the job as timed-out and retry it.
Since Laravel 5.6+ you can ensure your scheduled tasks only run on a single instance using the onOneServer function e.g.
$schedule->command('loggingTask')
->everyFiveMinutes()
->onOneServer();
This requires an APC or Redis cache to be set up because it seems to use a mutual exclusion lock, probably RedisLock if Redis is set up.
Using a queue you shouldn't really have such a problem because popping a task off a queue should be an atomic operation.
Source
I am trying to implement the queues on localhost in Laravel 5.4. I have Window 8.1
I have opened command prompt and found that the framework is continuously polling the jobs table to check if there is any pending job in queue. This process is happening after each 3 seconds.
Question: Is there any way to increase decrease the time?
You should be able to change it using the artisan command:
php artisan queue:work --sleep=3
Change the number 3 to whatever you want, in seconds. Sleep only applies when there are no jobs in the queue. If there are jobs in the queue, it pops them off without sleeping.
See this page for more options that you can configure:
https://laravel.com/docs/5.4/queues#running-the-queue-worker
I can't understand what's the difference between Laravel queue:work and Laravel queue:listen
I can see that:
Queue: Listen to a given queue
Work: Process the next job on a queue
But still don't get it, because I've tried both, both will run queue if there is any new queue ("work option" not just running once)
I'm not talking about the daemon option. Just these both.
Until Laravel 5.2 you had :listen and :work.
Work would process the first job in the queue.
Listen would process all jobs as they came through.
In Laravel 5.3+ this is no longer the case. Listen still exists, but it is deprecated and slated for removal in 5.5. You should prefer :work now.
Work now process jobs one after the other, but have a plethora of options you can configure.
Edit
The above was true at the time of the posting, but since then things have been changed a bit.
queue:work should be preferred when you want your queue's to run as a daemon. This would be a long-lived process that would be beneficial where performance was an issue. This will use a cached version of the application and does not re-bootstrap the application every time a job is processed.
queue:listen should be used when you don't care about performance or you don't want to have to restart the queue after making changes to the code.
They'll both pop jobs off the queue 1-by-1 as received.
They both share almost the exact same options that can be passed to them.
In Laravel 5.3+ queue:work runs a daemon listener. It could in 5.2 as well if you specified the --daemon flag. A daemon work boots the framework one time and then processes jobs repeatedly. The queue:listen command runs a queue:work --once sub-process in a loop which boots the framework each iteration.
queue:work should pretty much always be used in production as it's much more efficient and uses less RAM. However; you need to restart it after each core change. queue:listen is useful for development and local environments because you don't have to restart it after code changes (because the framework is booting fresh each job).
from here
The queue:work Artisan command includes a --daemon option for forcing
the queue worker to continue processing jobs without ever re-booting
the framework. This results in a significant reduction of CPU usage
when compared to the queue:listen command:
As you can see, the queue:work job supports most of the same options
available to queue:listen. You may use the php artisan help queue:work
job to view all of the available options.
https://laravel.com/docs/5.1/queues#running-the-queue-listener
There are two different issues listed.
There is artisan queue:work and artisan queue:listen
queue:work will simply pop off the next job in the queue, and process only that one job. This is a 'one off' command that will return to the command prompt once the one queue command is processed.
queue:listen will listen to the queue, and continue to process any queue commands it receives. This will continue running indefinitely until you stop it.
In Laravel >=4.2 there was a --daemon command added. The way it works is simply keeps running the queues directly, rather than rebooting the entire framework after every queue is processed. This is an optional command that significantly reduces the memory and cpu requirements of your queue.
The important point with the --daemon command is that when you upgrade your application, you need to specifically restart your queue with queue:restart, otherwise you could potentially get all sorts of strange errors as your queue would still have the old code in memory.
So to answer your question "Which command should I use for running my daemons?" - the answer is almost always queue:work --daemon
Here's the situation:
I have a Laravel 4.2 application that retrieves (from a third party API) an asset. This is a long-lived asset (it only changes once every 12-24 hours) and is kind of time consuming (a large image file). I do cache the asset, so the impact has been more or less minimized, but there's still the case where the first person who logs in to my application in the morning has to wait while the application loads the asset for the first time.
I have set up a job which will be queued up and will run every eight hours. This ought to ensure that the asset in the cache is always fresh. It works by re-enqueueing the job for eight hours later after it runs.
The problem is this: I'm about to deploy this job system to production & I'm not sure how to start this thing running for the first time.
Ideally, I'd like to have an administration option where I have a button which says "Click here to submit the job", but I'd like to make it as foolproof as possible & prevent people (I'm not the only administrator) from submitting the job multiple times. To do this, however, the application would need to check & see if the job is already in the queue. I can't find a way to do that in an implementation-independent way (I'm using redis, but that may change in the future).
Another option would be to add an artisan command to run the initial process. That way I could deploy the application, run an artisan command, and forget about it.
So, to recap, I have two questions:
Is there a way to check a queue to see what jobs are in there?
Is there a better way to do this?
Thanks
When a job is in laravel queue, it will be saved in jobs table, so you can check by DB.
If it's guaranteed to be the only thing ever in the queue, you could use something like:
if (Queue::size() === 0) {
Queue::push(...);
}
You would need to run the php artisanqueue:listen in the terminal.
Here is the complete documentation if you want to learn more about:
https://laravel.com/docs/5.2/queues#running-the-queue-listener
You can use the Laravel Telescope package.
Laravel Telescope is an elegant debug assistant for the Laravel framework. Telescope provides insight into the requests coming into your application, exceptions, log entries, database queries, queued jobs, mail, notifications, cache operations, scheduled tasks, variable dumps and more. Telescope makes a wonderful companion to your local Laravel development environment.
(Source: https://laravel.com/docs/7.x/telescope)