How to run Laravel queue:work in shared hosting without overlapping
i am using this code in a cronjob but this is using too much CPU resource due to overlapping command
what is the best way to do this.
/usr/local/bin/php /home/electro/public_html/artisan queue:work
The problem of high CPU is caused because the worker loads the complete framework everytime it checks for a job in the queue.
You can use:
php artisan queue:work --daemon
in your case:
/usr/local/bin/php /home/electro/public_html/artisan queue:work --daemon
This will load the framework once and the checking/processing of jobs happen inside a while loop, which lets CPU breathe easy.
Late to this but people are still looking for an answer
Create an entry on app\Console\Kernel.php
$schedule->command('queue:work --stop-when-empty')
->everyMinute()
->withoutOverlapping();
This will fire a queue worker once per minute, but then stop when the queue has been emptied. If there are a lot of jobs (more than one minutes worth) then the ->withoutOverlapping prevents the scheduler from starting another worker.
https://talltips.novate.co.uk/laravel/using-queues-on-shared-hosting-with-laravel
Related
We are running Laravel 7 and Horizon 4.3.5. Horizon runs with Supervisor.
We have 10 different queues configured, but workers responsible for one particular queue constantly dies without any output. After restarting Horizon, I can see these workers up and running for several seconds via top and ps commands. Then they are gone.
I checked supervisor's stdout_logfile: nothing suspicious there. I can see Jobs related to this queue are being processed successfully. Each worker processes exactly 2 jobs before crash.
I checked supervisor's stderr_logfile, but it's empty.
Laravel logs and failed_jobs table both are empty.
I even checked syslog, but nothing related there.
There are no problems with other queues at all. Only this particular queue keeps piling up: jobs are being pushed to queue by application, but never processed until I restart Horizon.
There are lot of free space on disk, free RAM, CPU usage is low.
Worker command: /usr/bin/php7.4 artisan horizon:work redis --delay=0 --memory=128 --queue=main --sleep=3 --timeout=1800 --tries=1 --supervisor=php01-Mexm:business
Turned out it was Out Of Memory problem. We had one job in this queue which caused crash.
Still not sure why logs were empty. Probably there wasn't enough memory to log anything.
Laravel's database queue runs a database query every second to see if there are jobs to be processed.
I know it's not a complex query, but we want to reduce the number of connections hitting our database and we don't have that many jobs right now to consume. We'd like to modify it to run every 15-30 seconds or even longer.
I don't see config option to do something like this in the documentation and haven't found questions that cover this type of use case.
I do see that rate limiting can be enabled when using Redis queues, but our project needs to use the database queue for the time being.
We're on Laravel 5.5 and PHP 7.0 right now and it will be a while before we upgrade to newer versions. I wanted to go with Laravel Horizon, but that requires an upgrade to PHP 7.1.
Any help is appreciated.
From the documentation of laravel
Worker Sleep Duration
When jobs are available on the queue, the worker will keep processing jobs with no delay
in between them. However, the sleep option determines how long (in seconds) the worker
will "sleep" if there are no new jobs available. While sleeping, the worker will not
process any new jobs - the jobs will be processed after the worker wakes up again.
php artisan queue:work --sleep=30
I am using laravel 5 and queue driver beanstalkd already installed in application, could someone please suggest what i have to do for achieving parallel processing, i want to run jobs in parallel in a same or different queue. currently it oly process one job at a time which is very time-consuming.
You can have multiple workers each watching a number of tubes. Which jobs will be run first depends on any priority when they were put into the system, or simple first-come-first.
It's a very common pattern to start and keep workers (one, or many) running with a tool such as 'SupervisorD'.
Check the Laravel docs:
https://laravel.com/docs/5.5/queues#supervisor-configuration
Supervisor is that you need. Supervisor will control your workers, if they die, supervisor will restart them again. For parallel processing, check the attribute numprocs of Supervisor
From Laravel docs:
the numprocs directive will instruct Supervisor to run x queue:work processes and monitor all of them, automatically restarting them if they fail.
I am trying to implement the queues on localhost in Laravel 5.4. I have Window 8.1
I have opened command prompt and found that the framework is continuously polling the jobs table to check if there is any pending job in queue. This process is happening after each 3 seconds.
Question: Is there any way to increase decrease the time?
You should be able to change it using the artisan command:
php artisan queue:work --sleep=3
Change the number 3 to whatever you want, in seconds. Sleep only applies when there are no jobs in the queue. If there are jobs in the queue, it pops them off without sleeping.
See this page for more options that you can configure:
https://laravel.com/docs/5.4/queues#running-the-queue-worker
I can't understand what's the difference between Laravel queue:work and Laravel queue:listen
I can see that:
Queue: Listen to a given queue
Work: Process the next job on a queue
But still don't get it, because I've tried both, both will run queue if there is any new queue ("work option" not just running once)
I'm not talking about the daemon option. Just these both.
Until Laravel 5.2 you had :listen and :work.
Work would process the first job in the queue.
Listen would process all jobs as they came through.
In Laravel 5.3+ this is no longer the case. Listen still exists, but it is deprecated and slated for removal in 5.5. You should prefer :work now.
Work now process jobs one after the other, but have a plethora of options you can configure.
Edit
The above was true at the time of the posting, but since then things have been changed a bit.
queue:work should be preferred when you want your queue's to run as a daemon. This would be a long-lived process that would be beneficial where performance was an issue. This will use a cached version of the application and does not re-bootstrap the application every time a job is processed.
queue:listen should be used when you don't care about performance or you don't want to have to restart the queue after making changes to the code.
They'll both pop jobs off the queue 1-by-1 as received.
They both share almost the exact same options that can be passed to them.
In Laravel 5.3+ queue:work runs a daemon listener. It could in 5.2 as well if you specified the --daemon flag. A daemon work boots the framework one time and then processes jobs repeatedly. The queue:listen command runs a queue:work --once sub-process in a loop which boots the framework each iteration.
queue:work should pretty much always be used in production as it's much more efficient and uses less RAM. However; you need to restart it after each core change. queue:listen is useful for development and local environments because you don't have to restart it after code changes (because the framework is booting fresh each job).
from here
The queue:work Artisan command includes a --daemon option for forcing
the queue worker to continue processing jobs without ever re-booting
the framework. This results in a significant reduction of CPU usage
when compared to the queue:listen command:
As you can see, the queue:work job supports most of the same options
available to queue:listen. You may use the php artisan help queue:work
job to view all of the available options.
https://laravel.com/docs/5.1/queues#running-the-queue-listener
There are two different issues listed.
There is artisan queue:work and artisan queue:listen
queue:work will simply pop off the next job in the queue, and process only that one job. This is a 'one off' command that will return to the command prompt once the one queue command is processed.
queue:listen will listen to the queue, and continue to process any queue commands it receives. This will continue running indefinitely until you stop it.
In Laravel >=4.2 there was a --daemon command added. The way it works is simply keeps running the queues directly, rather than rebooting the entire framework after every queue is processed. This is an optional command that significantly reduces the memory and cpu requirements of your queue.
The important point with the --daemon command is that when you upgrade your application, you need to specifically restart your queue with queue:restart, otherwise you could potentially get all sorts of strange errors as your queue would still have the old code in memory.
So to answer your question "Which command should I use for running my daemons?" - the answer is almost always queue:work --daemon