I have an API function that stores pdf file to my s3 bucket and then sends email with the pdf file as an attachment.
Since this is my first time, I got confused because from what I understood, jobs executes from the background, thus, it should not affect the execution time of the function.
But instead, having these jobs makes the execution time almost 8 seconds.
Here's my function
$is_exist = CoachingApplication::where('user_id', $userID)->first();
if ($is_exist == null) {
$application = new CoachingApplication();
$application->user_id = $userID;
$application->applicant_name = $applicantName;
$application->attachment = $filename;
$application->instrument_rate = $instrumentRate;
if ($application->save()) {
if ($filename !== 'none') {
StoreBucketJob::dispatch($userID, $filename, $attachment_fileArray)->delay(Carbon::now()->addSeconds(3));
}
SendEmailJob::dispatch($userID, $userName, $userSlug, $userEmail, $filename)->delay(Carbon::now()->addSeconds(3));
}
}
If I remove these jobs, the function's execution time is 469ms.
Any idea why these jobs affects the api's execution time?
By default, the queue driver is setup to sync and you are probably using it.
This queue driver means your jobs will be executed within the current process and will not be dispatched in an actual queue (which is pretty useful during development).
A good way to be 100% sure that your job are indeed executed synchronously would be to just put a dd("ok"); at the first line of the handle method inside your job. handle is only executed when the job runs, not when it is dispatched.
The queue driver can be updated by editing your .env file (look for QUEUE_CONNECTION).
There are many queue drivers available and some requires additional dependencies, so you should check out the documentation available https://laravel.com/docs/8.x/queues.
Related
I am adding jobs to a batch in a loop that is fed from an (3rd party) api. It sometimes loads all data and sometimes it takes time to get the data and the batch closes before the whole data is fetched.
do {
$page++;
$response = $api->getProducts(['page'=>$page])->json();
foreach ($response['products'] as $product) {
$batch->add(new \App\Jobs\ProcessJob($product));
}
$total_pages = $response['params']['total_pages'];
} while ($page < $total_pages);
It seems like in some cases the \App\Jobs\ProcessJob works faster than the the feeding class and the batch closes before the feeder adds all the needed data. How can I keep the batch open until the feeder class finishes feeding till the last page?
After some sifting around, I found out the batch actually does not close itself. My setting in queue config retry_after was set to short and the job running the batch was timed out by that setting. Now it does not timeout and works till the end.
I have a batched series of chained jobs, and inside those chains I need to be able to batch other jobs.
Say I have 3 Clients
For Each Client I need to
Sync their details with an external API
Create 0 or more new cases and sync them individually
Update 0 or more existing cases and sync them individually
And I need the wrapping batch to keep track of when this is all finished.
I currently have the following structure:
$jobs = $clients->map(fn(Client $client) => [
new SyncClientJob(...),
new CreateMultipleCasesJob(...),
new UpdateMultipleCasesJob(...)
]);
Bus::batch($jobs)->name('BatchA')->etc()
In CreateCasesJob, something along the lines of
public function handle()
{
$jobs = $collection_of_new_cases->map(fn(Case $case) => new CreateSingleCaseJob($case));
Bus::batch($jobs)->dispatch();
}
CreateCasesJob and UpdateCasesJob should both dispatch their own batch of jobs, since each case needs to be synced individually
The problem is of course that the Create/Update jobs are "complete" in the chain when they're dispatched, not when all their internal jobs are completed. So the BatchA job will be marked as completed when it hasn't yet synced any cases.
I solved this by having each batch of jobs dispatch an event in the ->finally() callback. The listener for that event would then build and start the next batch.
End Goal
The aim is for my application to fire off potentially a lot of emails to the Redis queue (This bit is working) and then Redis throttle the processing of these to only a set number of emails every selected number of minutes.
For this example, I have a test job that appends the time to a file and I am attempting to throttle it to once every 60 seconds.
The story so far....
So far, I have the application successfully pushing a test amount of 50 jobs to the Redis queue. I can log in to Horizon and see these 50 jobs in the "processjob" queue. I can also log in to redis-cli and see 50 sets under the list key "queues:processjob".
My issue is that as soon as I attempt to put the throttle on, only 1 job runs and the rest fail with the following error:
Predis\Response\ServerException: ERR Error running script (call to f_29cc07bd431ccbf64637e5dcb60484560fdfa2da): #user_script:10: WRONGTYPE Operation against a key holding the wrong kind of value in /var/www/html/smhub/vendor/predis/predis/src/Client.php:370
If I remove the throttle, all works file and 5 jobs are instantly ran.
I thought maybe it was the incorrect key name but if I change the following:
public function handle()
{
//
Redis::throttle('queues:processjob')->allow(1)->every(60)->then(function(){
Storage::disk('local')->append('testFile.txt',date("Y-m-d H:i:s"));
}, function (){
return $this->release(10);
});
}
to this:
public function handle()
{
//
Redis::funnel('queues:processjob')->limit(1)->then(function(){
Storage::disk('local')->append('testFile.txt',date("Y-m-d H:i:s"));
}, function (){
return $this->release(10);
});
}
then it all works fine.
My thoughts...
Something tells me that the issue is that the redis key is of type "list" and that the jobs are all under a single list. That being said, if it didn't work this way, how would we throttle a queue as the throttle requires a unique key.
For anybody else that is having issues attempting to get this to work and is getting the same issue as I was, this is what resolved my issues:
The Fault
I assumed that Redis::throttle('queues:processjob') was meant to be referring to the queue that you wanted to be throttled. However, after some re-reading of the documentation and testing of the code, I realized that this was not the case.
The Fix
Redis::throttle('queues:processjob') is meant to point to it's own 'holding' queue and so must be a unique Redis key name. Therefore, changing it to Redis::throttle('throttle:queues:processjob') worked fine for me.
The workings
When I first looked in to this, I assumed that that Redis::throttle('this') throttled the queue that you specified. To some degree this is correct but it will not work if the job was created via another means.
Redis::throttle('this') actually creates a new 'holding' queue where the jobs go until the condition(s) you specify are met. So jobs will go to the queue 'this' in this example and when the throttle trigger is released, they will be passed to the queue specified in their execution code. In this case, 'queues:processjob'.
I hope this helps!
In need run other function without stop.
How to use thread in laravel 5.6?
For example:
public function index()
{
$id = "123456";
$this->run_bot($id);
return view("index");
}
Funtion run_bot it takes about 10 minutes !!!!
I need run run_bot in a thread.
How to craete thread in laravel 5.6?
Look into Symfomy's Process Component.
As an example, you can start the process and then later wait for it to complete:
$process = new Process('ls -lsa');
$process->start();
// ... do other things
// this is optional, you don't need to wait if not necessary
$process->wait();
The solution you are looking for is how to run asynchronous jobs. This can be done with a queue service (like AWS SQS) and the Laravel queue worker.
It will allow you to send the job (really light work so it's really speed). And then, asynchronously, retrieve and execute the job.
Everything you will need to know is here :
https://laravel.com/docs/5.6/queues
Let me know if it helped you :)
Is it possible to use dispatchShell from a Controller?
My mission is to start a shell job when the user has signed up.
I'm using CakePHP 2.0
If you can't mitigate the need to do this as dogmatic suggests then, read on.
So you have a (potentially) long-running job you want to perform and you don't want the user to wait.
As the PHP code your user is executing happens during a request that has been started by Apache, any code that is executed will stall that request until it completion (unless you hit Apache's request timeout).
If the above isn't acceptable for your application then you will need to trigger PHP outwith the Apache request (ie. from the command line).
Usability-wise, at this point it would make sense to notify your user that you are processing data in the background. Anything from a message telling them they can check back later to a spinning progress bar that polls your application over ajax to detect job completion.
The simplest approach is to have a cronjob that executes a PHP script (ie. CakePHP shell) on some interval (at minimum, this is once per minute). Here you can perform such tasks in the background.
Some issues arise with background jobs however. How do you know when they failed? How do you know when you need to retry? What if it doesn't complete within the cron interval.. will a race-condition occur?
The proper, but more complicated setup, would be to use a work/message queue system. They allow you to handle the above issues more gracefully, but generally require you to run a background daemon on a server to catch and handle any incoming jobs.
The way this works is, in your code (when a user registers) you insert a job into the queue. The queue daemon picks up the job instantly (it doesn't run on an interval so it's always waiting) and hands it to a worker process (a CakePHP shell for example). It's instant and - if you tell it - it knows if it worked, it knows if it failed, it can retry if you want and it doesn't accidentally handle the same job twice.
There are a number of these available, such as Beanstalkd, dropr, Gearman, RabbitMQ, etc. There are also a number of CakePHP plugins (of varying age) that can help:
cakephp-queue (MySQL)
CakePHP-Queue-Plugin (MySQL)
CakeResque (Redis)
cakephp-gearman (Gearman)
and others.
I have had experience using CakePHP with both Beanstalkd (+ the PHP Pheanstalk library) and the CakePHP Queue plugin (first one above). I have to credit Beanstalkd (written in C) for being very lightweight, simple and fast. However, with regards to CakePHP development, I found the plugin faster to get up and running because:
The plugin comes with all the PHP code you need to get started. With Beanstalkd, you need to write more code (such as a PHP daemon that polls the queue looking for jobs)
The Beanstalkd server infrastructure becomes more complex. I had to install multiple instances of beanstalkd for dev/test/prod, and install supervisord to look after the processes).
Developing/testing is a bit easier since it's a self-contained CakePHP + MySQL solution. You simply need to type cake queue add user signup and cake queue runworker.
I was able to run consolle from controller/action, see the example below.
App::uses('ShellDispatcher', 'Console');
...
public function aco_sync() {
$command = '-app '.APP.' AclExtras.AclExtras aco_sync -r adminControllers -p UserAdmin';
$args = explode(' ', $command);
$dispatcher = new ShellDispatcher($args, false);
if($dispatcher->dispatch()) {
$this->Session->flash('OK');
} else {
$this->Session->flash('Error');
}
return $this->redirect(array('action' => 'index'));
}
In CakePHP-3 you can dispatch shells from the controller & do it almost the same as in CakePHP-2. The documentation does not mention this.
// in your controller:
$shell = new \Cake\Console\Shell;
$shell->dispatchShell('shell_class param1 param2');
// or how the docs suggest
$shell->dispatchShell('shell_class', 'param1', 'param2');
Beware of stdout & stderr in unit tests.
Dispatching a shell turns on stdout and stderr logging with ConsoleLogger, and will give you all the logging in your console if you have something like the code snippet above in code that you are testing from phpunit.
function getEbayOrder(){
$this->autoRender = false;
App::import('Console/Command', 'AppShell');
App::import('Console/Command', 'EbayShell');
$job = new EbayShell();
$job->dispatchMethod('get_orders');
echo "REPONSE";
}
anything is possible, but why would you want to. If you find you need to do something in a shell and the actual application look at using libs.
you stick the code in the lib and then call the lib from both your app and the shell.
If this is to intialize AclExtras the best way is:
App::import('Console/Command', 'AppShell');
App::import('Plugin/AclExtras/Console/Command', 'AclExtrasShell');
$job = new AclExtrasShell();
$job->startup();
$job->dispatchMethod('aco_sync');
But avoid this unless you have no possibilities to run the console script.