I have a batched series of chained jobs, and inside those chains I need to be able to batch other jobs.
Say I have 3 Clients
For Each Client I need to
Sync their details with an external API
Create 0 or more new cases and sync them individually
Update 0 or more existing cases and sync them individually
And I need the wrapping batch to keep track of when this is all finished.
I currently have the following structure:
$jobs = $clients->map(fn(Client $client) => [
new SyncClientJob(...),
new CreateMultipleCasesJob(...),
new UpdateMultipleCasesJob(...)
]);
Bus::batch($jobs)->name('BatchA')->etc()
In CreateCasesJob, something along the lines of
public function handle()
{
$jobs = $collection_of_new_cases->map(fn(Case $case) => new CreateSingleCaseJob($case));
Bus::batch($jobs)->dispatch();
}
CreateCasesJob and UpdateCasesJob should both dispatch their own batch of jobs, since each case needs to be synced individually
The problem is of course that the Create/Update jobs are "complete" in the chain when they're dispatched, not when all their internal jobs are completed. So the BatchA job will be marked as completed when it hasn't yet synced any cases.
I solved this by having each batch of jobs dispatch an event in the ->finally() callback. The listener for that event would then build and start the next batch.
Related
I have an API function that stores pdf file to my s3 bucket and then sends email with the pdf file as an attachment.
Since this is my first time, I got confused because from what I understood, jobs executes from the background, thus, it should not affect the execution time of the function.
But instead, having these jobs makes the execution time almost 8 seconds.
Here's my function
$is_exist = CoachingApplication::where('user_id', $userID)->first();
if ($is_exist == null) {
$application = new CoachingApplication();
$application->user_id = $userID;
$application->applicant_name = $applicantName;
$application->attachment = $filename;
$application->instrument_rate = $instrumentRate;
if ($application->save()) {
if ($filename !== 'none') {
StoreBucketJob::dispatch($userID, $filename, $attachment_fileArray)->delay(Carbon::now()->addSeconds(3));
}
SendEmailJob::dispatch($userID, $userName, $userSlug, $userEmail, $filename)->delay(Carbon::now()->addSeconds(3));
}
}
If I remove these jobs, the function's execution time is 469ms.
Any idea why these jobs affects the api's execution time?
By default, the queue driver is setup to sync and you are probably using it.
This queue driver means your jobs will be executed within the current process and will not be dispatched in an actual queue (which is pretty useful during development).
A good way to be 100% sure that your job are indeed executed synchronously would be to just put a dd("ok"); at the first line of the handle method inside your job. handle is only executed when the job runs, not when it is dispatched.
The queue driver can be updated by editing your .env file (look for QUEUE_CONNECTION).
There are many queue drivers available and some requires additional dependencies, so you should check out the documentation available https://laravel.com/docs/8.x/queues.
I have job A which downloads xml and then calls other job B which will create data on database. This job B will be called in loop and can be more than 10.000 items. First tried to use chain method but problem is that, if someone will call queue in wrong sequence it will not work. Then tried to use batch from new Laravel 8. Collecting all jobs (more than 10000) to one batch can cause out of memory exception. Other problem is calling job C at the end. This job will update some credentials. Thats why job A and B must be runned successfully. May be there is any good idea for this situation?
Laravel's job batching feature allows you to easily execute a batch of jobs and then perform some action when the batch of jobs has completed executing.
If you have an out-of-memory problem with Jobs Batching you are doing things wrong. Since the queues are executed one by one if you have it configured that way there should be no problems, even if they are more than 100k records. So, make sure you glue one Job for each item, and execute the action, you won't have problems with this.
Then, you could do something like this.
$chain = [
new ProcessPodcast(Podcast::find(1)),
new ProcessPodcast(Podcast::find(2)),
new ProcessPodcast(Podcast::find(3)),
new ProcessPodcast(Podcast::find(4)),
new ProcessPodcast(Podcast::find(5)),
...
// And so on for all your items.
// This should be generated by a foreach with all his items.
];
Bus::batch($chain)->then(function (Batch $batch) {
// All jobs completed successfully...
// Uupdate some credentials...
})->catch(function (Batch $batch, Throwable $e) {
// First batch job failure detected...
})->finally(function (Batch $batch) {
// The batch has finished executing...
})->dispatch();
I'm using Laravel Jobs to pull data from the Stripe API in a paginated way. Basically, each job gets a "brand id" (a user can have multiple brands per account) and a "start after" parameter. It uses that to know which stripe token to use and where to start in the paginated calls (this job call itself if more stripe responses are available in the pagination). This runs fine when the job is started once.
But there is a use case where a user could add stripe keys to multiple brands in a short time and that job class could get called multiple times concurrently with different parameters. When this happens, whichever process is started last overwrites the others because the parameters are being overwritten to just the last called. So if I start stripe job with brand_id = 1, then job with brand_id = 2, then brand_id = 3, 3 overwrites the other two after one cycle and only 3 gets passed for all future calls.
How do I keep this from happening?
I've tried static vars, I've tried protected, private and public vars. I thought might be able to solve it with dynamically created queues for each brand, but this seems like a huge headache.
public function __construct($brand_id, $start_after = null)
{
$this->brand_id = $brand_id;
$this->start_after = $start_after;
}
public function handle()
{
// Do stripe calls with $brand_id & $start_after
if ($response->has_more) {
// Call next job with new "start_at".
dispatch(new ThisJob($this->brand_id, $new_start_after));
}
}
According to Laravel Documentation
if you dispatch a job without explicitly defining which queue it
should be dispatched to, the job will be placed on the queue that is
defined in the queue attribute of the connection configuration.
// This job is sent to the default queue...
dispatch(new Job);
// This job is sent to the "emails" queue...
dispatch((new Job)->onQueue('emails'));
However, pushing jobs to multiple queues with unique names can be especially useful for your use case.
The queue name may be any string that uniquely identifies the queue itself. For example, you may wish to construct the queue name based on the uniqid() and $brand_id.
E.g:
dispatch(new ThisJob($this->brand_id, $new_start_after)->onQueue(uniqid() . '_' . $this->brand_id));
I'm developing a binary multilevel marketing system in Laravel, at the registration time there we have to perform a task to entries for many types of bonus for each parent nodes of a new user. This task is time-consuming.
No one user want to see buffering and task taking more than 30 second that is not the right way.
I want to run this mechanism in the background and send a success message that your account created successfully.
You could use observers that trigger queued jobs.
After the user does an action on a model, the observers create queued jobs in the background. While the queue is being processed the user can continue working.
either implement laravel job and queues or use https://github.com/spatie/async.
you can invoke sub processes to make your task
use Spatie\Async\Pool;
$pool = Pool::create();
foreach ($things as $thing) {
$pool->add(function () use ($thing) {
// Do a thing
})->then(function ($output) {
// Handle success
})->catch(function (Throwable $exception) {
// Handle exception
});
}
$pool->wait();
I'm trying to get department details from an API which supports pagination, so if I spawn one job per page like following
/departments?id=1&page=1 -> job1
/departments?id=1page=2 -> job2
How can I keep track of these jobs for a particular department as I have to write the responses to txt file.
The jobs are instantiated via controller class like:
class ParseAllDeptsJob implements ShouldQueue
{
public function handle()
{
foreach (Departments::all() as $dept) {
ParseDeptJob::dispatch($dept);
}
}
}
You can chain a job, using withChain(). This job will not run if the jobs higher up the chain fail.
From the documentation:
Job chaining allows you to specify a list of queued jobs that should
be run in sequence. If one job in the sequence fails, the rest of the
jobs will not be run. To execute a queued job chain, you may use the
withChain method on any of your dispatchable jobs:
In your case, this is how you'd do it:
ParseAllDeptsJob::withChain([
new SendEmailNotification
])->dispatch();
SendEmailNotification won't be dispatched if an error occurs while processing ParseAllDeptsJob.