Laravel Single Job Class dispatched multiple times with different parameters getting overwritten - laravel

I'm using Laravel Jobs to pull data from the Stripe API in a paginated way. Basically, each job gets a "brand id" (a user can have multiple brands per account) and a "start after" parameter. It uses that to know which stripe token to use and where to start in the paginated calls (this job call itself if more stripe responses are available in the pagination). This runs fine when the job is started once.
But there is a use case where a user could add stripe keys to multiple brands in a short time and that job class could get called multiple times concurrently with different parameters. When this happens, whichever process is started last overwrites the others because the parameters are being overwritten to just the last called. So if I start stripe job with brand_id = 1, then job with brand_id = 2, then brand_id = 3, 3 overwrites the other two after one cycle and only 3 gets passed for all future calls.
How do I keep this from happening?
I've tried static vars, I've tried protected, private and public vars. I thought might be able to solve it with dynamically created queues for each brand, but this seems like a huge headache.
public function __construct($brand_id, $start_after = null)
{
$this->brand_id = $brand_id;
$this->start_after = $start_after;
}
public function handle()
{
// Do stripe calls with $brand_id & $start_after
if ($response->has_more) {
// Call next job with new "start_at".
dispatch(new ThisJob($this->brand_id, $new_start_after));
}
}

According to Laravel Documentation
if you dispatch a job without explicitly defining which queue it
should be dispatched to, the job will be placed on the queue that is
defined in the queue attribute of the connection configuration.
// This job is sent to the default queue...
dispatch(new Job);
// This job is sent to the "emails" queue...
dispatch((new Job)->onQueue('emails'));
However, pushing jobs to multiple queues with unique names can be especially useful for your use case.
The queue name may be any string that uniquely identifies the queue itself. For example, you may wish to construct the queue name based on the uniqid() and $brand_id.
E.g:
dispatch(new ThisJob($this->brand_id, $new_start_after)->onQueue(uniqid() . '_' . $this->brand_id));

Related

How to lock the job during execution in Laravel?

I see withoutOverlapping() mutex for commands, but I don't see it for jobs. How can I protect jobs of the same type from overlapping each other?
Thanks!
I think it's possible using the following:
https://laravel.com/docs/8.x/queues#unique-jobs
You can specify a needed key that you can pass to the job to mark its uniqueness. In my case, I need to limit requests to a third-party API that happens in the job so if I have more than one worker handling the queue, it's possible to get 429 from the API. As soon as I have many API-keys (per user of the app), I can use it to have the same type of job being exxecuted independently across the app users but lock the job execution if the current job with a specific key is not completed.
Like this:
//In the class defining you must use ShouldBeUnique interface
class UpdateSpreadsheet implements ShouldQueue, ShouldBeUnique
//some other code
public function __construct($keyValue)
{
//some other constructor code if needed
$this->keyValue= $keyValue;
}
//This function allows to set the unique key
public function uniqueId()
{
return $this->keyValue;
}
//If you don't need to wait until the job is processed, you may also specify
//the time for the force lock removing (so you'll be able to queue another
//job with this key after 10 seconds even if the current job is
//still in process)
public $uniqueFor = 10;

Keep track of laravel queued jobs

I'm trying to get department details from an API which supports pagination, so if I spawn one job per page like following
/departments?id=1&page=1 -> job1
/departments?id=1page=2 -> job2
How can I keep track of these jobs for a particular department as I have to write the responses to txt file.
The jobs are instantiated via controller class like:
class ParseAllDeptsJob implements ShouldQueue
{
public function handle()
{
foreach (Departments::all() as $dept) {
ParseDeptJob::dispatch($dept);
}
}
}
You can chain a job, using withChain(). This job will not run if the jobs higher up the chain fail.
From the documentation:
Job chaining allows you to specify a list of queued jobs that should
be run in sequence. If one job in the sequence fails, the rest of the
jobs will not be run. To execute a queued job chain, you may use the
withChain method on any of your dispatchable jobs:
In your case, this is how you'd do it:
ParseAllDeptsJob::withChain([
new SendEmailNotification
])->dispatch();
SendEmailNotification won't be dispatched if an error occurs while processing ParseAllDeptsJob.

Laravel Jobs fail on Redis when attempting to use throttle

End Goal
The aim is for my application to fire off potentially a lot of emails to the Redis queue (This bit is working) and then Redis throttle the processing of these to only a set number of emails every selected number of minutes.
For this example, I have a test job that appends the time to a file and I am attempting to throttle it to once every 60 seconds.
The story so far....
So far, I have the application successfully pushing a test amount of 50 jobs to the Redis queue. I can log in to Horizon and see these 50 jobs in the "processjob" queue. I can also log in to redis-cli and see 50 sets under the list key "queues:processjob".
My issue is that as soon as I attempt to put the throttle on, only 1 job runs and the rest fail with the following error:
Predis\Response\ServerException: ERR Error running script (call to f_29cc07bd431ccbf64637e5dcb60484560fdfa2da): #user_script:10: WRONGTYPE Operation against a key holding the wrong kind of value in /var/www/html/smhub/vendor/predis/predis/src/Client.php:370
If I remove the throttle, all works file and 5 jobs are instantly ran.
I thought maybe it was the incorrect key name but if I change the following:
public function handle()
{
//
Redis::throttle('queues:processjob')->allow(1)->every(60)->then(function(){
Storage::disk('local')->append('testFile.txt',date("Y-m-d H:i:s"));
}, function (){
return $this->release(10);
});
}
to this:
public function handle()
{
//
Redis::funnel('queues:processjob')->limit(1)->then(function(){
Storage::disk('local')->append('testFile.txt',date("Y-m-d H:i:s"));
}, function (){
return $this->release(10);
});
}
then it all works fine.
My thoughts...
Something tells me that the issue is that the redis key is of type "list" and that the jobs are all under a single list. That being said, if it didn't work this way, how would we throttle a queue as the throttle requires a unique key.
For anybody else that is having issues attempting to get this to work and is getting the same issue as I was, this is what resolved my issues:
The Fault
I assumed that Redis::throttle('queues:processjob') was meant to be referring to the queue that you wanted to be throttled. However, after some re-reading of the documentation and testing of the code, I realized that this was not the case.
The Fix
Redis::throttle('queues:processjob') is meant to point to it's own 'holding' queue and so must be a unique Redis key name. Therefore, changing it to Redis::throttle('throttle:queues:processjob') worked fine for me.
The workings
When I first looked in to this, I assumed that that Redis::throttle('this') throttled the queue that you specified. To some degree this is correct but it will not work if the job was created via another means.
Redis::throttle('this') actually creates a new 'holding' queue where the jobs go until the condition(s) you specify are met. So jobs will go to the queue 'this' in this example and when the throttle trigger is released, they will be passed to the queue specified in their execution code. In this case, 'queues:processjob'.
I hope this helps!

Quartz .NET - Prevent parallel Job Execution

I am using Quartz .NET for job scheduling.
So I created one job class (implementing IJob).
public class TransferData : IJob
{
public Task Execute(IJobExecutionContext context){
string tableName = context.JobDetail.JobDataMap.Get("table");
// Transfer the table here.
}
}
So I want to transfer different and multiple tables. For this purpose I am doing something like this:
foreach (Table table in tables)
{
IJobDetail job = JobBuilder.Create<TransferData>()
.WithIdentity(new JobKey(table.Name, "table_transfer"))
.UsingJobData("table", table.Name)
.Build();
ITrigger trigger = TriggerBuilder.Create()
.WithIdentity(new TriggerKey("trigger_" + table.Name, "table_trigger"))
.WithCronSchedule("*/5 * * * *")
.ForJob(job)
.Build();
await this.scheduler.ScheduleJob(job, trigger);
}
So every table should be transfered every 5 minutes. To achieve this I create several jobs with different job names.
The question is: how to prevent the parallel job execution for the same jobName? (e.g. the previous run takes longer for one table, so I do not want to start the next transfer for the same table.)
I know about the attribute #DisallowConcurrentExecution, but this is used to prevent the parallel execution for the same Job class. I do not want to write an extra Job class per table, because the "main" code for the transfer is always the same, the one and only difference is the table name. So I want to use the same job class for this purpose.
The Quatz .NET documentation is a little bit confusing.
DisallowConcurrentExecution is an attribute that can be added to the
Job class that tells Quartz not to execute multiple instances of a
given job definition (that refers to the given job class)
concurrently. Notice the wording there, as it was chosen very
carefully. In the example from the previous section, if
“SalesReportJob” has this attribute, than only one instance of
“SalesReportForJoe” can execute at a given time, but it can execute
concurrently with an instance of “SalesReportForMike”. The constraint
is based upon an instance definition (JobDetail), not on instances of
the job class. However, it was decided (during the design of Quartz)
to have the attribute carried on the class itself, because it does
often make a difference to how the class is coded.
Source: https://www.quartz-scheduler.net/documentation/quartz-3.x/tutorial/more-about-jobs.html
But if you read the API documentation, it's says: the bold text is important!
An attribute that marks a IJob class as one that must not have
multiple instances executed concurrently (where instance is based-upon
a IJobDetail definition - or in other words based upon a JobKey).
Source: https://quartznet.sourceforge.io/apidoc/3.0/html/
In other words: the DisallowConcurrentExecution attribute works for my purposes.

How does a Laravel queued job handle a deleted model as input?

What happens if a Laravel queued job is passed an Eloquent model as input, but the model is deleted before the job gets run in the queue?
For example, I am building an eCommerce site with Laravel 5.2 where a customer can enter addresses and payment methods. A payment method belongs to an address. But if a customer tries to delete an address, rather than cascading down and deleting any payment methods that are associated with it, I soft delete the address by marking it as disabled. That way, the payment method can still be used until the customer updates the billing address associated with it.
However, if the payment method is deleted and it references an address that has been soft-deleted, I want to do some garbage collection and delete the address from the database. This doesn't need to happen synchronously, so I wrote a simple queueable job to accomplish this. The handle method looks like this:
public function handle(PaymentMethodRepository $paymentMethodRepository, AddressRepository $addressRepository)
{
$billingAddress = $paymentMethodRepository->address($this->paymentMethod);
if ( ! $billingAddress->enabled) {
$addressRepository->delete($billingAddress);
}
}
I dispatch this job in the destroy method of the PaymentMethodsController. However, if the payment method passed to the job is deleted from the database before the job gets executed in the queue, will the job fail?
I'm still developing the site so I don't have a server to deploy and test out what happens. I know that the model gets serialized to be put in the queue, but I wonder if an issue would occur when the model is restored to execute the job.
Yes, the job will fail if the "serialized model" is deleted before the job is executed. The model is not really serialized - the job stores the model class and model identifier and fetches the model before execution.
To get around this, you could store the primary key of the model in the job and then when executing the job, check to see if the record exists:
class DeleteAddressJob extends Job implements ShouldQueue
{
private $addressId;
public function __construct(int $addressId)
{
$this->addressId = $addressId;
}
public function handle(AddressRepository $addressRepository)
{
$address = $addressRepository->find($this->addressId);
if (is_null($address)) {
// Address doesn't exist. Complete job...
return;
}
// Delete the address...
}
}

Resources