An application that I'm making will allow users to set up automatic email campaigns to email their list of users (up to x per day).
I need a way of making sure that this is throttled so too many aren't sent within some range. Right now I'm trying to work within the confines of a free Mailtrap plan. But even on production using Sendgrid, I want a sensible throttle.
So say a user has set their automatic time to 9am and there are 30 users eligible to receive requests on that date and time. Every review_request gets a record in the DB. Upon Model creation, an event listener is triggered to then dispatch a job.
This is the handle method of the job that is dispatched:
/**
* Execute the job.
*
* #return void
*/
public function handle()
{
Redis::throttle('request-' . $this->reviewRequest->id)
->block(0)->allow(1)->every(5)
->then(function () {
// Lock obtained...
$message = new ReviewRequestMailer($this->location, $this->reviewRequest, $this->type);
Mail::to($this->customer->email)
->send(
$message
);
}, function () {
// Could not obtain lock...
return $this->release(5);
});
}
the above is taken from https://laravel.com/docs/8.x/queues#job-middleware
"For example, consider the following handle method which leverages Laravel's Redis rate limiting features to allow only one job to process every five seconds:"
I am using Horizon to view the jobs. When I run my command to send emails (about 25 requests to be sent), all jobs seems to process instantly. Not 1 every 5 seconds as I would expect.
The exception for the failed jobs are:
Swift_TransportException: Expected response code 354 but got code "550", with message "550 5.7.0 Requested action not taken: too many emails per second
Why does the above Redis throttle not process a single job every 5 seconds? And how can I achieve this?
Related
Configuration
I'm using Laravel 8 with sentry/sentry-laravel plugin.
There is a Job that works just fine 99% of time. It retries N times in case of any problems due to:
public $backoff = 120;
public function retryUntil()
{
return now()->addHours(6);
}
And it simply calls some service:
public function handle()
{
// Service calls some external API
$service->doSomeWork(...);
}
Method doSomeWork sometimes throws an exception due to network problems, like Curl error: Operation timed out after 15001 milliseconds with 0 bytes received. This is fine due to automatic retries. In most cases next retry will succeed.
Problem
Every curl error is sent to Sentry. As an administrator I must check every alert, because this job is pretty important and I can't miss actually failed job. For example:
There is some network problem that is not resolved for an hour.
Application queues a Job
Every 2 minutes application generates similar message to Sentry
After network problems resolved job succeeds, so no attention required
But we are seing dozens of errors, that theoretically could be ignored. But what if there an actual problem in that pile and I will miss it?
Question
How to make that only "final" job fail would send a message to Sentry? I mean after 6 hours of failed retries: only then I'd like to receive one alert.
What I tried
There is one workaround that kind of "works". We can replace Exception with SomeCustomException and add it to \App\Exceptions\Handler::$dontReport array. In that case there are no "intermediate" messages sent to Sentry.
But when job finally fails, Laravel sends standard ... job has been attempted too many times or run too long message without details of actual error.
I'm using Laravel Jobs to pull data from the Stripe API in a paginated way. Basically, each job gets a "brand id" (a user can have multiple brands per account) and a "start after" parameter. It uses that to know which stripe token to use and where to start in the paginated calls (this job call itself if more stripe responses are available in the pagination). This runs fine when the job is started once.
But there is a use case where a user could add stripe keys to multiple brands in a short time and that job class could get called multiple times concurrently with different parameters. When this happens, whichever process is started last overwrites the others because the parameters are being overwritten to just the last called. So if I start stripe job with brand_id = 1, then job with brand_id = 2, then brand_id = 3, 3 overwrites the other two after one cycle and only 3 gets passed for all future calls.
How do I keep this from happening?
I've tried static vars, I've tried protected, private and public vars. I thought might be able to solve it with dynamically created queues for each brand, but this seems like a huge headache.
public function __construct($brand_id, $start_after = null)
{
$this->brand_id = $brand_id;
$this->start_after = $start_after;
}
public function handle()
{
// Do stripe calls with $brand_id & $start_after
if ($response->has_more) {
// Call next job with new "start_at".
dispatch(new ThisJob($this->brand_id, $new_start_after));
}
}
According to Laravel Documentation
if you dispatch a job without explicitly defining which queue it
should be dispatched to, the job will be placed on the queue that is
defined in the queue attribute of the connection configuration.
// This job is sent to the default queue...
dispatch(new Job);
// This job is sent to the "emails" queue...
dispatch((new Job)->onQueue('emails'));
However, pushing jobs to multiple queues with unique names can be especially useful for your use case.
The queue name may be any string that uniquely identifies the queue itself. For example, you may wish to construct the queue name based on the uniqid() and $brand_id.
E.g:
dispatch(new ThisJob($this->brand_id, $new_start_after)->onQueue(uniqid() . '_' . $this->brand_id));
I'm developing a binary multilevel marketing system in Laravel, at the registration time there we have to perform a task to entries for many types of bonus for each parent nodes of a new user. This task is time-consuming.
No one user want to see buffering and task taking more than 30 second that is not the right way.
I want to run this mechanism in the background and send a success message that your account created successfully.
You could use observers that trigger queued jobs.
After the user does an action on a model, the observers create queued jobs in the background. While the queue is being processed the user can continue working.
either implement laravel job and queues or use https://github.com/spatie/async.
you can invoke sub processes to make your task
use Spatie\Async\Pool;
$pool = Pool::create();
foreach ($things as $thing) {
$pool->add(function () use ($thing) {
// Do a thing
})->then(function ($output) {
// Handle success
})->catch(function (Throwable $exception) {
// Handle exception
});
}
$pool->wait();
End Goal
The aim is for my application to fire off potentially a lot of emails to the Redis queue (This bit is working) and then Redis throttle the processing of these to only a set number of emails every selected number of minutes.
For this example, I have a test job that appends the time to a file and I am attempting to throttle it to once every 60 seconds.
The story so far....
So far, I have the application successfully pushing a test amount of 50 jobs to the Redis queue. I can log in to Horizon and see these 50 jobs in the "processjob" queue. I can also log in to redis-cli and see 50 sets under the list key "queues:processjob".
My issue is that as soon as I attempt to put the throttle on, only 1 job runs and the rest fail with the following error:
Predis\Response\ServerException: ERR Error running script (call to f_29cc07bd431ccbf64637e5dcb60484560fdfa2da): #user_script:10: WRONGTYPE Operation against a key holding the wrong kind of value in /var/www/html/smhub/vendor/predis/predis/src/Client.php:370
If I remove the throttle, all works file and 5 jobs are instantly ran.
I thought maybe it was the incorrect key name but if I change the following:
public function handle()
{
//
Redis::throttle('queues:processjob')->allow(1)->every(60)->then(function(){
Storage::disk('local')->append('testFile.txt',date("Y-m-d H:i:s"));
}, function (){
return $this->release(10);
});
}
to this:
public function handle()
{
//
Redis::funnel('queues:processjob')->limit(1)->then(function(){
Storage::disk('local')->append('testFile.txt',date("Y-m-d H:i:s"));
}, function (){
return $this->release(10);
});
}
then it all works fine.
My thoughts...
Something tells me that the issue is that the redis key is of type "list" and that the jobs are all under a single list. That being said, if it didn't work this way, how would we throttle a queue as the throttle requires a unique key.
For anybody else that is having issues attempting to get this to work and is getting the same issue as I was, this is what resolved my issues:
The Fault
I assumed that Redis::throttle('queues:processjob') was meant to be referring to the queue that you wanted to be throttled. However, after some re-reading of the documentation and testing of the code, I realized that this was not the case.
The Fix
Redis::throttle('queues:processjob') is meant to point to it's own 'holding' queue and so must be a unique Redis key name. Therefore, changing it to Redis::throttle('throttle:queues:processjob') worked fine for me.
The workings
When I first looked in to this, I assumed that that Redis::throttle('this') throttled the queue that you specified. To some degree this is correct but it will not work if the job was created via another means.
Redis::throttle('this') actually creates a new 'holding' queue where the jobs go until the condition(s) you specify are met. So jobs will go to the queue 'this' in this example and when the throttle trigger is released, they will be passed to the queue specified in their execution code. In this case, 'queues:processjob'.
I hope this helps!
I'm trying to send multiple emails with queues (beanstalkd). My application sends some number of emails and then I receive timeout exception.
foreach ($emails as $e) {
Mail::queue('emails.invite', ["username" => Auth::user()->username, "grupa" => $naziv, "id" => $id, "email" => $e], function($message) use ($e){
$message->to($e)->subject("Pridruži nam se!");
});
}
Is there a way to put all emails to queue, so when system is available the email should be sent.
EDIT: Full message for timeout exception:
{"error":{"type":"Symfony\\Component\\Debug\\Exception\\FatalErrorException","message":"Maximum execution time of 30 seconds exceeded","file":"\/home\/forge\/default\/vendor\/nikic\/php-parser\/lib\/PHPParser\/NodeAbstract.php","line":110}}
How are you doing it? If you're using beanstalk (or any queue), you're doing stuff from your application (producer, send to the queue) and from the worker process that consumes data from the queue and sends the email.
The producer just puts the email into beanstalk, so easy...
The consumer should be a long running process and it should be executed from cli, with no maximum time execution (you have to tweak the php.ini of php-cli). In the loop, you should check if there's something new in the queue and send the email.
Basically, sounds like your problem is that the consumer part has maximum execution time set, so it can't consume the emails in a while loop after X seconds. Tweak that and make sure there's no limit.