Users are able to set up a marketing email send time within my app for random dates as they need them. It is crucial that the emails start to go out exactly when they are scheduled. So, the app needs to create something that fires one time for a specific group of emails at a specific date and time down to the minute. There will be many more dates for other sends in the future, but they are all distinct (so need something other than 'run every xxx date')
I could have a Scheduler task that runs every minute that looks at the dates of any pending tasks in the database and moves to the command that sends those that need sending. But I'm running a multi-tenanted app -- likely not overlap, but seems like a huge hit to multiple databases every minute for every tenant to search through potentially thousands of records per tenant.
What I need (I think) is a way to programmatically set a schedule for each send date, and then remove it. I liked this answer or perhaps using ->between($startMin, $endMin), without the every xxx instruction, or even using the cron function within Scheduler to set up one single time for each group that needs to be sent. Not sure I'm using this the way it was intended though?
Would this even work? I programmatically created a schedule from within a test method, and it was added to the schedule queue based on the dump of the $schedule I created, showing all schedules - but it did not show up via this method found from this answer:
$schedule = app()->make(\Illuminate\Console\Scheduling\Schedule::class);
$events = collect($schedule->events())->filter(function (\Illuminate\Console\Scheduling\Event $event) {
return stripos($event->command, 'YourCommandHere');
});
It also did not output anything, so I'm wondering if programmatically creating a schedule outside of Kernel.php is not the way to go.
Is there another way? I don't know scheduler well enough to know if these one-off schedules permanently remain somewhere, are they deleted after their intended single use, taking up memory, etc?
Really appreciate any help or guidance.
Related
Background
A workflow I have allows users to book meetings. The time/day of the meeting is stored in a table along with other details like the email address.
I send them an email reminder 30 minutes in advance.
Problem
In order to send them an email, a recurring event is set up once a week to go through the table and schedule the email to be sent on time - 30 minutes.
I've added the ability to reschedule the meeting. The problem that creates is that the emails are already scheduled, so users get the reminders at the original time, which is confusing.
What I want to do
I want to be able to send them the email at the rescheduled time, but there are technical limitations to the platform I use, which are:
I cannot set up cron/recurring more frequently than every day. This would probably be better than every week, but if someone rescheduled within the day, they would still get the wrong email.
I cannot remove scheduled events - so any recurring events-based workflow would still send the original email.
I know - this is pretty limiting, but am I even approaching this in the right way?
given your constraints, I'd probably go with 'resign'.
But in all seriousness, if you can't remove scheduled events (and I'm guessing you can't 'move' them because this is too advanced for your CTO to get their head around) then the only way I see it is to break the email send process into two steps - send scheduled event to PROXY in-front of your email sender, check if there is another event (i.e. can you add some 'cancelled/moved to data to the original one) and if so don't send it.
I am working on a project where the objective is to keep track of whether a client has uploaded data within their expected time-window or not. These windows can occur daily, weekly, bi-weekly, bi-monthly, quarterly... etc. I'm currently using the CRON syntax to describe these schedules which kind-of works, but it falls short in some scenarios, for example the bi-weekly one.
I was wondering if there is some industry standard way to efficiently describe and store schedules that I don't know of. I tried to educate myself via Google, but when I put the word schedule into the mix, it always assumes I'm trying to schedule some task, but I only want to calculate dates based on these schedule definitions and compare the incoming file dates with them.
Thank you
We are running our website on the AWS serverless infrastructure through Laravel vapor now. It runs on a serverless Aurora database and uses Amazon SQS as the queue driver. Basically the defaults for a Laravel Vapor deploy.
We are currently running into a performance issue that while we are trying to resolve it, has us smash our head into other walls we meet on our journey. So here we go.
What we want to do
Our customers can have various types of subscriptions that need to be renewed. These subscriptions come in all kinds of flavours, can have different billing intervals, different currencies and different prices.
On our customer dashboard we want to present them with a nice card informing the customer about their upcoming renewals, with a good estimate of what the cost will be, so they can make sure they have enough credits:
As our subscriptions come with different flavours, we have one table that holds the basic subscription details, such as net_renewal_date. It also references a subscriptionable_id of subscriptionable_type which is a morph relation ship to the different types of models that a user can have a subscription for.
Our first attempt
In our first attempt, we basically added a REST endpoint which fetched the upcoming renewal forecast.
It basically took all Subscriptions that were up for renewal the coming 30 days. For each of those items, we would calculate the current price in its currency, and add tax caculations.
That was then returned into a collection that we further used to:
1/ calculated the total per currency
2/ filter that collection for items within the next 14 days, and calculate the same total per currency.
We would then basically just convert the different amounts to our base currency (EUR) and return the sum thereof.
This worked great for the vast majority of our clients. Even customers with 100 subscriptions were no issue at all.
But then we migrated one of our larger customers to the new platform. He basically had 1500+ subcsriptions renewing in the upcoming 30 days, so that didn't go well...
Our second attempt
Because going through the above code simply doens't work in an acceptable amount of time, we decided we had to move the simulation calculation into a seperate job.
We added an attribute to the subscriptions table and called it 'simulated_renewal_amount'
This job would need to run every time when:
- a price changes
- the customer's discount would change (based on his loyalty, we provide seperate prices
- the exchanges rates change.
So the idea was to listen for any of these changes, and then dispatch a job to recalculate the simulated amount to any of the involved subscriptions. This however means that a change in an exchange rate for instance can easily trigger 10,000 jobs to be processed.
And this is where it becomes tricky.
Even though running just one job only takes less than 1200ms in most cases, it seems that dispatching a lot of jobs that need to do the same calculations for a set of subscriptions is causing jobs running 60+ seconds when they are being aborted.
What is the best practice to setup such a queued job? Should I just created one job in stead and process them sequentially?
Any insights on how we can best set this up to start with, would be very welcome. We've played a lot with it, and it always seems to be ending up with the same kind of issues.
FYI - we host the site on laravel vapor, so serverless on AWS infrastructure with an Aurora database.
We have got the same issue. Vapor supports multiple queues but it does not allow you to set job concurrency on a per queue basis, so its not very configurable for drip feeding lots of jobs. We have solved this by making a seeder job that pulls out serialized jobs from an "instant jobs" table. We added a sleep loop also to allow granular processing throughout the whole minute (a new seeder job is scheduled each minute).
public function handle()
{
$killAt = Carbon::now()->addSeconds(50);
do{
InstantJob::orderBy('id')->cursor()->each(function(InstantJob $job){
if($job->isThrottled()){
return true;
}
$job->dispatch();
});
sleep(5);
} while (Carbon::now()->lessThan($killAt));
}
The throttle, if you are interested works off a throttle key (job group/name etc.) and looks like:
public function isThrottled(): bool
{
Redis::connection('cache')
->throttle($this->throttle_key)
->block(0)
->allow(10) //jobs
->every(5) // seconds
->then(function () use(&$throttled) {
$throttled = false;
}, function () use(&$throttled){
$throttled = true;
});
return $throttled;
}
This actually solves our problem of drip feeding jobs onto the queue without actually starting them.
One question for you... We are currently using a small RDS instance and we get a lot of issues with too many concurrent connections. Do you see this issue with serverless db's? Do they scale fast enough to ensure no drop outs?
Goal:
I want to check if a specific job can be processed every minute.
For that, I want to use Task Scheduling.
However, I'm not sure how to solve it with my API end goal of:
SmsNotificaton::send($user, new OrderReceived($order));
How would you do this?
Explination:
I want to save some bucks by not sending an SMS directly to the user, only after 15 minutes went past and the status of the order didn't change.
Inside of new OrderReceived($order) I'm going to have a handle method. That method simply checks if the order status has changed or not. If changed, we send an SMS via my SmsGateway.
Problem:
How would you save the data of $user and new OrderReceived($order) in the database, so that you can call a cron command every minute, to fetch the old data of $user and new OrderReceived($order) again, to check if it is sendable?
It should be very similar to how the Laravel queue system works, but I tried to understand how serialization works, but I cannot grasp how the dispatcher saves the data and so on.
What you want to do is use Jobs & Queues built-in in laravel
You will need to create a new Job for example CheckStatusThenSendSMSJob() which when dispatched will check if order status changed or it should send message
Then instead of sending a message you need to Dispatch A job with a delay
CheckStatusThenSendSMSJob($order)::dispatch->delay(now()->addMinutes(15));
Check more about queues and delayed dispatch in laravel documentation: https://laravel.com/docs/5.8/queues#delayed-dispatching
Also make sure that you actually setup your queue to run in background (in a different process) by either using redis driver or database driver for queues ( By default laravel installation runs queues once they are called synchronously)
I want to send emails to various users based on the schedules they have set.
I read about beanstalkd, queues and Delayed Message Queueing and for now it looks like fitting in:
$when = Carbon::now()->addMinutes($minutes); // i can calculate minutes at this moment
\Mail::to($user)->later($when, new \App\Mail\TestMail);
But i'm not quite sure on few things:
User can cancel a future schedule. In that case how do i cancel an email that's suppose to send in future. Can i set condition somewhere that gets checked before sending the actual email? Tried return false on handle method of \App\Mail\TestMail and it started throwing error
Am i using the right approach. I also read about Scheduler but i don't get how i am going to cancel future emails(if they need to be)
There are many ways to approach this. Personally I would queue the emails on a schedule rather than adding them to the queue for later.
So you run a scheduled task once a day (or hour, or minute) which runs a query to select which users require an email, then using that result set, you add a job to the queue for each result.
This way, if a user unsubscribes, you don't have to worry about removing already queued jobs.
Laravel offers quite a nice interface for creating scheduled jobs (https://laravel.com/docs/5.4/scheduling) which can then be called via a cronjob.