I want to send emails to various users based on the schedules they have set.
I read about beanstalkd, queues and Delayed Message Queueing and for now it looks like fitting in:
$when = Carbon::now()->addMinutes($minutes); // i can calculate minutes at this moment
\Mail::to($user)->later($when, new \App\Mail\TestMail);
But i'm not quite sure on few things:
User can cancel a future schedule. In that case how do i cancel an email that's suppose to send in future. Can i set condition somewhere that gets checked before sending the actual email? Tried return false on handle method of \App\Mail\TestMail and it started throwing error
Am i using the right approach. I also read about Scheduler but i don't get how i am going to cancel future emails(if they need to be)
There are many ways to approach this. Personally I would queue the emails on a schedule rather than adding them to the queue for later.
So you run a scheduled task once a day (or hour, or minute) which runs a query to select which users require an email, then using that result set, you add a job to the queue for each result.
This way, if a user unsubscribes, you don't have to worry about removing already queued jobs.
Laravel offers quite a nice interface for creating scheduled jobs (https://laravel.com/docs/5.4/scheduling) which can then be called via a cronjob.
Related
Background
A workflow I have allows users to book meetings. The time/day of the meeting is stored in a table along with other details like the email address.
I send them an email reminder 30 minutes in advance.
Problem
In order to send them an email, a recurring event is set up once a week to go through the table and schedule the email to be sent on time - 30 minutes.
I've added the ability to reschedule the meeting. The problem that creates is that the emails are already scheduled, so users get the reminders at the original time, which is confusing.
What I want to do
I want to be able to send them the email at the rescheduled time, but there are technical limitations to the platform I use, which are:
I cannot set up cron/recurring more frequently than every day. This would probably be better than every week, but if someone rescheduled within the day, they would still get the wrong email.
I cannot remove scheduled events - so any recurring events-based workflow would still send the original email.
I know - this is pretty limiting, but am I even approaching this in the right way?
given your constraints, I'd probably go with 'resign'.
But in all seriousness, if you can't remove scheduled events (and I'm guessing you can't 'move' them because this is too advanced for your CTO to get their head around) then the only way I see it is to break the email send process into two steps - send scheduled event to PROXY in-front of your email sender, check if there is another event (i.e. can you add some 'cancelled/moved to data to the original one) and if so don't send it.
Users are able to set up a marketing email send time within my app for random dates as they need them. It is crucial that the emails start to go out exactly when they are scheduled. So, the app needs to create something that fires one time for a specific group of emails at a specific date and time down to the minute. There will be many more dates for other sends in the future, but they are all distinct (so need something other than 'run every xxx date')
I could have a Scheduler task that runs every minute that looks at the dates of any pending tasks in the database and moves to the command that sends those that need sending. But I'm running a multi-tenanted app -- likely not overlap, but seems like a huge hit to multiple databases every minute for every tenant to search through potentially thousands of records per tenant.
What I need (I think) is a way to programmatically set a schedule for each send date, and then remove it. I liked this answer or perhaps using ->between($startMin, $endMin), without the every xxx instruction, or even using the cron function within Scheduler to set up one single time for each group that needs to be sent. Not sure I'm using this the way it was intended though?
Would this even work? I programmatically created a schedule from within a test method, and it was added to the schedule queue based on the dump of the $schedule I created, showing all schedules - but it did not show up via this method found from this answer:
$schedule = app()->make(\Illuminate\Console\Scheduling\Schedule::class);
$events = collect($schedule->events())->filter(function (\Illuminate\Console\Scheduling\Event $event) {
return stripos($event->command, 'YourCommandHere');
});
It also did not output anything, so I'm wondering if programmatically creating a schedule outside of Kernel.php is not the way to go.
Is there another way? I don't know scheduler well enough to know if these one-off schedules permanently remain somewhere, are they deleted after their intended single use, taking up memory, etc?
Really appreciate any help or guidance.
Goal:
I want to check if a specific job can be processed every minute.
For that, I want to use Task Scheduling.
However, I'm not sure how to solve it with my API end goal of:
SmsNotificaton::send($user, new OrderReceived($order));
How would you do this?
Explination:
I want to save some bucks by not sending an SMS directly to the user, only after 15 minutes went past and the status of the order didn't change.
Inside of new OrderReceived($order) I'm going to have a handle method. That method simply checks if the order status has changed or not. If changed, we send an SMS via my SmsGateway.
Problem:
How would you save the data of $user and new OrderReceived($order) in the database, so that you can call a cron command every minute, to fetch the old data of $user and new OrderReceived($order) again, to check if it is sendable?
It should be very similar to how the Laravel queue system works, but I tried to understand how serialization works, but I cannot grasp how the dispatcher saves the data and so on.
What you want to do is use Jobs & Queues built-in in laravel
You will need to create a new Job for example CheckStatusThenSendSMSJob() which when dispatched will check if order status changed or it should send message
Then instead of sending a message you need to Dispatch A job with a delay
CheckStatusThenSendSMSJob($order)::dispatch->delay(now()->addMinutes(15));
Check more about queues and delayed dispatch in laravel documentation: https://laravel.com/docs/5.8/queues#delayed-dispatching
Also make sure that you actually setup your queue to run in background (in a different process) by either using redis driver or database driver for queues ( By default laravel installation runs queues once they are called synchronously)
My Laravel 5 site allows users to send emails to their contacts. My problem is that the operation times out after it loops through around 50 contacts.
An overview of my process:
Get the list of contacts
Construct an email
Loop through the list of contacts and send them each an email using a mailer service.
return email details to the controller to be saved in the db.
Chunking is pretty easy to understand, but I'm not real comfortable with the mechanics of placing a job on a queue.
To prevent the timeout, would I be better off chunking the contacts to limit the size of the loop or putting the entire job on a queue? If the entire job should be placed on a queue - can I still chunk contacts? Can a queued job still use mail::queue or is that just redundant?
Loop Code:
public function sendIndividualEmail($members, $subject, $body, $file)
{
$view = 'emails.memberMessage';
//construct email details
$mData = $this->emailConstructor($subject, $body);
//check to see if there is an attachment, if so upload it.
if (!is_null($file)){
$uploadedFile = $this->attachment->saveAttachment($file);
$mData['path'] = 'https://'.$_SERVER["HTTP_HOST"].'/uploads/attachments/'.$uploadedFile['filename'];
$mData['display'] = $uploadedFile['display'];
}
foreach($members as $member){
if (!empty($member->contact_email)){
//add member details here
$mData['email'] = $member->contact_email;
$mData['name'] = $member->contact_firstName;
//email member
$this->emailTo($view, $mData);
$mData['emailCount']++;
}
}
return $mData;
}
Mailer Service:
public function emailTo($view, $mData)
{
Mail::queue($view, $mData, function($message) use ($mData)
{
$message->from($mData['from'], $mData['fromName']);
$message->to($mData['email'], $mData['name'])->subject($mData['subject']);
$message->getHeaders()->addTextHeader('X-MC-Metadata', $mData['meta']);
if(array_key_exists('path', $mData)){
$message->attach($mData['path'], array('as' => $mData['display']));
}
});
}
A few suggestions first.
Chunking them would use less computational power, but does not offer the flexibility that a queue system presents.
Chunking scenario ---
Get All Models, for count(x) as new chunk store chunk.
Foreach chunk as items
foreach item in items
item->doSomething
-- Pagination
I might adapt the pagination library to support this instead.
I.E get a collection and paginate it, store the page number, and just do a page worth of records at a time?
-- Event Observer Queue?
Another technique would be to use the event listener system.
You can to this a number of ways, just think about when an email should be sent.
Your model would have either a column like sent_mymail_email, or you could use a mutator getSentMymailEmailAttribute() to return a boolean value of true if the email was sent (job is complete).
You would then set up a new event or latch on to an existing event.
see: How Can I See All Laravel Events Being Fired
Queue system --
A queue system would receive events and data from one server via http request . The items are then added to a big list of jobs that need to be completed. When the event comes up (it's turn), it will send a http request to somewhere on your system. Your system then interprets what the task is and accomplishes the task, when complete, a response is sent back to the queue system(typically) notifying the system of a completed task. The queue will then move on to the next task, send the request ... and so on.
Comparing this to the chunk method, you can see that your application needs to send the email in both scenarios, however it does not need to send the task, receive the task or interpret what the task is in the chunking method.
You also have less moving parts in the chunking method, however if you wanted the ability to scale and put part of your applications workload onto a secondary system or cluster to handle the jobs, you would want to go with the queue system.
The queue system depends on 3rd party services that have their own built in flaws. For example you cant define an order for these items.
This blog post explains:
http://cloudstacking.com/posts/amazon-sqs-gotchas.html
Here is another decent post about using queue systems:
http://www.isixsigma.com/industries/retail/queuing-theory-and-practice-source-competitive-advantage/
The advantage in a nutshell primarily being that you have the ability to scale to a very large amount of these requests being processed with zero impact on your front-end(application).
You can run these jobs on a separate system or cluster of systems to push the work that is needed to a server that does not impact user experience.
You are able to queue the entire job and still queue the mail, I do that in my application and don't feel it is redundant. It allows you to break up the processing into logical steps that can be completed independent of one another. For example the job can continue processing the contacts and scheduling the emails without needing to wait for them to send.
While I haven't done it myself, I believe you are able to chunk the contacts and still use a queue on the job.
Laravel 4 has a great list of features in terms of Queues. This question is with respect to the Queue method Queue.later() API Documentation the first parameter being the delay.
A Cron is basically made use of to execute recurring tasks.
If the below snippet were to be made more generic with the time being configurable as well can:
This be used as an alternative to CRON jobs?
Would this be fail safe approach to use assuming we use IronMQ
-
class SendEmail {
public function fire($job, $data)
{
//Connect to SMTP and send email
$job->delete();
//Recall the queue with a delay
Queue::later(60,'SendEmail#send', array('message' => $message));
}
}
//app/events/MailHandler.php
public class MailHandler(){
public function onMailListenerStarted(){
Queue::push('SendEmail#send', array('message' => $message));
}
}
You have to keep in mind that Queuing and Cron-tasking are 2 different things.
A cron job will start (depending on how you configure it) every exact minute.
A queue job will run after the delay time is over AND it is his turn to be processed.
So to compare this to your definition of a Cron "execute recurring tasks", the Queue does nothing like that. Jobs will simply wait in the queue, and they don't do anything. Delayed jobs will give you the advantage that it will at least wait till the time is there to send it, but it won't try and send all emails in one go. The downside if this is that it might take to long before it's send, but to prevent that, you can simply use more workers to process the queue.
And you need a script that processes the queue, which you'll most likely want to start with a cron.
Another problem i see with the approach in the code snippet, is that, if something goes wrong with adding the job back to the queue, the job will be lost, and will never be added back to the queue.
So to answer your questions:
No, Queues are not an alternative to jobs, but Queues do make data processing in cron-scripts easier
In theory this won't be fail safe approach, no matter how good your queue provider is. But it's possible to create some scripts that check if the queue is still doing everything it should, but this does require some logging (e.g. Save when the job has been run for the last time).
..2 years later..
Would this be fail safe approach?
Laravel Forge has made adding and supervising queue workers very reliable and much less painful, it's worth checking out.
So yeah don't know if fail-safe or not, but definitely more reliable than it used to be.