I have an SaaS application where each paying customer may have thousands of members they may want to send emails to every now and then.
For now, simple BCC sending via AWS SES have done the trick, but now I am looking at sending personalized emails so I must be able to send the emails one by one.
SES does not have any queue system as per my knowledge, you must make an API call per email. In short, it takes forever to send a batch (my limit is 14 per second), and the user cannot close the page while it is executing (even AJAX calls stop executing if you leave the page, am I right?).
I was thinking of building a system where I store the emails in a database table and then either:
1) Use a CRON that executes every 5 seconds or so, grab a few emails and send them.
2) Execute an AJAX script each 5 seconds that grabs the emails for said logged in customer in a batch ONLY and send them out, but again, if the customer logs out while it executes chances are that specific a batch is interrupted (the remaining ones would still keep sending the next time the customer logs in).
Does any have any better ideas? Or, which of the two above would be preferred?
You should use templates and the SendBulkTemplatedEmail endpoint that AWS introduced a few months ago: https://aws.amazon.com/blogs/ses/introducing-email-templates-and-bulk-sending/.
That way you can send up to 50 personalized emails with a single SES API call. So 700 with 14 calls.
You shouldn't consider queuing them up in a user's browser and sending them by making a series of AJAX requests though. You should only send one Ajax request to start a job. In most server-side languages (any I can think of) you can respond to an HTTP request and still continue doing processing after responding. You can also implement a progress checker in a multitude of ways.
Use a cronjob that sends to the SES SMTP server. This way you can personalize the emails and also control how many emails to send. Your cronjob can sleep in between each batch of emails.
You can use celery to run background job. A user submits a request on a webpage which starts a background job through celery. The background job take care of sending emails. Once sending emails is completed, inform the user by email.
http://www.celeryproject.org/
Related
I'm using PayPal webhooks to get subscription information automatically.
However, we have to wait about 20 seconds between the payment and the subscription activation.
Is it because of the sandbox environment? Is the production environment faster?
This is important because the customers have to wait and if waiting time could be avoided, it would be better.
The sandbox is slower in general, but you will need to test yourself in live -- and the speed of asynchronous notifications vary in different conditions.
If you need a faster notification, what you can do is have the client-side onApprove event call your server (with a JS fetch similar to this demo, plus a body payload if desired), and have the server route that handles that fetch use the Subscriptions API to get the status of the subscription, and see whether it is in fact active in that API response direct from PayPal.
Such a client-side trigger of a server route would happen in parallel to waiting for the webhook notification, so whichever completes first will mark the subscription as active in your records. This way you are not relying on either the client-side trigger nor waiting for the webhook, but rather whichever happens first.
I want to inform the seller, that the buyer is coming soon (about 2 hours before pickup time) via mail.
I would normally do it the hard way with CRON and a database table. Checking hourly if I find an order with pickup time minus 2 hours, only then sending the mail out.
Now, I would like to know if you would recommend using Queueing Jobs for sending Mails out.
With
$when = now()->addDays(10); //I would dynamically set the date
Mail::to($order->seller())
->later($when, new BuyerIsComing($order));
I can delay the delivery of a queued email message.
But how safe would this be? Especially, if someone is ordering something but is picking it up in let us exaggerate two months?
Is the Laravel queueing system rigid enough to behave correctly after long delays (i.e. 2 months)?
Edit
I'm using Redis for Queueing
You actually have nothing to worry about. Sending mail usually increases the response time of your application, so it's a good thing you want to delay the sending.
Queues are the way to go and it's pretty easy to setup in Laravel. Laravel supports a couple of them out of the box. I would advise you start with database and then try beanstalk etc.
Lastly and somehow more importantly, use a process manager like Supervisor to monitor and maintain your queue workers...
Take a look at https://laravel.com/docs/5.7/queues for more insight.
Cheers.
If by safe, you mean reliable, then it would be little different than sending an email immediately. If there's ever a possibility that your server "hiccups" and doesn't send an email, that possibility would be the same now as 10 minutes from now. Once the job is in the queue, it is persisted until completion (unless you use a memory-based driver, like Redis, which could get reset if the server reboots).
If you are using a database queue driver or remote, the log of queued jobs will remain even if the server is unavailable for a short period of time. Your queue will be honored even if the exact time stamp for when you want to send the job has expired. For instance if you schedule to send an email at 1:00pm but your server is down at that exact moment, when it comes back online it will still see the job because it is stored as incomplete and the time for the job is in the past, which will trigger the execution of the job at the next time your queue worker checks the job list.
Of course, this assumes that you have your queue worker set up to always check jobs and automatically restart, even after a server failure, but that's a different discussion with lots of solutions...such as those shown here.
If you're using database driver with Laravel queues to process your email then you don't need to worry about anything.
Jobs are only removed from Jobs table if they are successfully completed otherwise their next attempt time is set which is few minutes in future and they are executed again (if your queue worker is online).
So its completely safe to use Laravel queues
I have built an application that allows users to buy plane tickets, now I want to be able to set reminders in my application as in a script that continuously runs in the background checking the database.
Reminders should be sent via SMS a day before travel, I have covered the SMS part already. My problem now is generating a script that runs in the background and sends a reminder 1 day early to all customers. I have never implemented something like this before.
What I would do:
create a function to loop through your records and get the ones that are one day before take of. And set a cron job on that route
Then send emails to all that users
If you run the cron at night you can queue the mail job so mails will be send at a later time (morning).
How can I trigger a shell script on an email arrival that extracts the mail in a text file? I want to extract the information in the mail, process it to determine the request and send an automated response to that request. The mail will basically consist of a data request and the response will have the requested data in a text file attached to it.
Look into the documentation of your MTA (mail transfer agent). Many of them allow to run scripts or hooks when mail arrives and certain other conditions are met.
If you're using Linux and want a pure client solution (i.e. independent of the mail server software), then you should look at procmail. The documentation contains lots of useful tips and hints how to set up the tool (like performance considerations) and how to properly set up the environment so your script executes correctly.
It also contains examples like a service which responds to "ping" mails.
I got some errors about sending mails too quick in 15 minutes when sending mails.
It seems there is a speed limit of sending mails.
But how many mails can be sent in 15 minutes?
Is this a standard or private rule based on vendor?
It depends on both your mail service provider, and also the target system. For example gmail will let you send 500/1000 (or more) emails per day, per account depending on your agreement with them.