I parse some HTML pages and API endpoints, for example, every 5 minutes to track changes. For this purpose, I have created ParseJob where I do parsing and save changes to a database. ParseJob implements interface ShouldQueue and I have changed queue driver to Redis. In order to run the ParseJob on a regular basis, I have created ParseCommand and added it to schedule:
class ParseCommand extends Command
{
protected $signature = 'application:my-parse-command';
public function handle()
{
$this->dispatch(new ParseJob());
}
}
class Kernel extends ConsoleKernel
{
protected $commands = [
Commands\ParseCommand::class
];
protected function schedule(Schedule $schedule)
{
$schedule->command('application:my-parse-command')
->everyFiveMinutes();
}
}
And the queue worker is started as a daemon to process the queue. So, every 5 minutes ParseJob is pushed to the queue and the queue worker is processing the job.
Sometimes queue worker process crashes, freezes or for other reasons is not working. But jobs every 5 minutes are pushed into the queue. After an hour of downtime, I have 12 jobs in the queue but they are for that time irrelevant because I do not need to parse 12 times at a certain time, I want just one parse job.
So I want to set TTL for a job that works like expire command in Redis. How to do that? Or maybe you have an alternative solution?
As far a i know it is not possible to set explicitly a Job expiration into Laravel queues. A solution could be setting an expires_at property within your ParseJob and check before executing:
class ParseCommand extends Command
{
protected $signature = 'application:my-parse-command';
public function handle()
{
$this->dispatch(new ParseJob(Carbon::now()->addMinutes(5)));
}
}
then in your Job class
class ParseJob {
protected $expires_at;
public function __construct(Carbon $expires_at) {
$this->expires_at = $expires_at;
}
public function handle()
{
if(!Carbon::now()->gt($this->expires_at)) {
// Parse data
}
}
}
At larvel 8/9 in job
MyJObClass
public function retryUntil(): Carbon
{
return now()->addMinutes(10);
}
Laravel :: Queues #time-based-attempts
Related
I am using Stancl Tenancy for a multi tenant Laravel 8 app. I have this testing well in a single core, however I want to take advantage of the new feature to run tests in parallel.
I think to do this sensibly I should create a new tenant for each database that is created by the parallelization. I've been trying to do this using the ParallelTesting::setUpTestDatabase callback, but am getting some odd behaviour.
When I go on to run tests in parallel, only database 5 of 7 (and consistently this one) has a tenant created for it:
AppServiceProvider
ParallelTesting::setUpTestDatabase(function ($database, $token) {
$tenant = Tenant::create(['id'=>"testTenant$token",'name'=>"PHPUnit Process $token"]);
});
Without the call to create a model, the databases are being setup correctly:
ParallelTesting::setUpTestDatabase(function ($database, $token) {
//$tenant = Tenant::create(['id'=>"testTenant$token",'name'=>"PHPUnit Process $token"]);
Log::info("Setup called by $token My database connection is ".DB::connection()->getDatabaseName());
});
[2021-10-21 14:45:32] testing.INFO: Setup called by 4 My database connection is project_test_4
[2021-10-21 14:45:32] testing.INFO: Setup called by 3 My database connection is project_test_3
[2021-10-21 14:45:32] testing.INFO: Setup called by 7 My database connection is project_test_7
[2021-10-21 14:45:32] testing.INFO: Setup called by 6 My database connection is project_test_6
[2021-10-21 14:45:34] testing.INFO: Setup called by 5 My database connection is project_test_5
With the line to create the model uncommented, nothing is logged and only the last connection 5 gets a Tenant created for it. Subsequent tests then appear to use the correct connection as 4/5 of them fail for TenantNotIdentified (because they have no tenant or domain).
Can anyone spot the problem?
The setupTestDatabase callback is not helpful for this as the database has not yet been migrated. Multi tenancy can work with parallel testing (after a lot of work on this!) if:
DatabaseTransactions is used rather than RefreshDatabbase
You manually handle restarting database transactions for your tenant tests
You do some of the cleanup manually
Register a new service provider
class ParallelTestingMultiTenantProvider extends ServiceProvider
{
/**
* Register services.
*
* #return void
*/
public function register()
{
//
}
/**
* Bootstrap services.
*
* #return void
*/
public function boot()
{
ParallelTesting::setUpTestCase(function ($token, $testCase) {
$tenant = Tenant::firstOrCreate(['id' => "test_{$token}_tenant", 'name' => "testTenant$token"]);
$tenant->domains()->firstOrCreate(['domain'=>"testtenant$token.site.test"]);
});
ParallelTesting::tearDownProcess(function ($token) {
try {
DB::disconnect();
config(['database.connections.mysql.database'=>"site_test_{$token}"]);
DB::reconnect();
if($tenant = Tenant::find("test_{$token}_tenant")){
$tenant->delete();
}
}
catch(\Exception $e){
if($e->getCode() !== 1049){ //Intercepts database does not exist errors - where tests do not use Database traits
throw $e;
}
}
});
}
}
Add to setup() in your base test classes
class TenantTestCase extends TestCase
{
use DatabaseTransactions;
public function setUp(): void
{
parent::setUp();
if($this->isTestingInParallel()){
$token = ParallelTesting::token();
tenancy()->initialize(Tenant::find("test_{$token}_tenant"));
DB::beginTransaction(); //Needed as otherwise database transactions don't work on the tenant database
URL::forceRootUrl("https://testtenant$token.site.test");
}
else {
//Test setup for when not running in parallel
}
}
protected function tearDown(): void
{
if($this->isTestingInParallel()){
DB::rollBack();
}
else {
//Tear down if not testing in parallel
}
parent::tearDown();
}
protected function isTestingInParallel() : bool
{
return (bool)ParallelTesting::token();
}
}
This allowed me to reduce time for a whole test suite run from 1m 30 to 16s, whilst retaining the ability to run tests not in parallel for red green using if statements as above.
let's me explain example
User click button
run function trigger in Controller
User wait 30sec because MyModel::doSomeThing take long time to process
MyModel::doSomeThing do many thing. I don't want user to wait for it.
Is it possible to run MyModel::doSomeThing by don't care about result and return to user immediately?
function trigger(Request $request){
$id= $request->get('id');
MyModel::doSomeThing($id); // this one take 30 sec.
return response()->json([], 200);
}
If the result of doSomeThing() method isn't necessary for your response & can be done in background, I suggest using Events and Listeners, which will use queues to run in the background, and the user won't need to wait for this procces to finish. The process is fairly simple. Create event and it's listened with these two commands:
php artisan make:event YourEvent
php artisan make:listener YourListener --event=YourEvent
After that, register your event and listener in the App\Providers\EventServiceProvider, under the $listen array:
protected $listen = [
YourEvent::class => [
YourListener::class,
],
];
Now, when you have that sorted out, you need to build your event instance. Inside your newly created method, in the construct method, add this:
public $yourModel;
public function __construct(YourModel $yourModel)
{
$this->yourModel = $yourModel;
}
After you created your model, time to edit your listener, which will hanlde all the logic ghat you need. Inside this handle method, you will have the access to $yourModel instance that we defined in our event:
public function handle(YourEvent $event)
{
// Access your model using $event->yourModel...
YourModel::doSomeThing($event->yourModel);
}
The only thing left do to is to make your listener queueable. You can do this by adding implements ShouldQueue your listened definition:
class YourListener implements ShouldQueue
{
//
}
Now when we have everything setup, you can change your controller code to call this newly created event, and let the queue handle all the logic:
function trigger(Request $request){
$id= $request->get('id');
YourEvent::dispatch($id); //Calling event which will handle all the logic
return response()->json([], 200);
}
And that should be it. I haven't tested this code, so if you encounter any problems, let me know.
I have a table users with 5000 records (5000 users) and in my server i can just send 1000 emails every hour.
How i can send 1000 emails every hour using queue ?
or how to make queues sleep inside loop?
EmailController:
class EmailController extends Controller
{
public function sendEmail(Request $request){
$event=(object)['content' => "Hello Laravel fans",'subject' => 'Test Email'];
$users=User::all();
App\Jobs\SendReminderEmail::dispatch($users,$event)
}
}
SendReminderEmail
class SendReminderEmail implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
public $event;
public $email;
public $users;
public function __construct($users,$event)
{
$this->users = $users;
$this->event = $event;
}
public function handle()
{
foreach ($this->users as $user) {
Mail::to($user->email)->queue(new Reminder($this->event));
}
}
}
Laravel has a neat feature which fits your case perfectly, it's called Task Scheduling, instead of making the job sleep for one hour, you could, instead, call it every hour.
To do so add the job schedule to the schedule() method located on App\Console\Kernel like this:
protected function schedule(Schedule $schedule)
{
$schedule->job(new SendReminderEmail)->hourly();
}
I also would recommend you to make the job self contained, that will make this task much simpler, I'm thinking in something like this:
class SendReminderEmail implements ShouldQueue
{
use Dispatchable, InteractsWithQueue, Queueable, SerializesModels;
public $event;
public $users;
public function __construct()
{
$this->users = User::all();
$this->event = (object)['content' => "Hello Laravel fans",'subject' => 'Test Email'];;
}
public function handle()
{
foreach ($this->users as $user) {
Mail::to($user->email)->queue(new Reminder($this->event));
}
}
}
Now you can just get rid of your controller because this job will be executed every hour automatically.
Remember that you will need to run a cron on your server to check if the job needs to be executed. You can also run it manually if you want to test it using php artisan schedule:run.
Hope this helps you.
Laravel has a built-in throttle feature for rate-limited jobs. From the docs:
If your application interacts with Redis, you may throttle your queued
jobs by time or concurrency. This feature can be of assistance when
your queued jobs are interacting with APIs that are also rate limited.
For example, using the throttle method, you may throttle a given type
of job to only run 10 times every 60 seconds. If a lock can not be
obtained, you should typically release the job back onto the queue so
it can be retried later:
Redis::throttle('key')->allow(10)->every(60)->then(function () {
// Job logic...
}, function () {
// Could not obtain lock...
return $this->release(10);
});
In your case, that might look like Redis::throttle(...)->allow(1000)->every(3600)->...
If you're not using Redis, another possible solution which is specific to queued mail is to add a delay. Again, from the docs:
Delayed Message Queueing
If you wish to delay the delivery of a queued
email message, you may use the later method. As its first argument,
the later method accepts a DateTime instance indicating when the
message should be sent:
$when = now()->addMinutes(10);
Mail::to($request->user())
->cc($moreUsers)
->bcc($evenMoreUsers)
->later($when, new OrderShipped($order));
Using this method would require you to calculate the delay for each email during the loop that dispatches emails to the queue. For example, every 1000th iteration of the loop you could increase the delay by 1 hour.
If you're using the Amazon SQS queue service, the same maximum delay of 15 minutes that applies to other queued jobs might also apply here (I'm not sure on this part), in which case you would have to come up with another solution for checking if you're over your rate limit and releasing the job back to the queue.
I have hooked up Redis with Laravel for queuing emails and all is fine...
but in dev environment I use mailtrap.io (free version).
And the problem is that mailtrap allows to receive only 2 emails per second, so I never get all the emails that are queued, because redis sends emails like crazy... maybe 10 per/s
Is there a way somehow to slow down the queue so that it sends max 2 mails per second?
Yes It is a solution and they're called jobs :)
you can create a file to send emails exclusively in the Jobs Folder and a class like this
class SendPushNotification extends Job implements ShouldQueue
{
use InteractsWithQueue, SerializesModels;
protected $pushNotification;
/**
* Create a new job instance.
*
* #return void
*/
public function __construct(PushNotification $pushNotification)
{
$this->pushNotification = $pushNotification;
}
/**
* Execute the job.
*
* #return void
*/
public function handle()
{
$this->pushNotification->send();
}
}
And then call the class in the controller
$this->dispatch(new SendPushNotification($pushNotification))->delay(1);
Delay is for seconds you can create a constructor with an array to receive two emails o many you want maybe some var than can be changed by the .env to change the number of emails per second
I am trying to send emails in laravel 5.1 by using queues. When running queue listen command on terminal,
php artisan queue:listen
Displays below error on terminal,
[Illuminate\Contracts\Queue\EntityNotFoundException]
Queueable entity [App\Setting] not found for ID [].
Values of jobs table is not process. Any idea ?
How can I process my queue ?
I know this question is a few months old, but I'd like to add an observation of mine while encountering this very same error message. It is due to the EventListener (interface of ShouldQueue in this example for asynchronous) not being able to resolve a dependant variable correctly (out of scope or not included in scope of Event object passed through the handle(Event $event) method of EventListener).
For me, this error was fired when I put my code within the __construct block within the EventListener:
public function __construct(Event $event)
{
$localProperty = $event->property
Mail::queue(etc...);
}
public function handle()
{
// Yeah I left this blank... whoops
}
Instead, the handle() method of the EventListener takes an Event interface and when called processes the job in the queue:
In the Event:
public function __construct(Object $ticket, AnotherObject $user)
{
$this->ticket = $ticket;
$this->user = $user;
}
And in Event Listener
class SomeEventListener implements ShouldQueue
{
use InteractsWithQueue;
use SerializesModels;
public function __construct()
{
// Leave me blank!
}
public function handle(Event $event)
{
$someArray = [
'ticket' = $event->ticket,
'user' = $event->user,
];
Mail::queue('some.view', $someArray, function($email) use ($someArray) {
// Do your code here
});
}
}
Although a tad late, I hope this helps someone. Queues are similar to Events (with the exception of Jobs being the main driving force behind Queues), so most of this should be relevant.
Turned out that it was because a model was added to the queue, that has since been deleted.