Lumen 5.1 SQS queue configuration - laravel-5

I've created a new project using Lumen, to take care of sending emails and some other small tasks. However, for some of the emails that need to be sent I want to use Amazon's queue.
In the documentation of Lumen, they only thing that is being mentioned is:
The QUEUE_DRIVER option in your .env file determines the queue "driver" that will be used by your application.
I have set this to SQS as suggested, however I'm a bit puzzled about the actual amazon credentials. How am I supposed to define those? The only "relevant" thing I've seen is from Amazon's documentation here and here. I have also installed the "aws/aws-sdk-php": "~3.0" package.
However these resources refer to using SQS's API directly. I want to be able to invoke the queue as it is being described in Lumen, so something like Queue::push() or $this->dispatch().
Here's the relevant code that I have.
This is in my .env file
QUEUE_DRIVER=sqs
AWS_KEY=key
AWS_SECRET=secretstuff
AWS_QUEUE=https://sqs.eu-west-1.amazonaws.com/
AWS_REGION=eu-west-1
This is my controller
<?php namespace App\Http\Controllers;
use App\Jobs\GuestEmailJob;
use Laravel\Lumen\Routing\Controller as BaseController;
use Log, Queue;
use Aws\Sqs\SqsClient;
class NotificationEmailController extends BaseController
{
public function pushToQueue($id)
{
Queue::push(new GuestEmailJob($id));
//$this->dispatch(new GuestEmailJob($id));
}
}
And finally this is my job
<?php namespace App\Jobs;
use Illuminate\Bus\Queueable;
use Illuminate\Queue\SerializesModels;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Contracts\Bus\SelfHandling;
use Illuminate\Contracts\Queue\ShouldQueue;
use Log;
abstract class GuestEmailJob implements SelfHandling, ShouldQueue
{
use InteractsWithQueue, Queueable, SerializesModels;
public function handle($id)
{
Log::info('within handle');
$user = User::find($id);
}
}
I've tried it both with Queue::push and $this->dispatch(). When a request hits the controller, everything goes fine until pushing to the queue and then nothing happens (no exceptions either). Anybody has any idea what am I doing wrong/missing?

After looking a bit into the source code, apparently you should have the queue.php config file under the config directory. So I just pasted the default config file from laravel 5 and customized it and now it works fine. Pretty silly issue after all, but wasn't clear from the documentation itself. Here's the default queue configuration just in case somebody also runs into this.
<?php
return [
'default' => env('QUEUE_DRIVER', 'sync'),
'connections' => [
'sync' => [
'driver' => 'sync',
],
'database' => [
'driver' => 'database',
'table' => 'jobs',
'queue' => 'default',
'expire' => 60,
],
'beanstalkd' => [
'driver' => 'beanstalkd',
'host' => 'localhost',
'queue' => 'default',
'ttr' => 60,
],
'sqs' => [
'driver' => 'sqs',
'key' => 'your-public-key',
'secret' => 'your-secret-key',
'queue' => 'your-queue-url',
'region' => 'us-east-1',
],
'iron' => [
'driver' => 'iron',
'host' => 'mq-aws-us-east-1.iron.io',
'token' => 'your-token',
'project' => 'your-project-id',
'queue' => 'your-queue-name',
'encrypt' => true,
],
'redis' => [
'driver' => 'redis',
'connection' => 'default',
'queue' => 'default',
'expire' => 60,
],
],
'failed' => [
'database' => 'mysql', 'table' => 'failed_jobs',
],
];

Related

Cannot track lumen job dispatch twice

I'm using Laravel lumen 8.x for creating REST API's.
In my project i have case where i'm calling third party API's for those API's i have created jobs for calling API's. in my Log i am getting Job Execution once but on that Third party API they are saying I'm calling one API multiple times.
I have tested this with a Single php File without any Framework. on that API call they said its fine because it was called one time.
I think something wrong with my Lumen project or in queue config. anyone can help ?
Running Command:
php artisan queue:work --timeout=600 --sleep=5 --tries=1 --memory=1024
Here is my queue.php config
return [
'default' => env('QUEUE_CONNECTION', 'redis'),
'connections' => [
'sync' => [
'driver' => 'sync',
],
'database' => [
'driver' => 'database',
'table' => 'jobs',
'queue' => 'default',
'retry_after' => null,
],
'redis' => [
'driver' => 'redis',
'connection' => 'default',
// 'queue' => env('REDIS_QUEUE', 'default'),
'queue' =>'default',
'retry_after' => null,
'block_for' => null,
],
],
'failed' => [
'database' => env('DB_CONNECTION', 'mysql'),
'table' => 'failed_jobs',
],
];
The only explanation I could think of is that your job takes longer then expected, therefore it times out, and it gets retried again.
You might wanna checkout the timeout configuration for jobs in laravels docs
https://laravel.com/docs/8.x/queues#timeout
Also you can specify how many times the job might be attempted before failing, by default its 3 I guess
https://laravel.com/docs/8.x/queues#max-job-attempts-and-timeout

How can i set custom retry_after for long running jobs | laravel

Question: How to customize long-running job without attempting multiple time after each retry_after seconds?
I have one job which will take 1 to 3 hours to run,
I already created job-based on laravel documentation, here is my job file.
<?php
namespace App\Modules\Csv\Jobs;
use App\Jobs\Job;
use Illuminate\Bus\Queueable;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Bus\Dispatchable;
use Illuminate\Queue\InteractsWithQueue;
use Illuminate\Queue\SerializesModels;
use Illuminate\Support\Facades\DB;
use Illuminate\Support\Facades\Storage;
use Illuminate\Support\Str;
use League\Csv\Reader;
use Phone;
/**
* A single excel import job, which can be pushed on to a queue
*/
class UploadCsvDataInTable extends Job implements ShouldQueue
{
use InteractsWithQueue, SerializesModels, Dispatchable,Queueable;
public $timeout = 172800;
/**
* The excel to import
*
* #var App\BulkUpload
*/
protected $csvUpload;
/**
* Create a new job instance.
*
* #param App\FeedImport
*
* #return void
*/
public function __construct(CsvUpload $csvUpload)
{
$this->csvUpload = $csvUpload;
}
public function handle()
{
app(CsvUploadService::class)->uploadCsv($this->csvUpload);
}
}
here is Laravel document to specify a timeout for jobs.
here is code for how I am calling that job.
UploadCsvDataInTable::dispatch($csvUpload)->onConnection('redis')->onQueue('low');
my command for queue:work In supervisor.
php artisan queue:work --queue=high,low,default --sleep=3 --tries=3
here is my configuration for queue & horizon
// horizon.php
'production' => [
'supervisor-1' => [
'connection' => 'redis',
'queue' => ['high', 'default', 'low'],
'balance' => 'simple',
'processes' => 6,
'tries' => 3,
],
],
//queue.php
'redis' => [
'driver' => 'redis',
'connection' => 'default',
'queue' => env('REDIS_QUEUE', 'default'),
'retry_after' => 90,
'block_for' => null,
],
I can see because of retry_after my job is attempting multiple time and after reaching 3 try as per horizon configuration its throwing MaxAttemptsExceededException.
if I increase $timeout to 24 hours I am getting duplicate records in my db as retry_after is attempting that job multiple time.
is there any way I can set custom retry_after for this job?
I have created another connection for long-running jobs and its working properly for me.
created new supervisor connection in horizon.php for long running process
'supervisor-long-running' => [
'connection' => 'redis-long-processes',
'queue' => 'long-running',
'balance' => 'simple',
'processes' => 3,
'tries' => 1,
'timeout' => 86000 // should be shorter than retry_after out
]
and new redis connection in queue.php
'redis-long-processes' => [
'driver' => 'redis',
'connection' => 'default',
'queue' => 'long-running',
'retry_after' => 86400,
'block_for' => null,
],
in database.php added new queue for long running jobs.
'queue' => [
[
'connection' => 'redis',
'queue' => ['high', 'default', 'low','long-running'],
'balance' => 'simple',
'processes' => 6,
'tries' => 3,
'url' => env('REDIS_URL'),
'host' => env('REDIS_HOST', '127.0.0.1'),
'password' => env('REDIS_PASSWORD', null),
'port' => env('REDIS_PORT', '6379'),
'database' => env('REDIS_CACHE_DB', '1'),
],
],
also don't forgot to call jobs using onConnection and onQueue to specify from which queue jobs should execute.
UploadDataInTable::dispatch($upload)->onConnection('redis-long-processes')->onQueue('long-running');
you don't need to set retry_after, you need to set tries to 1
public $tries = 1;
https://laravel.com/docs/8.x/queues#max-attempts

Configure Redis Sentinel in Laravel 5.5

How to configure the Redis sentinel? Redis can be configured in stand alone easily using laravel configs but when using sentinel how to configure is not documented anywhere?
There is one similar question asked on redit: but no help.
In Laravel 5.5 one can do it like this:
Reference: https://github.com/laravel/framework/pull/18850#issue-116339448
database.php:
'redis' => [
'client' => 'predis',
// Keep Default as is you want to use both redis and sentinel for different service(cache, queue)'
'default' => [
'host' => env('REDIS_HOST', '127.0.0.1'),
'password' => env('REDIS_PASSWORD', null),
'port' => env('REDIS_PORT', 6379),
'database' => 0,
],
// Create a custom connection to use redis sentinel
'cache_sentinel' => [
// Set the Sentinel Host from Environment (optinal you can hardcode if want to use in prod only)
env('CACHE_REDIS_SENTINEL_1'),
env('CACHE_REDIS_SENTINEL_2'),
env('CACHE_REDIS_SENTINEL_3'),
'options' => [
'replication' => 'sentinel',
'service' => 'cachemaster'),
'parameters' => [
'password' => env('REDIS_PASSWORD', null),
'database' => 0,
],
],
],
],
```
Specify the redis connection in your service where you want to use. example if cache needs redis sentinal can create new cache connection to use the above sentinal connection like this:
'stores' = [
//Keep default too as is
'redis' => [
'driver' => 'redis',
'connection' => 'default',
],
// create own cache connection
'sentinel_redis' => [
'driver' => 'redis',
'connection' => 'cache_sentinel',
],
And in Laravel app you can easily use via Cache Facade:
Cache::store('sentinel_redis')->get('key');

custom auth guards in laravel 5.5

I have multiple entities in my application that need to be authenticated. So I defined multiple guards to do so:
'guards' => [
'web' => [
'driver' => 'session',
'provider' => 'customers',
],
'customer' => [
'driver' => 'passport',
'provider' => 'customers',
],
'partner' => [
'driver' => 'passport',
'provider' => 'partners',
],
'admin' => [
'driver' => 'passport',
'provider' => 'admins',
],
],
Then in my Login controllers I have the following condition:
if (Auth::guard('admin')->attempt(['email' => $email, 'password' => $password])) {
...
...
}
when I try to do so (even though it's written in the docs) I get an error telling me that "method attempt does not exist".
So I try defining the guard method on the controller (also like the docs) like this:
protected function guard()
{
return Auth::guard('admin');
}
authentication still doesn't work with the right credentials (I tried to check what guard the Auth facade was using before the attempt using dd(Auth::guard()) and it was the default web guard.
I tried removing the web guard from the default guards array in auth config. didn't work.
I also tried config:clear and rerunning the server but nothing worked!
I am out of ideas. I know that I can write my own simple login, but why reinvent the wheel, and I was following the docs to the letter. help.

Laravel pusher Illuminate \ Broadcasting \ BroadcastException No message

I'am using Laravel 5.5 with pusher to make a real time notification , the notification made from the Api
after i made the configuration
in the Api
public function store(Request $request)
{
$advertising = Advertising::create($request->all());
$admins = \App\Admin::all();
\Notification::send( $admins, new \App\Notifications\AdvertisingAdded($advertising) );
return $advertising;
}
in AdvertisingAdded
namespace App\Notifications;
use Illuminate\Bus\Queueable;
use Illuminate\Notifications\Notification;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Notifications\Messages\MailMessage;
use Illuminate\Notifications\Messages\BroadcastMessage;
use App\Advertising;
class AdvertisingAdded extends Notification
{
use Queueable;
//-must be public fir pusher
public $advertising;
public function __construct(Advertising $advertising)
{
$this->advertising = $advertising;
}
public function via($notifiable)
{
return ['database','broadcast'];
}
public function toArray($notifiable)
{
return [
'msg' => 'Advertising '.$this->advertising->title_ar.' is added ',
'advertising_id' => $this->advertising->id
];
}
public function toBroadcast($notifiable)
{
return new BroadcastMessage([
'msg' => 'Advertising '.$this->advertising->title_ar.' is added ',
'advertising_id' => $this->advertising->id
]);
}
}
when i post from postman i get an error
Illuminate \ Broadcasting \ BroadcastException No message
error image
i followed this video https://www.youtube.com/watch?v=i6Rdkv-DLwk
i solve my problem by : making the encrypted: false
Add curl options to broadcasting.php
`'pusher' => [
'driver' => 'pusher',
'key' => env('PUSHER_APP_KEY'),
'secret' => env('PUSHER_APP_SECRET'),
'app_id' => env('PUSHER_APP_ID'),
'options' => [
'cluster' => env('PUSHER_APP_CLUSTER'),
'encrypted' => true,
'curl_options' => [
CURLOPT_SSL_VERIFYHOST => 0,
CURLOPT_SSL_VERIFYPEER => 0,
]
],`
I solved this problem. In config/broadcasting.php use this code
'options' => [
'cluster' => 'eu',
'useTLS' => false
],
make useTLS false
In laravel 7, configured like below to config/broadcasting.php and and run artisan command cache:clear. solved for me.
'pusher' => [
'driver' => 'pusher',
'key' => env('PUSHER_APP_KEY'),
'secret' => env('PUSHER_APP_SECRET'),
'app_id' => env('PUSHER_APP_ID'),
'options' => [
'cluster' => 'mt1',
'useTLS' => false,
],
],
I solved my problem by setting my .env file
Set:
APP_URL=http://localhost
DB_HOST=localhost
And run
php artisan config:cache

Resources