Using laravel 5.5, we need to use both Redis and SQS queues. Redis for our internal messaging and SQS for messages coming from a 3rd party.
config/queue.php has various connection information. The first key is the default connection. That default is the one used by queue:work artisan command.
'default' => 'redis',
'connections' => [
'sqs' => [
'driver' => 'sqs',
'key' => env('ACCESS_KEY_ID', ''),
'secret' => env('SECRET_ACCESS_KEY', ''),
'prefix' => 'https://sqs.us-west-1.amazonaws.com/account-id/',
'queue' => 'my-sqs-que'),
'region' => 'us-west-1',
],
'redis' => [
'driver' => 'redis',
'connection' => 'default',
'queue' => env('REDIS_QUE' , 'default'),
'retry_after' => 90,
],
The question is how can we use different queue connection for queue:work.
If --queue=my-sqs-que is supplied, with default connection set to redis, laravel looks under redis and obviously does not find my-sqs-que
Setting default to sqs will disable processing our internal messages.
You can specify the connection when running queue:work, see Specifying the Connection and Queue:
You may also specify which queue connection the worker should utilize. The connection name passed to the work command should correspond to one of the connections defined in your config/queue.php configuration file:
php artisan queue:work redis
You will need to setup the corresponding connections per queue, as well.
However, any given queue connection may have multiple "queues" which may be thought of as different stacks or piles of queued jobs.
Related
I have problem with laravel queues. In my project, default connection is sync, I want to add sqs connection for one type of jobs.
When I dispatch job in this way:
TestAction::dispatch()->onQueue('test');
Job is performed immediately (by sync connection).
If I dispatch job in this way:
TestAction::dispatch()->onQueue('test')->onConnection('sqsTestAction');
everything is ok.
Queue "test" is in sqsTestAction connection, why in first example job is being sent by sync connection?
My config/queue.php:
'default' => env('QUEUE_CONNECTION', 'sync'),
'connections' => [
'sync' => [
'driver' => 'sync',
],
'sqsTestAction' => [
'driver' => 'sqs',
'key' => env('AWS_ACCESS_KEY_ID'),
'secret' => env('AWS_SECRET_ACCESS_KEY'),
'prefix' => env('AWS_SQS_PREFIX', 'https://sqs.us-east-1.amazonaws.com/your-account-id'),
'queue' => "test",
'region' => env('AWS_DEFAULT_REGION', 'us-east-1'),
],
],
Laravel ver 5.8
why in first example job is being sent by sync connection?
Because it is set by default as you can see. Change .env file value for QUEUE_CONNECTION to sqsTestAction and that will become default.
In default key of config/queue file, second parameter is fallback parameter for case when .env value doesn't exist.
I have a library, which imports lots of images and I am trying to use Laravel background jobs. For queued jobs, I am following Laravel documentation.
First (create table):
php artisan queue:table
php artisan migrate
Then configuration in .env file for Redis:
REDIS_HOST=127.0.0.1
REDIS_PASSWORD=null
REDIS_PORT=43216
Create a job:
php artisan make:job CarsJob
CarsJob:
public function handle()
{
$cars = new CarsLibrary();
$CarsLibrary->importAll();
}
Dispatching a job in a some action in the controller:
First what I have tried:
$importCarsJob = (new ImportCarsJob())->onQueue('import_cars');
$this->dispatch($importCarsJob );
Second what I have tried:
$importCarsJob = new importCarsJob();
$this->dispatch($importCarsJob);
I have enabled Redis in my hosting. It is shared hosting.
If I access the URL, I see that this job is not in the background, because it needs more than a minute to finish the request.
EDIT:
The env file is above, this is config/queue.php:
'connections' => [
'sync' => [
'driver' => 'redis',
],
... other drivers like beanstalkd
'database' => [
'driver' => 'database',
'table' => 'jobs',
'queue' => 'default',
'retry_after' => 90,
],
'redis' => [
'driver' => 'redis',
'connection' => 'default',
'queue' => env('REDIS_QUEUE', 'default'),
'retry_after' => 90,
'block_for' => null,
],
],
I have no REDIS_QUEUE in env file.
It seems that you have not updated your queue connection in config/queue.php from sync to redis (or the environment variable QUEUE_CONNECTION). The sync driver will execute jobs immediately without pushing them on a queue.
By the way, you don't need the queue database table if you use the redis queue driver.
I have studied about the laravel queues from here: https://laravel.com/docs/5.6/queues.
I have a situtation that my project had no queues in particular, and only 'default' queue was running. Now I have two queues: JobA and JobB. And I want to set priority of JobB higher than JobA.
To attach the jobs with queues I have used:
->onQueue('jobA');
->onQueue('jobB');
In .env file I have added :
QUEUE_DRIVER=sync
And in queue.php this is the code:
return [
'default' => env('QUEUE_DRIVER', 'sync'),
'connections' => [
'sync' => [
'driver' => 'sync',
],
'database' => [
'driver' => 'database',
'table' => 'jobs',
'queue' => 'default',
'retry_after' => 90,
]
],
'failed' => [
'database' => env('DB_CONNECTION', 'mysql'),
'table' => 'failed_jobs',
],
];
And after making these changes, I run these commands on the server:
php artisan queue:work --queue=jobB,jobA
php artisan queue:restart
The first command runs all pending jobs from specified table (jobs table).
But now the new jobs which are getting created, how will I get to know that those are in my mentioned priority order? And also I checked in the database table (jobs table) still name of queue is appearing as default.
Please let me know what I am doing wrong.
Thanks
I'm having a problem connecting to ElastiCache Redis from Laravel application installed on EC2 instance or even using redis-cli from EC2 instance.
Laravel
I tried to use predis with configurations in database.php like
'redis' => [
'client' => 'predis',
'default' => [
'host' => env('REDIS_HOST', '127.0.0.1'),
'password' => env('REDIS_PASSWORD', null),
'port' => env('REDIS_PORT', 6379),
'database' => 0,
'read_write_timeout' => -1,
'timeout' => 0
],
],
and got 'Error while reading line from the server. [tcp:server here]'
I tried with phpRedis extension with same configurations only change 'client' => 'phpredis' and got error read error on connection {"exception":"[object] (RedisException(code: 0): read error on connection at vendor/laravel/framework/src/Illuminate/Redis/Connectors/PhpRedisConnector.php:69)
Redis cli
Using redis cli redis-cli -h host_here -p 6379 -a password_here I see prompt like host:6379> but typing any command throws error Error: Connection reset by peer
ElastiCache Redis configurations
My EC2 and elastic cache are in the same VPC and using telnet I can connect to redis instance
~$ telnet host 6379
Trying 172.31.23.113...
Connected to host.
Escape character is '^]'.
Thanks for any help!
I know this is pretty old but I was having the same issue myself. If anyone encounters this issue then see the solution here and here.
It seems that when you enable Encryption in-transit in AWS Elasticache it prevents you from using redis-cli as it doesn't support TLS connections. Switching to another client should work. This answer has a list of TLS enabled clients.
Edit:
Did some more digging and found that using stunnel you can wrap your connection of redis-cli with ssl. Here is a guide for doing it.
Related: Laravel + Redis Cache via SSL?
To which I've answered here: https://stackoverflow.com/a/48876398/663058
Relevant details below:
Since you have clustering and TLS then you'll need a different config entirely:
'redis' => [
'client' => 'predis',
'cluster' => env('REDIS_CLUSTER', false),
// Note! for single redis nodes, the default is defined here.
// keeping it here for clusters will actually prevent the cluster config
// from being used, it'll assume single node only.
//'default' => [
// ...
//],
// #pro-tip, you can use the Cluster config even for single instances!
'clusters' => [
'default' => [
[
'scheme' => env('REDIS_SCHEME', 'tcp'),
'host' => env('REDIS_HOST', 'localhost'),
'password' => env('REDIS_PASSWORD', null),
'port' => env('REDIS_PORT', 6379),
'database' => env('REDIS_DATABASE', 0),
],
],
'options' => [ // Clustering specific options
'cluster' => 'redis', // This tells Redis Client lib to follow redirects (from cluster)
]
],
'options' => [
'parameters' => [ // Parameters provide defaults for the Connection Factory
'password' => env('REDIS_PASSWORD', null), // Redirects need PW for the other nodes
'scheme' => env('REDIS_SCHEME', 'tcp'), // Redirects also must match scheme
],
'ssl' => ['verify_peer' => false], // Since we dont have TLS cert to verify
]
]
Explaining the above:
'client' => 'predis': This specifies the PHP Library Redis driver to use (predis).
'cluster' => 'redis': This tells Predis to assume server-side clustering. Which just means "follow redirects" (e.g. -MOVED responses). When running with a cluster, a node will respond with a -MOVED to the node that you must ask for a specific key.
If you don't have this enabled with Redis Clusters, Laravel will throw a -MOVED exception 1/n times, n being the number of nodes in Redis cluster (it'll get lucky and ask the right node every once in awhile)
'clusters' => [...]: Specifies a list of nodes, but setting just a 'default' and pointing it to the AWS 'Configuration endpoint' will let it find any/all other nodes dynamically (recommended for Elasticache, because you don't know when nodes are comin' or goin').
'options': For Laravel, can be specified at the top-level, cluster-level, and node option. (they get combined in Illuminate before being passed off to Predis)
'parameters': These 'override' the default connection settings/assumptions that Predis uses for new connections. Since we set them explicitly for the 'default' connection, these aren't used. But for a cluster setup, they are critical. A 'master' node may send back a redirect (-MOVED) and unless the parameters are set for password and scheme it'll assume defaults, and that new connection to the new node will fail.
If you are using predis as client.
Then you can change Redis connection in config/database.php
'redis' => [
'client' => 'predis',
'default' => [
'scheme' => 'tls',
'host' => env('REDIS_HOST', '127.0.0.1'),
'password' => env('REDIS_PASSWORD', null),
'port' => env('REDIS_PORT', 6379),
'database' => 0,
],
],
You can add 'scheme' in that.
I am using Laravel 5.8 And then everything works great within Laravel.
But yes, as Redis doesn't provide TLS connection so redis-cli will still not work.
I am building a site in Laravel 5, I need to manage session as something like this:
For Example. I have a site x#x.com hosting on server X and another y#y.com hosting on server Y along with database,(both server are of different countries) I need to use same database for the both site but session management is typical task for me in x#x.com as database is hosting on y#y.com. I am using Auth in laravel for authentication How that will be possible please help-.
You should use the same database connection on both servers. You can either make a small third server just for session management or you can simply tell X server to connect to the Y server database. You will first start by setting the environment variable SESSION_DRIVER or the configuration property session.driver to: database or redis depending on what you are using. Then create a connection the config file database.php under connections property if it's a RDBMS or under redis if it's a redis database.
'connections' => [
// ...
'session' => [
'driver' => 'mysql',
'host' => env('SESSION_DB_HOST'),
'database' => env('SESSION_DB_NAME'),
'username' => env('SESSION_DB_USERNAME'),
'password' => env('SESSION_DB_PASSWORD'),
'charset' => 'utf8',
'collation' => 'utf8_unicode_ci',
'prefix' => '',
'strict' => false,
]
]
// Or
'redis' => [
// ...
'session' => [
'host' => env('SESSION_REDIS_HOST'),
'port' => env('SESSION_REDIS_PORT', 6379),
'database' => 0,
],
]
Then in the file session.php, change the the value of connection to the name of the connection you want, session in this case.
Be mindful that you need to open the required ports and do some authorization process and take security measures on the host server (Y server since it has the database).
---- Answering the comment
Session driver is the mechanism that laravel uses to manage sessions. It can be on file, database, redis... So, when you choose the session driver in its config file and the driver is a database, you would specify a database connection for it or it will use the default database connection. So, if you create a third server and you want to use Redis as the database, you would open port 6379 (in case you're using the default redis configuration), then you create a connection in your Laravel installations. The connection in database.php will be like so:
'redis' => [
// ...
'my_session' => [
'host' => env('SESSION_REDIS_HOST'),
'port' => env('SESSION_REDIS_PORT', 6379),
'database' => 0,
],
]
Then in your environment or in the .env you add:
SESSION_REDIS_HOST=xx.xx.xx.xx // the ip address or domain of the third server
SESSION_REDIS_PORT= 6379
SESSION_DRIVER=redis
And finally, in you session.php config, you would set these values:
//...
'connection' => 'my_session',
You would do this on all Laravel installations that you want to be connected to the same session database.
Laravel provides database sessions which stores sessions in database so that you can use multiple servers for your application. Have a look at https://laravel.com/docs/5.3/session#database-sessions