I'm using Laravel lumen 8.x for creating REST API's.
In my project i have case where i'm calling third party API's for those API's i have created jobs for calling API's. in my Log i am getting Job Execution once but on that Third party API they are saying I'm calling one API multiple times.
I have tested this with a Single php File without any Framework. on that API call they said its fine because it was called one time.
I think something wrong with my Lumen project or in queue config. anyone can help ?
Running Command:
php artisan queue:work --timeout=600 --sleep=5 --tries=1 --memory=1024
Here is my queue.php config
return [
'default' => env('QUEUE_CONNECTION', 'redis'),
'connections' => [
'sync' => [
'driver' => 'sync',
],
'database' => [
'driver' => 'database',
'table' => 'jobs',
'queue' => 'default',
'retry_after' => null,
],
'redis' => [
'driver' => 'redis',
'connection' => 'default',
// 'queue' => env('REDIS_QUEUE', 'default'),
'queue' =>'default',
'retry_after' => null,
'block_for' => null,
],
],
'failed' => [
'database' => env('DB_CONNECTION', 'mysql'),
'table' => 'failed_jobs',
],
];
The only explanation I could think of is that your job takes longer then expected, therefore it times out, and it gets retried again.
You might wanna checkout the timeout configuration for jobs in laravels docs
https://laravel.com/docs/8.x/queues#timeout
Also you can specify how many times the job might be attempted before failing, by default its 3 I guess
https://laravel.com/docs/8.x/queues#max-job-attempts-and-timeout
How to configure the Redis sentinel? Redis can be configured in stand alone easily using laravel configs but when using sentinel how to configure is not documented anywhere?
There is one similar question asked on redit: but no help.
In Laravel 5.5 one can do it like this:
Reference: https://github.com/laravel/framework/pull/18850#issue-116339448
database.php:
'redis' => [
'client' => 'predis',
// Keep Default as is you want to use both redis and sentinel for different service(cache, queue)'
'default' => [
'host' => env('REDIS_HOST', '127.0.0.1'),
'password' => env('REDIS_PASSWORD', null),
'port' => env('REDIS_PORT', 6379),
'database' => 0,
],
// Create a custom connection to use redis sentinel
'cache_sentinel' => [
// Set the Sentinel Host from Environment (optinal you can hardcode if want to use in prod only)
env('CACHE_REDIS_SENTINEL_1'),
env('CACHE_REDIS_SENTINEL_2'),
env('CACHE_REDIS_SENTINEL_3'),
'options' => [
'replication' => 'sentinel',
'service' => 'cachemaster'),
'parameters' => [
'password' => env('REDIS_PASSWORD', null),
'database' => 0,
],
],
],
],
```
Specify the redis connection in your service where you want to use. example if cache needs redis sentinal can create new cache connection to use the above sentinal connection like this:
'stores' = [
//Keep default too as is
'redis' => [
'driver' => 'redis',
'connection' => 'default',
],
// create own cache connection
'sentinel_redis' => [
'driver' => 'redis',
'connection' => 'cache_sentinel',
],
And in Laravel app you can easily use via Cache Facade:
Cache::store('sentinel_redis')->get('key');
I am learning how to use Redis Queue. I am using Window 8.1 XAMPP. I have successfully implemented Database queue. So far, as per this article: https://laravel.com/docs/5.4/redis I have installed package in composer require predis/predis
I took subscription which is limited to 30MB for testing from here: https://app.redislabs.com
In config/queue.php, I saw the below code.
'redis' => [
'driver' => 'redis',
'connection' => 'default',
'queue' => 'default',
'retry_after' => 90,
],
Below is the code in config/database.php but my default database is mysql
'redis' => [
'client' => 'predis',
'default' => [
'host' => env('REDIS_HOST', '127.0.0.1'),
'password' => env('REDIS_PASSWORD', null),
'port' => env('REDIS_PORT', 6379),
'database' => 0,
],
],
My Question is: where should i get the info for redis for above two sections?
I already checked by selecting the Database->Configuration Tab
To obtain the connection information for your Redis Cloud instance:
Login to the dashboard
Select the database
Click the 'Configuration' tab
The host, port & password are obtainable from there.
I have a laravel (5.3) app with redis used for sessions (using predis). Everything works as long as I use a single redis node (using default approach from config/database.php). As soon as I switch to a Redis cluster though I am starting to get MOVED error like 50% of the time (based on googling I understand that this should be managed by predis, but somehow isn't).
I tried changing the cluster parameter to true, but then I get a weird error
No connection could be made because the target machine actively refused it. [tcp://127.0.0.1:6379]
Although the redis cluster that I use is deployed in Azure (and is configured via .env file) and the parameters are accepted without any problem when a single node is used.
Configuration
Here is the laravel configuration that I have (as mentioned earlier, it's the standard default)
'redis' => [
'client' => 'predis',
'cluster' => false,
'default' => [
'host' => env('REDIS_HOST', 'localhost'),
'password' => env('REDIS_PASSWORD', null),
'port' => env('REDIS_PORT', 6379),
'database' => 0,
],
],
For Redis, I use Azure Redis Cache Cluster Premium P1, 2 shards (as described here).
UPDATE 2
So far I also tried the following variations of the config:
Setting cluster to true
Setting cluster to redis
Adding default -> cluster set to redis
Adding default -> options set to array('cluster', 'redis')
All the time I am getting MOVED error...
My Redis version is 3.2, predis/predis package 1.1.1
Working config for predis 1.1+
'redis' => [
'cluster' => true,
'default' => [
'host' => env('REDIS_HOST', 'localhost'),
'password' => env('REDIS_PASSWORD', null),
'port' => env('REDIS_PORT', 6379),
'database' => 0,
] ,
'options' => [
'cluster' => 'redis',
'parameters' => ['password' => env('REDIS_PASSWORD', null)],
],
],
Big thank you for all the help :)
Note
The information and config specified here is for Laravel 5.3 and below. The redis config structure was changed in Laravel 5.4+.
TL;DR
'cluster' => true should be true to create one aggregate client that handles multiple nodes.
'options' => ['cluster' => 'redis'] needs to be added to the configuration as a sibling of default (not a child) in order to tell Predis to handle the server-side clustering provided by Azure.
if using auth with server-side clustering, 'options' => [ 'cluster' => 'redis', 'parameters' => ['password' => env('REDIS_PASSWORD', null)], ] will be needed to auth newly discovered cluster nodes.
Full Text
In the redis configuration, you can set up multiple connections to multiple redis instances. The cluster option tells Laravel how to handle those multiple defined connections.
If cluster is set to false, Laravel will create individual \Predis\Client instances for each connection. Each connection can be accessed individually, and will not have any relation to another connection.
If cluster is set to true, Laravel will create an aggregate \Predis\Client instance using all the defined connections. With no other configuration, this is kind of a "fake" cluster. It uses client-side sharding to distribute the keyspace and may require external monitoring and maintenance to ensure a proper key load balance.
The issue you're running into, however, is that Azure implements (presumably) a real server-side Redis cluster, which handles automatic sharding of the keyspace. In this instance, the nodes know about each other and talk to each other, and may go up and down. This is where MOVED and ASK responses come from.
The Predis library can automatically handle these responses, but only when you tell it that it needs to. In this case, you need to tell the Predis client that it needs to handle clustering, and this is done by Laravel through the options array on the redis configuration.
On the redis configuration, the options key should be a sibling of your connections (i.e. default), not a child. Additionally, the options should be specified as key => value pairs.
So, your configuration should look like:
'redis' => [
'cluster' => true,
'default' => [
'host' => env('REDIS_HOST', 'localhost'),
'password' => env('REDIS_PASSWORD', null),
'port' => env('REDIS_PORT', 6379),
'database' => 0,
],
'options' => [
'cluster' => 'redis',
],
],
The cluster key under the redis config will tell Laravel to create an aggregate Predis\Client instance that may handle multiple nodes, and the cluster key under the options array will tell that instance that it needs to handle server-side clustering, not client-side clustering.
Auth
The original connection parameters (including authentication) are not shared with connections to new nodes discovered via -MOVED and -ASK responses. So, any errors you previously got from -MOVED responses will now just convert to NOAUTH errors. However, the server-side 'cluster' configuration allows for a 'parameters' sibling which defines a list of parameters to use with newly discovered nodes. This is where you can put your auth parameters to use with new nodes.
I believe this will look something like:
'redis' => [
'cluster' => true,
'default' => [
'host' => env('REDIS_HOST', 'localhost'),
'password' => env('REDIS_PASSWORD', null),
'port' => env('REDIS_PORT', 6379),
'database' => 0,
],
'options' => [
'cluster' => 'redis',
'parameters' => ['password' => env('REDIS_PASSWORD', null)],
],
],
Fair warning, this is all information I just got from research and code diving. While I have used Redis with Laravel, I have not used server side clustering (yet), so this still may not work.
Some useful pieces of information I came across while looking into this:
Predis issue discussing connecting to a redis-cluster:
https://github.com/nrk/predis/issues/259#issuecomment-117339028
It looks like you did not configure Predis to use redis-cluster but instead you are using it with the plain old client-side sharding logic (which is also the default behaviour). You should configure the client setting the option cluster with the value redis to let the client know it must play along with redis-cluster. Quick example:
$client = new Predis\Client([$node1, $node2, ...], ['cluster' => 'redis']);
Doing so will make it possible for the client to automatically handle -MOVED or -ASK responses coming from Redis nodes.
MS article discussing clustering on redis cache:
https://learn.microsoft.com/en-us/azure/redis-cache/cache-how-to-premium-clustering#how-do-i-connect-to-my-cache-when-clustering-is-enabled
You can connect to your cache using the same endpoints, ports, and keys that you use when connecting to a cache that does not have clustering enabled. Redis manages the clustering on the backend so you don't have to manage it from your client.
Laravel code for creating Predis\Client instances:
https://github.com/laravel/framework/blob/v5.3.28/src/Illuminate/Redis/Database.php#L25-L66
Related: Laravel + Redis Cache via SSL?
To which I've answered here: https://stackoverflow.com/a/48876398/663058
Relevant details below:
If you have clustering and TLS then you'll need a the following config (tested with AWS Elasticache):
'redis' => [
'client' => 'predis',
'cluster' => env('REDIS_CLUSTER', false),
// Note! for single redis nodes, the default is defined here.
// keeping it here for clusters will actually prevent the cluster config
// from being used, it'll assume single node only.
//'default' => [
// ...
//],
// #pro-tip, you can use the Cluster config even for single instances!
'clusters' => [
'default' => [
[
'scheme' => env('REDIS_SCHEME', 'tcp'),
'host' => env('REDIS_HOST', 'localhost'),
'password' => env('REDIS_PASSWORD', null),
'port' => env('REDIS_PORT', 6379),
'database' => env('REDIS_DATABASE', 0),
],
],
'options' => [ // Clustering specific options
'cluster' => 'redis', // This tells Redis Client lib to follow redirects (from cluster)
]
],
'options' => [
'parameters' => [ // Parameters provide defaults for the Connection Factory
'password' => env('REDIS_PASSWORD', null), // Redirects need PW for the other nodes
'scheme' => env('REDIS_SCHEME', 'tcp'), // Redirects also must match scheme
],
'ssl' => ['verify_peer' => false], // Since we dont have TLS cert to verify
]
]
Explaining the above:
'client' => 'predis': This specifies the PHP Library Redis driver to use (predis).
'cluster' => 'redis': This tells Predis to assume server-side clustering. Which just means "follow redirects" (e.g. -MOVED responses). When running with a cluster, a node will respond with a -MOVED to the node that you must ask for a specific key.
If you don't have this enabled with Redis Clusters, Laravel will throw a -MOVED exception 1/n times, n being the number of nodes in Redis cluster (it'll get lucky and ask the right node every once in awhile)
'clusters' => [...]: Specifies a list of nodes, but setting just a 'default' and pointing it to the AWS 'Configuration endpoint' will let it find any/all other nodes dynamically (recommended for Elasticache, because you don't know when nodes are comin' or goin').
'options': For Laravel, can be specified at the top-level, cluster-level, and node option. (they get combined in Illuminate before being passed off to Predis)
'parameters': These 'override' the default connection settings/assumptions that Predis uses for new connections. Since we set them explicitly for the 'default' connection, these aren't used. But for a cluster setup, they are critical. A 'master' node may send back a redirect (-MOVED) and unless the parameters are set for password and scheme it'll assume defaults, and that new connection to the new node will fail.
After applying a lot of suggestions at last I found the solution by using phpredis instead of Redis. Use the below code:
'redis' => [
'client' => 'phpredis',
'clusters' => [
'default' => [
[
'host' => env('REDIS_HOST', 'localhost'),
'password' => env('REDIS_PASSWORD', null),
'port' => env('REDIS_PORT', 6379),
'database' => 0,
],
],
]
],
use official doc for better understanding Laravel Redis with clusters
For AWS elasticcache redis cluster the above configuration did not work, however below are working for me. Also mentioned in the documentation: https://laravel.com/docs/5.4/redis#configuration
'redis' => [
'client' => 'predis',
'options' => [
'cluster' => 'redis',
],
'clusters' => [
'default' => [
[
'host' => env('REDIS_HOST', 'localhost'),
'password' => env('REDIS_PASSWORD', null),
'port' => env('REDIS_PORT', 6379),
'database' => 0,
],
],
],
],
Please refer to [ https://laravel.com/docs/5.5/redis ].
Make sure you have the proper library
composer require predis/predis
In config/database.php as well as config/queue.php [ if your queue is also using clustered reddis]
'redis' => [
'client' => 'predis',
'options' => [
'cluster' => 'redis',
],
'clusters' => [
'default' => [
[
'host' => env('REDIS_HOST', 'localhost'),
'password' => env('REDIS_PASSWORD', null),
'port' => env('REDIS_PORT', 6379),
'database' => 0,
],
],
],
]
this worked for me:
'redis' => [
'client' => 'predis',
'cluster' => true,
'options' => [
'cluster' => 'redis',
'parameters' => [
'host' => env('REDIS_DEFAULT_HOST', '127.0.01'),
'password' => env('REDIS_PASSWORD', null),
'port' => env('REDIS_DEFAULT_PORT', 6379),
'database' => 0,
],
],
'clusters' => [
'default' => [
'host' => env('REDIS_DEFAULT_HOST', '127.0.01'),
'password' => env('REDIS_PASSWORD', null),
'port' => env('REDIS_DEFAULT_PORT', 6379),
'database' => 0,
],
'jobs' => [
'host' => env('REDIS_JOBS_HOST', '127.0.01'),
'password' => env('REDIS_PASSWORD', null),
'port' => env('REDIS_JOBS_PORT', 6379),
'database' => 0,
],
'content' => [
'host' => env('REDIS_CONTENT_HOST', '127.0.01'),
'password' => env('REDIS_PASSWORD', null),
'port' => env('REDIS_CONTENT_PORT', 6379),
'database' => 0,
],
'options' => [
'cluster' => 'redis'
],
]
]
to see how I got there, see my answer here
This result comes up in Google for predis+redis-cluster (no laravel). I and my team were struggling for hours with this issue, so I am putting this answer here for those who find this page getting that error while trying to connect to the Redis cluster directly:
$redis = new Predis\Client(
['tcp://127.0.0.1:6379'],
['cluster' => 'redis']
);