So I have Laravel 5.2 project where Redis is used as cache driver.
There is a controller, which has method that connects Redis and increases a value and adds value to the set each time this method is called, just like
$redis = Redis::connection();
$redis->incr($value);
$redis->sadd("set", $value);
But the problem is that sometimes there are many connections and many calls for this method at the same time, and there is a data loss, because if two callers call this method while $value is 2, after incr it will become 3, but should be 4 (after two incrs basically).
I have thought about using Redis transactions, but I can't imagine when should I call multi command to start a queue and when to exec it.
Also I had an idea to collect all incrs and sadds as strings to another set and then transact them with cron job, but it would cost too much RAM.
So, any suggestions, how can this data loss be avoided?
Laravel uses Predis as a Redis driver.
To execute a transaction with Predis you have to invoke the transaction method of the driver and give it a callback:
$responses = $redis->transaction(function ($tx) {
$redis->incr($value);
$redis->sadd("set", $value);
});
Related
I placed this code inside a Route::get() method only to test it quicker. So this is how it looks:
use Illuminate\Support\Facades\Cache;
Route::get('/cache', function(){
$lock = Cache::lock('test', 4);
if($lock->get()){
Cache::put('name', 'SomeName'.now());
dump(Cache::get('name'));
sleep(5);
// dump('inside get');
}else{
dump('locked');
}
// $lock->release();
});
If you reach this route from two browsers (almost)at the same time. They both will respond with the result from dump(Cache::get('name'));. Shouldn't the second browser respond be "locked"? Because when it calls the $lock->get() that is supposed to return false? And that because when the second browser tries to reach this route the lock should be still set.
That same code works just fine if the time required for the code after the $lock = Cache::lock('test', 4) to be executed is less than 4. If you set the sleep($sec) when $sec<4 you will see that the first browser reaching this route will respond with the result from Cache::get('name') and the second browser will respond with "locked" as expected.
Can anyone explain why is this happening? Isn't it suppose that any get() method to that lock, expect the first one, to return false for that amount of time the lock has been set? I used 2 different browsers but it works the same with 2 tabs from the same browser too.
Quote from the 5.6 docs https://laravel.com/docs/5.6/cache#atomic-locks:
To utilize this feature, your application must be using the memcached or redis cache driver as your application's default cache driver. In addition, all servers must be communicating with the same central cache server.
Quote from the 5.8 docs https://laravel.com/docs/5.8/cache#atomic-locks:
To utilize this feature, your application must be using the memcached, dynamodb, or redis cache driver as your application's default cache driver. In addition, all servers must be communicating with the same central cache server.
Quote from the 8.0 docs https://laravel.com/docs/8.x/cache#atomic-locks:
To utilize this feature, your application must be using the memcached, redis, dynamodb, database, file, or array cache driver as your application's default cache driver. In addition, all servers must be communicating with the same central cache server.
Apparently, they have been adding support for more drivers to make use of this lock functionality. Check which Cache driver you are using and if it fits the support list of your Laravel version.
There is likely an atomicity issue here where the cache driver you are using is not able to lock a file atomically. What should happen is that when a process (i.e. a php request) is writing to the lock file, all other processes requiring the lock file should at least wait until the lock file available to be read again. If not, they read the lock file before it has been written to, which obviously causes a race condition.
I saw this question I asked, well now I can say that the problem I was trying to solve here was not because of the atomic lock. The problem here is the sleep method. If the time provided to the sleep method is bigger than the time that a lock will live, it means when the next request it's able to hit the route the lock time will expire(will be released). And that's because let's say you have defined a route like this:
Route::get('case/{value}', function($value){
if($value){
dump('hit-1');
}else{
sleep(5);
dump('hit-0');
}
});
And you open two browser tabs with the same URL that hits this route something like:
127.0.0.1:8000/case/0
and
127.0.0.1:8000/case/1
It will show you that the first route will take 5sec to finish execution and even if the second request is sent almost at the same time with the first request, still it will wait to finish the first one and then run. This means the second request will last 5sec(from the first request) plus the time it took to run.
Back to the asked question the lock time will expire by the time the second request will get it or said differently run the $lock->get() statement.
I am using https://github.com/JosephSilber/page-cache to cache pages. To prepare pages beforehand (about 100,000), I used to run 8 http requests in parallel via GuzzleHttp. It worked, but was pretty slow, because of the overhead.
I am looking for a way to process an instance of Illuminate\Http\Request directly via the app instance, preventing a real http request. I noticed, that this is much faster. However, parallelizing this with https://github.com/amphp/parallel-functions poses some problems.
The basic code is this:
wait(parallelMap($urlChunks->all(), function($urls) {
foreach($urls as $url) {
//handle the request
}
}, $pool));
I tried several variants for handling the request.
1.
$request = \Illuminate\Http\Request::create($url, 'GET');
$response = app()->handle($request);
In this case app() returns an instance of Illuminate\Container\Container, not an instance of app. So it does not have the method handle()and so on.
2.
$request = \Illuminate\Http\Request::create($url, 'GET');
$response = $app->handle($request);
Only difference here: The variable $app was injected into the closure. Its value is the correct return value from app() called outside the closure. It is the application, but amp fails, because the PDO connections contained in the Application instance can not be serialized.
3.
$request = \Illuminate\Http\Request::create($url, 'GET');
$app = require __DIR__.'/../../../bootstrap/app.php';
$app->handle($request);
This works for a short while. But with each instantiation of the app, one or two mysql connections start to linger around in status "Sleep". They only get closed, when the script ends. Important: This does not have to do with parallelization. I actually tried the same with a sequential loop and noticed the same effect. This looks like an error in the framework to me, because one should expect, that the Application instance closes all connections, when it is destroyed. Or can I do this manually? This would be one way to get this thing to work.
Any ideas?
The third version is my recommended way to do it. Resources in PHP are usually cleaned up when PHP exists, but that doesn't work for long running applications. To change that, I'd file an issue in the Laravel repository or whatever is creating that database connection.
End Goal
The aim is for my application to fire off potentially a lot of emails to the Redis queue (This bit is working) and then Redis throttle the processing of these to only a set number of emails every selected number of minutes.
For this example, I have a test job that appends the time to a file and I am attempting to throttle it to once every 60 seconds.
The story so far....
So far, I have the application successfully pushing a test amount of 50 jobs to the Redis queue. I can log in to Horizon and see these 50 jobs in the "processjob" queue. I can also log in to redis-cli and see 50 sets under the list key "queues:processjob".
My issue is that as soon as I attempt to put the throttle on, only 1 job runs and the rest fail with the following error:
Predis\Response\ServerException: ERR Error running script (call to f_29cc07bd431ccbf64637e5dcb60484560fdfa2da): #user_script:10: WRONGTYPE Operation against a key holding the wrong kind of value in /var/www/html/smhub/vendor/predis/predis/src/Client.php:370
If I remove the throttle, all works file and 5 jobs are instantly ran.
I thought maybe it was the incorrect key name but if I change the following:
public function handle()
{
//
Redis::throttle('queues:processjob')->allow(1)->every(60)->then(function(){
Storage::disk('local')->append('testFile.txt',date("Y-m-d H:i:s"));
}, function (){
return $this->release(10);
});
}
to this:
public function handle()
{
//
Redis::funnel('queues:processjob')->limit(1)->then(function(){
Storage::disk('local')->append('testFile.txt',date("Y-m-d H:i:s"));
}, function (){
return $this->release(10);
});
}
then it all works fine.
My thoughts...
Something tells me that the issue is that the redis key is of type "list" and that the jobs are all under a single list. That being said, if it didn't work this way, how would we throttle a queue as the throttle requires a unique key.
For anybody else that is having issues attempting to get this to work and is getting the same issue as I was, this is what resolved my issues:
The Fault
I assumed that Redis::throttle('queues:processjob') was meant to be referring to the queue that you wanted to be throttled. However, after some re-reading of the documentation and testing of the code, I realized that this was not the case.
The Fix
Redis::throttle('queues:processjob') is meant to point to it's own 'holding' queue and so must be a unique Redis key name. Therefore, changing it to Redis::throttle('throttle:queues:processjob') worked fine for me.
The workings
When I first looked in to this, I assumed that that Redis::throttle('this') throttled the queue that you specified. To some degree this is correct but it will not work if the job was created via another means.
Redis::throttle('this') actually creates a new 'holding' queue where the jobs go until the condition(s) you specify are met. So jobs will go to the queue 'this' in this example and when the throttle trigger is released, they will be passed to the queue specified in their execution code. In this case, 'queues:processjob'.
I hope this helps!
I use cache DatabaseStore for managing mutex files with Laravel scheduler.
I modified app/Console/Kernel.php a bit to make this work.
protected function defineConsoleSchedule()
{
$this->app->bind(SchedulingMutex::class, function() {
$container = Container::getInstance();
$mutex = $container->make(CacheSchedulingMutex::class);
return $mutex->useStore('database');
});
parent::defineConsoleSchedule();
}
I need it to be able to run scheduler on multiple servers, but it requires to have a shared storage. Since I have different Redis instances for all servers I decided to use database cache storage which is provided out of the box.
All works fine, but the db table named cache, where all things are stored does not get cleaned up even after cache expired.
Here is some tuples from the table:
key value expiration
laravelframework/schedule-015655105069... b:1; 1539126032
laravelframework/schedule-015655105069... b:1; 1539126654
So the first one, has expiration 1539126032 (2018-10-09 23:00:32), current time is 2018-10-10 08:09:45. I expect it to be cleaned up.
The question is - should I implement something to maintain this table or Laravel should handle it? What I'm doing wrong if it's Laravel's duty?
I'm trying to figure something out, i built a logger that keeps track of certain events happening. These happen fairly often (ie 1-500 times a minute).
To optimize this properly, i'm storing to redis and then i have a task that grabs the queue object from redis, clears the cache key and will insert each individual entry into the db.
I have the enqueue happening in the destructor when my logger is done observing the data.
For obvious reasons, i dont want a db write happening every time, so to speed this up i write to redis and then flush to db on a task.
The issue is that my queue implementation is as follows:
fetch object with key xyz from redis
append new entry to object
store object with key xyz in redis
This is inefficient, i would like to be able to just enqueue straight into redis. Redis has a list type built-in which i could use, but laravel redis driver doesn't support it. I tried to figure out a way to send raw commands to redis from laravel, but I can't seem to make it work.
I was thinking of just storing keys into a tag in redis, but i quickly discovered laravel's implementation of tags is not 'proper' and will not allow fetching tagged items without a key, so i can't use tags as a queue and each key an object in it.
If anyone has any idea how i could either talk to redis directly and make use of list, or if there's something i missed, it would really help.
EDIT
While the way below does work, there is a more proper way of doing it using a facade for redis as a reply mentioned, more on it here: Documentation
Okay, if anyone runs into this. I did not see it documented properly anywhere, you need to do the following:
-> get an instance of redis through the Cache facade laravel provides.
$redis = Cache::getRedis();
-> call redis functions through it.
$redis->rpush('test.key', 'data');
I believe you'll want predis as your driver.
If you're building a log driver, you'll want these 2 functions implemented:
public function cacheEnqueue($data) : int
{
$size = $this->_redis->lpush($this->getCacheKey("c_stack_queue"), $data);
if ($size > 2000)
{
//prevent runaway logs
$this->_redis->ltrim($this->getCacheKey("c_stack_queue"), 0, 2000);
}
return $size;
}
/**
* Fetch items from stack. Multi-thread safe,
*
* #param int $number Fetch last x items.
*
* #return array
*/
public function cachePopMulti(int $number) : array
{
$data = $this->_redis->lrange($this->getCacheKey("c_stack_queue"), -$number, -1);
$this->_redis->ltrim($this->getCacheKey("c_stack_queue"), 0, -1 * ($number + 1));
return $data;
}
Of course, write your own key generator getCacheKey()