Between 4 and 5 o'clock a remote system is regularly down.
This means some cron jobs produce exceptions.
Is there a way to ignore these exceptions.
But exceptions before or after that time period are important.
This is currently not possible with Sentry.
If you want you can watch this GitHub Sentry issue: Mute whole projects in case of maintenance downtime #1517.
Actually, there is a workaround for that;
Sentry.init(options -> {
options.setBeforeSend((event, hint) -> {
if (time is between 4-5 o-clock) {
return null;
}
return event;
});
}
);
Related
My framework is Laravel 7 and the Cache driver is Memcached. I want to perform atomic cache get/edit/put. For that I use Cache::lock() but it doesn't seem to work. The $lock->get() returns false (see below). How can I resolve this?
Fort testing, I reload Homestead, and run only the code below. And locking never happens. Is it possible Cache::has() break the lock mechanism?
if (Cache::store('memcached')->has('post_' . $post_id)) {
$lock = Cache::lock('post_' . $post_id, 10);
Log::info('checkpoint 1'); // comes here
if ($lock->get()) {
Log::info('checkpoint 2'); // but not here.
$post_data = Cache::store('memcached')->get('post_' . $post_id);
... // updating $post_data..
Cache::put('post_' . $post_id, $post_data, 5 * 60);
$lock->release();
}
} else {
Cache::store('memcached')->put('post_' . $post_id, $initial, 5 * 60);
}
So first of all a bit of background.
A mutual exclusion (mutex) lock as you correctly mentioned is meant to prevent race conditions by ensuring only one thread or process ever enters a critical section.
But first of all what is a critical section?
Consider this code:
public function withdrawMoney(User $user, $amount) {
if ($user->bankAccount->money >= $amount) {
$user->bankAccount->money = $user->bankAccount->money - $amount;
$user->bankAccount->save();
return true;
}
return false;
}
The problem here is if two processes run this function concurrently, they will both enter the if check at around the same time, and both succeed in withdrawing, however this might lead the user having negative balance or money being double-withdrawn without the balance being updated (depending on how out of phase the processes are).
The problem is the operation takes multiple steps and can be interrupted at any given step. In other words the operation is NOT atomic.
This is the sort of critical section problem that a mutual exclusion lock solves. You can modify the above to make it safer:
public function withdrawMoney(User $user, $amount) {
try {
if (acquireLockForUser($user)) {
if ($user->bankAccount->money >= $amount) {
$user->bankAccount->money = $user->bankAccount->money - $amount;
$user->bankAccount->save();
return true;
}
return false;
}
} finally {
releaseLockForUser($user);
}
}
The interesting things to point out are:
Atomic (or thread-safe) operations don't require such protection
The code we put between the lock acquire and release, can be considered to have been "converted" to an atomic operation.
Acquiring the lock itself needs to be a thread-safe or atomic operation.
At the operating system level, mutex locks are typically implemented using atomic processor instructions built for this specific purpose such as an atomic test-and-set operation. This would check if a value if set, and if it is not set, set it. This works as a mutex if you just say the lock itself is the existence of the value. If it exists, the lock is taken and if it's not then you acquire the lock by setting the value.
Laravel implements the locks in a similar manner. It takes advantage of the atomic nature of the "set if not already set" operations that certain cache drivers provide which is why locks only work when those specific cache drivers are there.
However here's the thing that's most important:
In the test-and-set lock, the lock itself is the cache key being tested for existence. If the key is set, then the lock is taken and cannot generally be re-acquired. Typically locks are implemented with a "bypass" in which if the same process tries to acquire the same lock multiple times it succeeds. This is called a reentrant mutex and allows to use the same lock object throughout your critical section without worrying about locking yourself out. This is useful when the critical section becomes complicated and spans multiple functions.
Now here's where you have two flaws with your logic:
Using the same key for both the lock and the value is what is breaking your lock. In the lock analogy you're trying to store your valuables in a safe which itself is part of your valuables. That's impossible.
You have if (Cache::store('memcached')->has('post_' . $post_id)) { outside your critical section but it should itself be part of the critical section.
To fix this issue you need to use a different key for the lock than you use for the cached entries and move your has check in the critical section:
$lock = Cache::lock('post_' . $post_id. '_lock', 10);
try {
if ($lock->get()) {
//Critical section starts
Log::info('checkpoint 1'); // if it comes here
if (Cache::store('memcached')->has('post_' . $post_id)) {
Log::info('checkpoint 2'); // it should also come here.
$post_data = Cache::store('memcached')->get('post_' . $post_id);
... // updating $post_data..
Cache::put('post_' . $post_id, $post_data, 5 * 60);
} else {
Cache::store('memcached')->put('post_' . $post_id, $initial, 5 * 60);
}
}
// Critical section ends
} finally {
$lock->release();
}
The reason for having the $lock->release() in the finally part is because in case there's an exception you still want the lock being released rather than staying "stuck".
Another thing to note is that due to the nature of PHP you also need to set a duration that the lock will be held before it is automatically released. This is because under certain circumstances (when PHP runs out of memory for example) the process terminates abruptly and therefore is unable to run any cleanup code. The duration of the lock ensures the lock is released even in those situations and the duration should be set as the absolute maximum time the lock would reasonably be held.
Cache::lock('post_' . $post_id, 10)->get() return false, because the 'post_' . $post_id is locked, the lock has not been released.
So you need to release the lock first:
Cache::lock('post_' . $post_id)->release()
// or release a lock without respecting its current owner
Cache::lock('post_' . $post_id)->forceRelease();
then try again, it will return true.
And recommend to use try catch or block to set a specified time limit, Laravel will wait for this time limit. An Illuminate\Contracts\Cache\LockTimeoutException will be thrown, the lock can be released.
use Illuminate\Contracts\Cache\LockTimeoutException;
$lock = Cache::lock('post_' . $post_id, 10);
try {
$lock->block(5);
...
Cache::put('post_' . $post_id, $post_data, 5 * 60);
$lock->release();
// Lock acquired after waiting maximum of 5 seconds...
} catch (LockTimeoutException $e) {
// Unable to acquire lock...
} finally {
optional($lock)->release();
}
Cache::lock('post_' . $post_id, 10)->block(5, function () use ($post_id, $post_data) {
// Lock acquired after waiting maximum of 5 seconds...
...
Cache::put('post_' . $post_id, $post_data, 5 * 60);
$lock->release();
});
In my case, my Redis configuration causes the issue that makes Cache:lock always return false. It is because I rename commands DEL and FLUSHDB on the configuration file which is used by Laravel to release the lock.
I think renaming the command will improve security but it causes problems on the application level. So, if someone uses Redis as the Driver then don't rename DEL and FLUSHDB. I need an hour to figure it out and hopefully, it help someone else.
The configuration file in Debian at /etc/redis/redis.conf like bellow
rename-command FLUSHDB ""
rename-command DEL ""
I am currently dispatching queued jobs to send API Events instantly, in busy times these queued jobs need to be held until overnight when the API is less busy, how can I hold these queued jobs or schedule them to only run from 01:00am the following day.
the Queued Job call currently looks like:
EliQueueIdentity::dispatch($EliIdentity->id)->onQueue('eli');
there are other jobs on the same queue, all of which will need to be held in busy times
Use delay to run job at a certain time.
EliQueueIdentity::dispatch($EliIdentity->id)
->onQueue('eli')
->delay($this->scheduleDate());
Helper for calculating the time, handling a edge case between 00:00 to 01:00, where it would delay it a whole day. While not specified how to handle busy, made an pseudo example you can implement.
private function scheduleDate()
{
$now = Carbon::now();
if (! $this->busy()) {
return $now;
}
// check for edge case of 00:00:00 to 01
if ($now->hour <= 1) {
$now->setTime(1, 0, 0);
return $now;
}
return Carbon::tomorrow()->addHour();
}
You can use delayed dispatching (see https://laravel.com/docs/6.x/queues#delayed-dispatching):
// Run it 10 minutes later:
EliQueueIdentity::dispatch($EliIdentity->id)->onQueue('eli')->delay(
now()->addMinutes(10)
);
Or pass another carbon instance like:
// Run it at the end of the current week (i believe this is sunday 23:59, havent checked).
->delay(Carbon::now()->endOfWeek());
// Or run it at february second 2020 at 00:00.
->delay(Carbon::createFromFormat('Y-m-d', '2020-02-02'));
You get the picture.
I have Spring Boot based cron job running:
#Scheduled(cron = "30 * * * * *}")
// #Scheduled(initialDelay = -1, fixedDelay = 60000)
public void cronCheck()
{
instance.refreshStatus();
if (instance.status.isVerified() && !instance.status.isExpired())
{
instance.updateCheckTime();
}
}
Most of the cases, it's running perfect. But when I changed the system time back , for example one month, it would run one single time and never run again. However, if I change system back forward, it would run as scheduled.
Anyone has any idea why this is happening and maybe a solution?
Highly appreciate it!
Need some help understanding on where disconnects occur (SocketJS, Vertx) and how timeouts can be configured.
I am creating SockJSServer along with creating eventBus bridge. Problem that I observer is frequent WebSocket connection disconnects. Looking at the websocket frames, I see pings every 5 seconds and heart-beats what I configured every 1/2 sec(what seem to take effect). However, once heart beats are being delayed for longer then 5 second disconnects comes with message c[3000,'Go away']. As observed it happens when server is busy(doing something else on separate thread).
I have searched Vertx documentation and looked over vertx code and found few configuration parameter(which appear to be different across versions and documentation).
.putNumber("ping_interval", 120000)
.putNumber("session_timeout", 1200000)
.putNumber("heartbeat_period",500)
To be absolutely sure, I have tried different config that did not appear to have any impact. At this point, I think I have reached dead wall and need some help.
Vertx version 2.1P3
Server snipet
final SockJSServer server = vertx.createSockJSServer(httpServer);
server.bridge(new JsonObject().putString("prefix", "/eventbus")
.putNumber("ping_interval", 120000)
.putNumber("session_timeout", 1200000)
.putNumber("heartbeat_period",500),
new JsonArray().addObject(new JsonObject()),
new JsonArray().addObject(new JsonObject()));
Client code:
var eventBus = new EventBus('//hostX:12001/eventbus');
When you receive a SOCKET_IDLE event, you can't complete the event with a "true" parameter, as it indicates the socket must be closed:
SockJSHandler.create(vertx,handlerOptions).bridge(options, event -> {
boolean result = true;
switch(event.type()) {
case SOCKET_CREATED:
LOGGER.info("Socket created");
break;
case SOCKET_IDLE:
result = false;
return;
case SOCKET_CLOSED:
LOGGER.info("Socket closed");
break;
}
event.complete(result);
});
The run method of my worker role is:
public override void Run()
{
Message msg=null;
while (true)
{
msg = queue.GetMessage();
if(msg!=null && msg.DequeueCount==1){
//delete message
...
//execute operations
...
}
else if(msg!=null && msg.DequeueCount>1){
//delete message
...
}
else{
int randomTime = ...
Thread.Sleep(randomTime);
}
}
}
For performance tests I would that a message could be analysed only by a worker (I don't consider failure problems on workers).
But seems by my tests, that two workers can pick up the same message and read DequeueCount equals to 1 (both workers). Is it possible?
Does exist a way that allow just a worker to read a message in a "mutex" way?
How is your "getAMessage(queue)" method defined? If you do PeekMessage(), a message will be visible by all workers. If you do GetMessage(), the message will be got only by the worker which firsts get it. But for the invisibility timeout either specified or the default (30 sec.). You have to delete the message before the invisibility timeout comes.
Check out the Queue Service API for more information. I am sure that there is something wrong in your code. I use queues and they behave as by documentation in dev storage and in production storage. You may want to explicitly put higher value of the Visibility Timeout when you do GetMessage. And make sure you do not sleep longer than the visibility timeout.