We all know that redis cache has ttl timeouts for the cache. I would like to know if there is a provision in redis cache to increase the ttl for every key based on fetch of the data .
That means if the data is fetched from the redis for a key then automatically the ttl is increased.
Pls help me to get some info on that.
You can achieve that with Lua scripting: get val and TTL (in millisecond), increment TTL, set new TTL:
local key = KEYS[1]
local pttl_incr = ARGV[1]
local val = redis.call("get", key)
if not val then return nil end
local pttl = redis.call("pttl", key)
pttl = pttl + pttl_incr
redis.call("expire", key, pttl)
return val
Related
I am using Laravel 9 with the Redis cache driver. However, I have an issue where the internal standard_ref and forever_ref map that Laravel uses to manage tagged cache exceed more than 10MB.
This map consists of numerous keys, 95% of which have already expired/decayed and no longer exist; this map seems to grow in size and has a TTL of -1 (never expire).
Other than "not using tags", has anyone else encountered and overcome this? I found this in the slow log of Redis Enterprise, which led me to realize this is happening:
I checked the key/s via SCAN and can confirm it's a massive set of cache misses. It seems highly inefficient and expensive to constantly transmit 10MB back and forth to find one key within the map.
This quickly and efficiently removes expired keys from the SET data-type that laravel uses to manage tagged cache.
use Illuminate\Support\Facades\Cache;
function flushExpiredKeysFromSet(string $referenceKey) : void
{
/** #var \Illuminate\Cache\RedisStore $store */
$store = Cache::store()->getStore();
$lua = <<<LUA
local keys = redis.call('SMEMBERS', '%s')
local expired = {}
for i, key in ipairs(keys) do
local ttl = redis.call('ttl', key)
if ttl == -2 or ttl == -1 then
table.insert(expired, key)
end
end
if #expired > 0 then
redis.call('SREM', '%s', unpack(expired))
end
LUA;
$store->connection()->eval(sprintf($lua, $key, $key), 1);
}
To show the calls that this LUA script generates, from the sample above:
10:32:19.392 [0 lua] "SMEMBERS" "63c0176959499233797039:standard_ref{0}"
10:32:19.392 [0 lua] "ttl" "i-dont-expire-for-an-hour"
10:32:19.392 [0 lua] "ttl" "aa9465100adaf4d7d0a1d12c8e4a5b255364442d:i-have-expired{1}"
10:32:19.392 [0 lua] "SREM" "63c0176959499233797039:standard_ref{0}" "aa9465100adaf4d7d0a1d12c8e4a5b255364442d:i-have-expired{1}"
Using a custom cache driver that wraps the RedisTaggedCache class; when cache is added to a tag, I dispatch a job using the above PHP script only once within that period by utilizing a 24-hour cache lock.
Here is how I obtain the reference key that is later passed into the cleanup script.
public function dispatchTidyEvent(mixed $ttl)
{
$referenceKeyType = $ttl === null ? self::REFERENCE_KEY_FOREVER : self::REFERENCE_KEY_STANDARD;
$lock = Cache::lock('tidy:'.$referenceKeyType, 60 * 60 * 24);
// if we were able to get a lock, then dispatch the event
if ($lock->get()) {
foreach (explode('|', $this->tags->getNamespace()) as $segment) {
dispatch(new \App\Events\CacheTidyEvent($this->referenceKey($segment, $referenceKeyType)));
}
}
// otherwise, we'll just let the lock live out its life to prevent repeating this numerous times per day
return true;
}
Remembering that a "cache lock" is simply just a SET/GET and Laravel is responsible for many of those already on every request to manage it's tags, adding a lock to achieve this "once per day" concept only adds negligible overhead.
i want to migrate 70million data redis(sentinel-mode) to redis(cluster-mode)
ScanOptions options = ScanOptions.scanOptions().build();
Cursor<byte[]> c = sentinelTemplate.getConnectionFactory().getConnection().scan(options);
while(c.hasNext()){
count++;
String key = new String(c.next());
key = key.trim();
String value = (String)sentinelTemplate.opsForHash().get(key,"tc");
//Thread.sleep(1);
clusterTemplate.opsForHash().put(key, "tc", value);
}
I want to scan again from a certain point because redis connection disconnected at some point.
How to set the starting point when using the Redis scan command in spring boot?
Moreover, whenever the program is executed using the above code, the connection is broken when almost 20 million data are moved.
I am working on a feature where a customer can update their KMS key in our platform so that they are using their KMS key to encrypt data instead of one generated by us. The way it works is when a customer signs up, we generate a KMS key for them and upload the objects using that key. If the customer wants to provide their own key, I want to be able to update this key without having to pull down the data and re-upload with the new key.
def enc_client
Aws::S3::Encryption::Client.new(
kms_client: Aws::KMS::Client.new(region: 'us-east-1'),
kms_key_id: ENV['MY_PRIVATE_KEY']
)
end
def s3_client
enc_client.client
end
bucket = "my_bucket_name"
key = "path/12345abcde/preview.html"
copy_source = "/#{key}"
server_side_encryption = "aws:kms"
# This returns the object with the key present. If I go in the AWS client and manually add or remove the key, it will update on this call.
resp = s3_client.get_object(bucket: bucket, key: key)
#<struct Aws::S3::Types::GetObjectOutput
body=#<StringIO:0x000000000bb45108>,
delete_marker=nil,
accept_ranges="bytes",
expiration=nil,
restore=nil,
last_modified=2019-04-12 15:40:09 +0000,
content_length=19863445,
etag="\"123123123123123123123123123123-1\"",
missing_meta=nil,
version_id=nil,
cache_control=nil,
content_disposition="inline; filename=\"preview.html\"",
content_encoding=nil,
content_language=nil,
content_range=nil,
content_type="text/html",
expires=nil,
expires_string=nil,
website_redirect_location=nil,
server_side_encryption="aws:kms",
metadata={},
sse_customer_algorithm=nil,
sse_customer_key_md5=nil,
ssekms_key_id="arn:aws:kms:us-east-1:123456789123:key/222b222b-bb22-2222-bb22-222bbb22bb2b",
storage_class=nil,
request_charged=nil,
replication_status=nil,
parts_count=nil,
tag_count=nil>
new_ssekms_key_id = "arn:aws:kms:us-east-1:123456789123:key/111a111a-aa11-1111-aa11-111aaa11aa1a"
resp = s3_client.copy_object(bucket: bucket, key: key, copy_source: copy_source, ssekms_key_id: ssekms_key_id)
Aws::S3::Errors::InvalidArgument: Server Side Encryption with AWS KMS managed key requires HTTP header x-amz-server-side-encryption : aws:kms
from /usr/local/bundle/gems/aws-sdk-core-3.6.0/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'
resp = s3_client.copy_object(bucket: bucket, key: key, copy_source: copy_source, ssekms_key_id: ssekms_key_id, server_side_encryption: server_side_encryption)
Aws::S3::Errors::AccessDenied: Access Denied
from /usr/local/bundle/gems/aws-sdk-core-3.6.0/lib/seahorse/client/plugins/raise_response_errors.rb:15:in `call'
I would like to be able to update the kms id do a new one on the server side
copy_source = "/#{key}" is incorrect. The value should be "/#{bucket}/#{key}".
The service is interpreting the first element of your key path as the name of a bucket -- probably someone else's bucket.
I tried enabling TTL for records in Ignite using 2 approaches, but didn't seems to be working. Need help to understand if I am missing something.
IgniteCache cache = ignite.getOrCreateCache(IgniteCfg.CACHE_NAME);
cache.query(new SqlFieldsQuery(
"CREATE TABLE IF NOT EXISTS City (id LONG primary key, name varchar, region varchar)"))
.getAll();
cache.withExpiryPolicy(new CreatedExpiryPolicy(new Duration(TimeUnit.SECONDS, 10)))
.query(new SqlFieldsQuery(
"INSERT INTO City (id, name, region) VALUES (?, ?, ?)").setArgs(1, "Forest Hill1", "GLB"))
.getAll();
So you see above I created table in Cache and inserted record mentioning expiry TTL for 10 seconds, but seems that it never expires.
I tried another approach of rather than setting TTL while inserting the record, I mentioned in CacheConfiguration while I initialize Ignite, below is the code sample
Ignition.setClientMode(true);
IgniteConfiguration cfg = new IgniteConfiguration();
// Disabling peer-class loading feature.
cfg.setPeerClassLoadingEnabled(false);
CacheConfiguration ccfg = createCacheConfiguration();
cfg.setCacheConfiguration(ccfg);
ccfg.setEagerTtl(true);
ccfg.setExpiryPolicyFactory(CreatedExpiryPolicy.factoryOf(new Duration(TimeUnit.SECONDS, 5)));
TcpCommunicationSpi commSpi = new TcpCommunicationSpi();
cfg.setCommunicationSpi(commSpi);
TcpDiscoveryVmIpFinder tcpDiscoveryFinder = new TcpDiscoveryVmIpFinder();
String[] addresses = { "127.0.0.1" };
tcpDiscoveryFinder.setAddresses(Arrays.asList(addresses));
TcpDiscoverySpi discoSpi = new TcpDiscoverySpi();
discoSpi.setIpFinder(tcpDiscoveryFinder);
cfg.setDiscoverySpi(discoSpi);
return Ignition.start(cfg);
Executing Ignite locally (not as in memory) as my final goal is to be able to connect to same Ignite from multiple instances of app or even multiple apps.
Ignite SQL currently doesn't interact with expiry policies and doesn't update TTL. There is a Feature Request for that: https://issues.apache.org/jira/browse/IGNITE-7687.
I am running a Spark Streaming application based on mapWithState DStream function . The application transforms input records into sessions based on a session ID field inside the records.
A session is simply all of the records with the same ID . Then I perform some analytics on a session level to find an anomaly score.
I couldn't stabilize my application because a handful of sessions are getting bigger at each batch time for extended period ( more than 1h) . My understanding is a single session (key - value pair) is always processed by a single core in spark . I want to know if I am mistaken , and if there is a solution to mitigate this issue and make the streaming application stable.
I am using Hadoop 2.7.2 and Spark 1.6.1 on Yarn . Changing batch time, blocking interval , partitions number, executor number and executor resources didn't solve the issue as one single task makes the application always choke. However, filtering those super long sessions solved the issue.
Below is a code updateState function I am using :
val updateState = (batchTime: Time, key: String, value: Option[scala.collection.Map[String,Any]], state: State[Seq[scala.collection.Map[String,Any]]]) => {
val session = Seq(value.getOrElse(scala.collection.Map[String,Any]())) ++ state.getOption.getOrElse(Seq[scala.collection.Map[String,Any]]())
if (state.isTimingOut()) {
Option(null)
} else {
state.update(session)
Some((key,value,session))
}
}
and the mapWithStae call :
def updateStreamingState(inputDstream:DStream[scala.collection.Map[String,Any]]): DStream[(String,Option[scala.collection.Map[String,Any]], Seq[scala.collection.Map[String,Any]])] ={//MapWithStateDStream[(String,Option[scala.collection.Map[String,Any]], Seq[scala.collection.Map[String,Any]])] = {
val spec = StateSpec.function(updateState)
spec.timeout(Duration(sessionTimeout))
spec.numPartitions(192)
inputDstream.map(ds => (ds(sessionizationFieldName).toString, ds)).mapWithState(spec)
}
Finally I am applying a feature computing session foreach DStream , as defined below :
def computeSessionFeatures(sessionId:String,sessionRecords: Seq[scala.collection.Map[String,Any]]): Session = {
val features = Functions.getSessionFeatures(sessionizationFeatures,recordFeatures,sessionRecords)
val resultSession = new Session(sessionId,sessionizationFieldName,sessionRecords)
resultSession.features = features
return resultSession
}