How to implement a counter that decays over time? - algorithm

Question

Requirements of special counter
I want to implement a special counter: all increment operations time out after a fixed period of time (say 30 days).
An example:
Day 0: counter = 0. TTL = 30 days
Day 1: increment counter (+1)
Day 2: increment counter (+1)
Day 3: value of counter == 2
Day 31: value of counter == 1
Day 32: value of counter == 0
Naive solution
A naïve implementation is to maintain a set of timestamps, where each timestamp equals the time of an increment. The value of the counter equals the size of the set after subtracting all timestamps that have timed out.
This naïve counter has O(n) space (size of the set), has O(n) lookup and O(1) inserts. The values are exact.
Better solution (for me)
Trade speed and memory for accuracy.
I want a counter with O(1) lookup and insert, O(1) space. The accuracy < exact.
Alternatively, I would accept O(log n) space and lookup.
The counter representation should be suited for storage in a database field, i.e., I should be able to update and poll the counter rapidly without too much (de)serialization overhead.
I'm essentially looking for a counter that resembles a HyperLogLog counter, but for a different type of approximate count: decaying increments vs. number of distinct elements
How could I implement such a counter?

Answer 1

If you can live with 24 hour granularity then you can bucket your counter into k buckets where k is the number of days in your longest TTL.
Incrementing is an O(1) operation - simply increment the value in the bucket with index (k-TTL), as well as the current sum total.
Reading is another O(1) operation as you simply read the current sum total.
A cronjob pops off the now-expired bucket each night (and adds a bucket with value 0 at the opposite end) and decreases your counter by the sum in that bucket (this is a background task so it would not affect your insert or read operations)

Answer 2

Decaying counter based on annealing
Here is a counter that is based on annealing (implemented in Python).
The counter exponentially decays over time; controlled by the rate alpha
When you read and write the counter, you provide a time index (increment or read the counter at time t)
You can read the counter in the present and future (w.r.t. index of last increment), but not in the past
Time indices of sequential increments must be weakly monotonically increasing
The algorithm is exact w.r.t. the alternative formulation (annealing vs. TTL). It has O(1) increment and read. It consumes O(1) space, in fact just three floating point fields.
class AnnealingCounter():
def __init__(self, alpha=0.9):
self.alpha = alpha # rate of decay
self.last_t = .0 # time of last increment
self.heat = .0 # value of counter at last_t
def increment(self, t=None, amount=1.0):
"""
t is a floating point temporal index.
If t is not provided, the value of last_t is used
"""
if t is None: t = self.last_t
elapsed = t - self.last_t
if elapsed < .0 :
raise ValueError('Cannot increment the counter in the past, i.e. before the last increment')
self.heat = amount + self.heat * (self.alpha ** elapsed)
self.last_t = t
def get_value(self, t=None):
"""
t is a floating point temporal index.
If t is not provided, the value of last_t is used
"""
if t is None: t = self.last_t
elapsed = t - self.last_t
if elapsed < .0 :
raise ValueError('Cannot increment the counter in the past, i.e. before the last increment')
return self.heat * (self.alpha ** elapsed)
def __str__(self):
return str('Counter value at time {}: {}'.format(self.last_t, self.heat))
def __repr__(self):
return self.__str__()
Here is how to use it:
>>> c = AnnealingCounter(alpha=0.9)
Counter has value 0.0 at time 0.0
>>> c.increment() # increment by 1.0, but don't move time forward
Counter has value 1.0 at time 0.0
>>> c.increment(amount=3.2, t=0.5) # increment by 3.2 and move time forward (t=0.5)
Counter has value 4.14868329805 at time 0.5
>>> c.increment() # increment by 1.0, but don't move time forward
Counter has value 5.14868329805 at time 0.5
>>> c.get_value() # get value as after last increment (t=0.5)
5.148683298050514
>>> c.get_value(t=2.0)
4.396022866630942 # get future value (t=2.0)

Answer 3

Since the increments expire in the same order as they happen, the timestamps form a simple queue.
The current value of the counter can be stored separately in O(1) additional memory. At the start of each operation (insert or query), while the front of the queue is expired, it's popped out of the queue, and the counter is decreased.
Note that each of the n timestamps is created and popped out once. Thus you have O(1) amortized time to access the current value, and O(n) memory to store the non-expired timestamps. The actual highest memory usage is also limited by the ratio of TTL / frequency of new timestamp insertions.