Pyspark Fetch data from database periodically if certain duration has elapsed - jdbc

Question

I am trying to read data from database periodically in Pyspark based on the condition, current_time - lastReadTime > refresh_interval.
The refresh_interval that I have provided is 5min.
It's a structured streaming with Kafka and I join the data coming from postgres later.
However, whenever I change the data in database within 5min, I am getting the new data from database even though 5min has not passed.
Below is the code I am using.
def __init__(self, config,spark):
self.refresh_frequency_sec = config.getint('postgres-config', 'refresh-frequency-sec')
self.spark = spark
self.lastMetaReadTime = time()
self.rules = self.fetchRules()
def fetchRules(self):
jdbcDF = self.spark.read \
.format("jdbc") \
.option("driver", "org.postgresql.Driver")\
.option("url", self.connection_url) \
.option("dbtable", self.dbtable) \
.option("user", self.user) \
.option("password", self.password) \
.load()
return jdbcDF
def getRules(self):
if time() - self.lastMetaReadTime > self.refresh_frequency_sec:
self.rules = self.fetchRules()
self.lastMetaReadTime = time()
return self.rules
What am I doing wrong?

Answer 1

You are calling the wrong method for 'self.rules' in your innit() method.
You innit method should rather be:
def __init__(self, config,spark):
self.refresh_frequency_sec = config.getint('postgres-config', 'refresh-frequency-sec')
self.spark = spark
self.lastMetaReadTime = time()
self.rules = self.getRules()