validate and processing data in Redis sorted set efficiently - performance

Question

we have micro service (written in Go lang) which it's primary purpose is to get the logs from multiple IoT devices and do some processing on them and put the result into a PostgreSQL table. The way the system works is that each device has its own sorted set which the logs will be saved there and for each log the score would be a timestamp ( of course I know time series would be a better decision but we currently want to worked with sorted sets). know this logs come every 1 second from each device.
I want to process the data inside these sets every 5 second, but for each set, the logs inside should pass some tests:
there should be more than one log inside the set
two logs can be removed from the set, if the time difference between timestamps is 1 second
when the logs are validated then they can be passed to other methods or functions to the the rest of the processing. If logs are invalid ( there exists a log that has time difference of more than 1 second with other logs) then it go's back to the set and wait for the next iteration to be checked again.
Problem:
My problem is basically that I don't know how to get the data out of the list, validate them and put them back again! to be more clear for each set,all or none of the logs inside can be removed, and this occurs while new data is coming in contently, and since I cant validate the data with redis it self I don't know what to do. My current solution is as follows:
Every 5 seconds, all data from each set should be removed from Redis and saved in some data structure inside the code ( like a list...) the after validating, some logs that are not yet validated should be putted back to Redis. as you can see these solution needs two database access from the code, and when putting the invalid logs, they should be sorted by Redis ...
when the logs are so much and there are many devices, I think this solution is not the best way to go. I'm not very experienced with Redis so would be thankful to give your comments on the problem. Thanks

Answer 1

Since you decided to use sorted sets, here are things to know first
"there should be more than one log inside the set". If there is no element in the sorted set, than the set/key doesn't exist. You can check if there is any log in the sorted set via two different commands; zcard and exists - both works in O(1).
There can't be same log(really same) in the sorted set more than once. You need an identifier(such as timestamp, uuid, hash etc) to separate each individual log from each other in a single sorted set. It will update the score of existing element (it may not be what you want)
127.0.0.1:6379> zadd mydevice 1234 "log-a"
(integer) 1
127.0.0.1:6379> zadd mydevice 12345 "log-a"
(integer) 0
127.0.0.1:6379> zrange mydevice 0 -1 withscores
1) "log-a"
2) "12345"
127.0.0.1:6379>
There is no single way to do this on data layer with built-in methods. You will need application layer with business logic to accomplish what you need.
My suggestions would be keeping the combination of every IOT device + minute separate in a different sorted set. So every minute each device will have a different key, you will append minute 2020:06:06:21:21 to the device identifier key, it will put at most 60 logs. you can check it with zcard
It would be something like this;
127.0.0.1:6379> zadd device:1:2020:06:06:21:21 1591442137 my-iot-payload
(integer) 1
127.0.0.1:6379> zadd device:1:2020:06:06:21:21 1591442138 my-iot-payload-another
(integer) 1
127.0.0.1:6379> zadd device:1:2020:06:06:21:21 1591442138 my-iot-payload-yet-another
(integer) 1
127.0.0.1:6379> zrange device:1:2020:06:06:21:21 0 -1
1) "my-iot-payload"
2) "my-iot-payload-another"
3) "my-iot-payload-yet-another"
127.0.0.1:6379>
In your application layer;
Every minute for every device you check for the sorted sets (I know you said 5 seconds but if you want to do it you need a modulo way to separate them in 5 seconds interval keys instead of minute one)
You have the list of devices(maybe in your database table), you know what time is it(convert to redis key)
Get minute/device separated keys with zrange(withscores option) to make calculations and validations at application level for each device and for that exact minute.
If they pass then save into your PostgreSQL database(delete sorted set key or execute expire whenever you add new element with zadd).
If they fail, that's totally up to you. You have minute separated logs for each device, you may delete it or parse them partially to save it.